Patent Description:
Computer systems use rendering procedures to present three-dimensional (3D) scene objects on a two-dimensional (2D) display. To create a 3D representation of the scene objects, the computer system obtains geometric information of the scene objects from multiple 2D images having different viewpoints. The computer system then creates a depth map from the obtained geometric information for use in creating and rendering the 3D scene on the 2D display.

A depth map is an image that contains information relating to the distance of the surfaces of scene objects from a viewpoint of imagers capturing 2D images of the scene objects. The depth is sometimes referred to as Z-depth, which refers to a convention that the central axis of view of an imager is in the direction of the imager's Z-axis, and not to the absolute Z-axis of a scene.

The computer system presents a 3D scene on a 2D display for viewing and manipulation by a user and the user is able to manipulate the scene objects of the 3D scene by changing the viewpoint of the 3D scene. For viewpoints of the scene objects where the 3D scene does not include accurate information (e.g., a color value, a depth value, and/or an object value because some aspects of an object/scene are not present in one or more of the 2D images obtained from the limited number of viewpoints used to create the 3D), the computer system will attempt to complete the scene using the information from adjacent pixels. When the computer system attempts to complete the scene (e.g., using information from adjacent pixels), the resultant scene often includes unrealistic looking shapes and/or colors (e.g., a "stretching" effect of colors).

To minimize the unrealistic looking shapes and/or colors, prior art techniques often obtain many more than two 2D images from two respective viewpoints (e.g., a panorama of images). Obtaining a panorama of images from multiple viewpoints increases the likelihood that at least one viewpoint of the scene objects includes information for use in generating the 3D scene from various viewpoints, thereby improving display accuracy. Such techniques, however, require a relatively large amount of processing time/power as compared to creating a 3D scene from just two 2D images.

United States patent application publication no. <CIT> relates to relates to depth images and obtaining higher resolution depth images through depth dependent measurement modeling. One example can receive a set of depth images of a scene captured by a depth camera. The example can obtain a depth dependent pixel averaging function for the depth camera. The example can also generate a high resolution depth image of the scene from the set of depth images utilizing the depth dependent pixel averaging function.

<NPL>, proposes solutions to remove artifacts and holes that are due to the synthesis of numerous viewpoints using a single reference view and its depth map in depth-image-based rendering, and to apply different filling strategies depending on the nature of each hole. Cracks are identified and filled using very local neighborhood information. Regions classified as ghosts are projected to their correct place. The remaining holes are classified as disocclusions or out-of-field areas, and filled with an appropriate adaptation of a popular inpainting method. In both adaptations, patch matching explores the spatial locality concept, using dynamically adaptive patch sizes from the reference image. For disocclusions we propose a filling order using depth and background terms, and a searching process that considers only background patches.

The invention pertains to a system as defined by independent claim <NUM> and to a method as defined by independent claim <NUM>. Optional features of the system and the method according to the invention are presented in the dependent claims.

The invention is best understood from the following detailed description when read in connection with the accompanying drawings, with like elements having the same reference numerals. When a plurality of similar elements are present, a single reference numeral may be assigned to the plurality of similar elements with a small letter designation referring to specific elements. When referring to the elements collectively or to a non-specific one or more of the elements, the small letter designation may be dropped. This emphasizes that according to common practice, the various features of the drawings are not drawn to scale unless otherwise indicated. On the contrary, the dimensions of the various features may be expanded or reduced for clarity. Included in the drawings are the following figures:.

In the following detailed description, numerous specific details are set forth by way of examples in order to provide a thorough understanding of the relevant teachings. However, it should be apparent to those skilled in the art that such details are not necessary to practice the present teachings. In other instances, a relatively high-level description, without detail, of well-known methods, procedures, components, and circuitry avoids unnecessarily obscuring aspects of the present teachings.

The term "coupled" as used herein refers to any logical, optical, physical or electrical connection, link or the like by which signals or light produced or supplied by one system element are imparted to another coupled element. Unless described otherwise, coupled elements or devices do not necessarily have to physically be in touch with one another and may be separated by airspace, intermediate components, elements or communication media that may modify, manipulate or carry the light or signals.

The orientations of the eyewear, associated components, and any devices shown in any of the drawings, are by way of example only, for illustration and discussion purposes. In operation, orientation of the eyewear may be in other directions suitable to the particular application of the eyewear, for example up, down, sideways, or any other orientation. Also, any directional term, such as front, rear, inwards, outwards, towards, left, right, lateral, longitudinal, up, down, upper, lower, top, bottom and side, is exemplary, and not limiting, as to direction or orientation.

<FIG> depicts a front perspective view of example eyewear <NUM> for capturing images. The illustrated eyewear <NUM> includes a support structure <NUM> that has temples 14A and 14B extending from a central frame portion <NUM>. The eyewear <NUM> additionally includes articulated joints 18A and 18B, electronic components 20A and 20B, and core wires 22A, 22B and <NUM>. Although the illustrated eyewear are glasses, the eyewear may take other forms such as a headset, head gear, helmet, or other device that may be worn by a user.

Support structure <NUM> supports one or more optical elements within a field of view of a user when worn by the user. For example, central frame portion <NUM> supports the one or more optical elements. As used herein, the term "optical elements" refers to lenses, transparent pieces of glass or plastic, projectors, screens, displays and other devices for presenting visual images or through which a user perceives visual images. In an example, respective temples 14A and 14B connect to the central frame portion <NUM> at respective articulated joints 18A and 18B. The illustrated temples 14A and 14B are elongate members having core wires 22A and 22B extending longitudinally therein.

Temple 14A is illustrated in a wearable condition and temple 14B is illustrated in a collapsed condition in <FIG>. As shown in <FIG>, articulated joint 18A connects temple 14A to a right end portion 26A of central frame portion <NUM>. Similarly, articulated joint 18B connects temple 14B to a left end portion 26B of central frame portion <NUM>. The right end portion 26A of central frame portion <NUM> includes a housing that carries electronic components 20A therein, and left end portion 26B includes a housing that carries electronic components 20B therein.

A plastics material or other material embeds core wire 22A, which extends longitudinally from adjacent articulated joint 18A toward a second longitudinal end of temple 14A. Similarly, the plastics material or other material also embeds core wire 22B, which extends longitudinally from adjacent articulated joint 18B toward a second longitudinal end of temple 14B. The plastics material or other material additionally embeds core wire <NUM>, which extends from the right end portion 26A (terminating adjacent electronic components 20A) to left end portion 26B (terminating adjacent electronic components 20B).

Electronic components 20A and 20B are carried by support structure <NUM> (e.g., by either or both of temple(s) 14A, 14B and/or central frame portion <NUM>). Electronic components 20A and 20B include a power source, power and communication related circuitry, communication devices, display devices, a computer, a memory, modules, and/or the like (not shown). Electronic components 20A and 20B may each include a respective imager 10A and 10B for capturing images and/or videos. In the illustrated example, imager 10A is adjacent the right temple 14A and imager 10B is adjacent the left temple 14B. The imagers 10A and 10B are spaced from one another in order to obtain images of scene objects from two different viewpoints for use in generating 3D scenes.

Support structure <NUM> defines a region (e.g., region <NUM> (<FIG>) defined by the frame <NUM> and temples 14A and 14B) for receiving a portion <NUM> (e.g., the main portion) of the head of the user/wearer. The defined region(s) are one or more regions containing at least a portion of the head of a user that are encompassed by, surrounded by, adjacent, and/or near the support structure when the user is wearing the eyewear <NUM>. In the illustrated example, the imagers 14A and 14B are positioned on the eyewear such that they are adjacent the respective eyes of a user when the eyewear <NUM> is worn, which facilitates obtaining a separation of viewpoints suitable for creating 3D scenes.

<FIG> is a block diagram of example electronic components coupled to a display system <NUM> (e. g, a display of a processing device or other technique for presenting information). The illustrated electronic components include a controller <NUM> (e.g., hardware processor) for controlling the various devices in the eyewear <NUM>; a wireless module (e.g., BluetoothTM) <NUM> for facilitating communication between the eyewear <NUM> and a client device (e.g., a personal computing device <NUM> such as a smartphone); a power circuit <NUM> (e.g., battery, filter, etc.) for powering eyewear <NUM>; a memory <NUM> such as flash storage for storing data (e.g., images, video, image processing software, etc.); a selector <NUM>; and one or more imagers <NUM> (two in the illustrated examples) for capturing one or more images (e.g., a picture or a video). Although the eyewear <NUM> and the personal computing device are illustrated as separate components, the functionality of the personal computing device may be incorporated into the eyewear enabling the personal computing device and/or the eyewear <NUM> to perform functionality described herein.

The selector <NUM> may trigger (e.g., via a momentary push of a button) controller <NUM> of eyewear <NUM> to capture images/video. In examples where a single selector <NUM> is utilized, the selector may be used in a set up mode (e.g., entered by pressing and holding the selector <NUM> for a period of time, e.g., <NUM> seconds) and in an image capture mode (e.g., entered after a period of time with no contact, e.g., <NUM> seconds) to capture images.

In an example, the selector <NUM> may be a physical button on the eyewear <NUM> that, when pressed, sends a user input signal to the controller <NUM>. The controller <NUM> may interpret pressing the button for a predetermined period of time (e.g., three seconds) as a request to transition to a different mode of operation (e.g., in/out of a set-up mode of operation). In other examples, the selector <NUM> may be a virtual button on the eyewear or another device. In yet another example, the selector may be a voice module that interprets voice commands or an eye detection module that detects where the focus of an eye is directed. Controller <NUM> may interpret signals from selector <NUM> as a trigger to cycle through illuminating LEDs <NUM> to select an intended recipient of the image(s).

Wireless module <NUM> may couple with a client/personal computing device <NUM> such as a smartphone, tablet, phablet, laptop computer, desktop computer, networked appliance, access point device, or any other such device capable of connecting with wireless module <NUM>. Bluetooth, Bluetooth LE, Wi-Fi, Wi-Fi direct, a cellular modem, and a near field communication system, as well as multiple instances of any of these systems, for example, may implement these connection to enable communication there between. For example, communication between the devices may facilitate transfer of software updates, images, videos, lighting schemes, and/or sound between eyewear <NUM> and the client device.

In addition, personal computing device <NUM> may be in communication with one or more recipients (e.g., recipient personal computing device <NUM>) via a network <NUM>. The network <NUM> may be a cellular network, Wi-Fi, the Internet or the like that allows personal computing devices to transmit and receive an image(s), e.g., via text, email, instant messaging, etc. The computing devices <NUM>/<NUM> may each include a processor and a display. Suitable processors and displays, which may be configured to perform one more functions described herein, may be found in current generation personal computing devices and smartphones such as the iPhone <NUM>™ available from Apple Inc. of Cupertino, California and the Samsung Galaxy Note <NUM>™ available from the Samsung Group of Seoul, South Korea.

The imager(s) <NUM> for capturing the images/video may include digital camera elements such as a charge-coupled device, a lens, or any other light capturing elements for capturing image data for conversion into an electrical signal(s).

The controller <NUM> controls the electronic components. For example, controller <NUM> includes circuitry to receive signals from imager <NUM> and process those signals into a format suitable for storage in memory <NUM> (e.g., flash storage). Controller <NUM> powers on and boots to operate in a normal operational mode, or to enter a sleep mode. In one example, controller <NUM> includes a microprocessor integrated circuit (IC) customized for processing sensor data from imager <NUM>, along with volatile memory used by the microprocessor to operate. The memory may store software code for execution by controller <NUM>.

Each of the electronic components require power to operate. Power circuit <NUM> may include a battery, power converter, and distribution circuitry (not shown). The battery may be a rechargeable battery such as lithium-ion or the like. Power converter and distribution circuitry may include electrical components for filtering and/or converting voltages for powering the various electronic components.

<FIG> depicts a flow chart <NUM> illustrating example operation of eyewear (e.g., eyewear <NUM> of <FIG>) and <FIG> scene rendering by a processing system (e.g., by a processor of eyewear <NUM> and/or the processor of a computing device remote to the eyewear. For ease of explanation, the steps of flow chart <NUM> are described with reference to eyewear <NUM> described herein. One of skill in the art will recognize other imager configurations not tied to eyewear for use in rendering 3D scenes. Additionally, it is to be understood that one or more of the steps may be omitted, performed by another component, or performed in a different order.

At step <NUM>, obtain a first 2D image of a scene object from a first viewpoint and a second 2D image of the scene object from a second viewpoint. In an example, a first imager 10A of the eyewear <NUM> captures the first 2D image of the scene object from the first viewpoint and a second imager 10B of the eyewear <NUM> captures the second 2D image of the scene object from the second viewpoint. The captured images pass from the imagers 10A and 10B to a processing system for rendering of a 3D scene. In one example, the controller <NUM> of the eyewear <NUM> obtains the 2D images and renders the 3D scene from the obtained 2D images. In another example, the controller <NUM> receives the 2D images and transmits them to a processing system of a remote computing device <NUM>/<NUM> for rendering of the 3D scene.

At step <NUM>, reconstruct the first and second 2D images into a 3D scene of the scene object. The processing system reconstructs the first and second 2D images into the 3D scene of the scene object. The processing system may apply a stereo-vision processing technique (i.e., to create a depth map) and geometric processing technique to create the 3D scene. In an example, the rendered 3D scene includes geometric features (e.g., vertices with x-axis, y-axis, and z-axis coordinates) along with image information (e.g., color information, depth information, and object information). The rendered 3D scene will also include connective as multi-angular faces, e.g., typically triangular faces or quadrangular faces, connecting the vertices to make up the textured surfaces of the scene objects. Suitable stereo-vision processing techniques and geometric processing techniques will be understood by one of skill in the art from the description herein.

<FIG> depicts a flowchart for example steps for reconstructing the first and second 2D images into a 3D scene of the scene object (step <NUM>; <FIG>). At step <NUM> create a depth map from the first and second 2D images. The processing system may create a depth map by processing the first and second 2D images using a stereo-vision processing technique. At step <NUM> create a 3D scene from the first and second 2D images. The processing system may create a 3D scene by geometrically processing the first and second 2D images along with the depth map created in step <NUM>.

At step <NUM> detect regions of the 3D scene with incomplete image information. The processing system may detect regions of the 3D scene with incomplete image information (e.g., missing color, depth, and/or object information). For example, the processing system may determine incomplete information by inspecting the shape of the faces making up the 3D scene and/or information such as confidence values associated with the vertices making up the faces.

The processing system processes faces of the reconstructed 3D scene (step 323a; <FIG>) to identify groups of contiguous faces exhibiting characteristics of incomplete information, e.g., very narrow faces and/or faces with one or more vertex having a low confidence value.

The processing system may identify contiguous regions having degenerated faces (e.g., relatively narrow faces) as regions with incomplete information (step 323b; <FIG>). Faces that are relatively narrow may be determined by comparing the angles between adjacent lines of the faces to a threshold value, e.g., faces including at least one angle that is less than a threshold such as <NUM> degrees, faces having one side with a length that is less than <NUM>% the length of another side, and/or faces having one side that is below a threshold dimension, such a <NUM> millimeter, may categorized as narrow.

The processing system may also, or alternatively, identity faces having a low confidence values (e.g., faces with at least one vertex having a confidence value below a threshold value) as regions with incomplete information (step 323c; <FIG>). The confidence value of a vertex may be the confidence value of a corresponding pixel determined during stereo processing of the first and second 2D images to create the depth map (described above).

The confidence value for a vertex corresponding to a pixel depends on the matching/correlation between the first and second 2D images in creating the pixel. If there is a high correlation (e.g., <NUM>% or above) there is a relatively high likelihood that the vertex includes accurate information that is useful for reconstruction into the 3D scene. On the other hand, if there is low correlation (e.g., below <NUM>%) there is a relatively high likelihood that the vertex does not include information that is accurate enough to be useful for reconstruction into the 3D scene.

At step <NUM> reconstruct the detected region. In an example, the processing system reconstructs the detected region of the 3D scene, e.g., using geometric processing such as described above for step <NUM>. In reconstructing the detected region, the processing system may ignore vertices having low confidence values and/or associated with degenerated faces. This results is fewer, if any, degenerated faces (e.g., relatively narrow faces). Thus, the faces in the detected region will have a different shape after this reconstruction step. The faces in the detected region may be removed prior to reconstruction. In an example, a data structure such as Indexed-Face-Set for 3D meshes may be used that includes an ordered list of all vertices (and their attributes, e.g., color, texture, etc.) and a list of faces, where each face refers to the vertex index in the vertices list. In this example, a face may be removed by removing it from the faces list.

At step <NUM> determine replacement image information and modify the reconstructed detected regions. The processing system may determine the replacement image information for the reconstructed detected regions and modify the 3D scene to include the replacement image information in the detected regions as they are being reconstructed. For example, the processing system may determine replacement image information for each of the detected regions by blending boundary information from the respective boundaries of each of the detected regions.

In an example, to determine replacement image information, the processing system identifies a boundary surrounding each detected region (step 325a; <FIG>). The processing system then identifies background information in the detected regions (step 325b; <FIG>); e.g., based on depth information associated with the vertices in along the boundary. The processing system also identifies foreground information in the detected regions (step 325c; <FIG>); e.g., also based on depth information. To identify background/foreground information, the processing system may compare the depth information of each vertex to a threshold value (e.g., an average value of the depth information from all vertices), identify information associated with a vertex having a depth greater than the threshold value as background information, and identify information associated with a vertex having a depth less than the threshold value as foreground information. The processing system then blends information from the boundaries through the respective regions (step 325d; <FIG>) giving the background information higher weight than the foreground information. This results in a diffusion of information primarily from the background to the foreground.

Referring back to <FIG>, at step <NUM> render the 3D scene from multiple viewpoints. The processing system may render the 3D scene from the multiple viewpoints, e.g., by applying an image synthesis technique to the 3D scene to create a 2D image from each view point. Suitable image synthesis techniques will be understood by one of skill in the art from the description herein.

At step <NUM> refine the rendered 3D scene. The processing system may refine the 3D scene from each of the multiple viewpoints. In an example, the processing system identified regions in the 2D images of the rendered 3D scene where there are gaps in image information (i.e., "holes'). The processing system then fills in these holes using replacement image information surrounding these holes. The processing system may fill in the holes giving preferential weight to background information surrounding the holes.

At step <NUM> present the rendered 3D scene. The processing system may present the rendered 3D scene on a display of the eyewear or a remote computing device by selectively presenting the 2D image within the rendered 3d scene associated with a selected point of view (e.g., based on a user input to the eyewear device or the remote computing device).

By performing the above process described with reference to flow chart <NUM>, a more aesthetically pleasing 3D scene viewable from more viewpoints (e.g., reduction is color stretching) is obtainable from just two 2D images without having to resort to panoramic views. Thus, superior results can be achieved without resorting to computationally intensive techniques.

It is to be understood that the steps of the processes described herein may be performed by a hardware processor upon loading and executing software code or instructions which are tangibly stored on a tangible computer readable medium, such as on a magnetic medium, a computer hard drive, an optical disc, solid-state memory, flash memory, or other storage media known in the art. Thus, any of the functionality performed by the processor described herein may be implemented in software code or instructions which are tangibly stored on a tangible computer readable medium. Upon loading and executing such software code or instructions by the processor, the processor may perform any of the functionalities described herein, including any steps of the methods described herein.

Although an overview of the inventive subject matter has been described with reference to specific examples, various modifications and changes may be made to these examples without departing from the broader scope of examples of the present disclosure. For example, although the description focuses on an eyewear device, other electronic devices such as headphones are considered within the scope of the inventive subject matter. Such examples of the inventive subject matter may be referred to herein, individually or collectively, by the term "invention" merely for convenience and without intending to voluntarily limit the scope of this application to any single disclosure or inventive concept if more than one is, in fact, disclosed.

Claim 1:
A system for creating a three-dimensional (3D) scene, the system comprising:
eyewear (<NUM>) including a first imager (10A) and a second imager (10B) spaced from the first imager (10A), the first imager (10A) configured to obtain a first two-dimensional (2D) image of a scene object from a first viewpoint and the second imager (10B) configured to obtain a second 2D image of the scene object from a second viewpoint that is different than the first viewpoint;
a processing system coupled to the eyewear (<NUM>), the processing system configured to:
obtain (<NUM>) the first 2D image and the second 2D image;
create (<NUM>) a depth map from the first and second 2D images;
create (<NUM>) a 3D scene from the depth map and the first and second 2D images;
detect (<NUM>) regions of the initial 3D scene with incomplete image information;
reconstruct (<NUM>) the detected regions of the 3D scene;
determine (<NUM>) replacement information and modify the reconstructed regions; and
render (<NUM>) the 3D scene with the modified reconstructed regions from
a plurality of viewpoints;
wherein the depth map includes pixel vertices and confidence values corresponding to each vertex, to create (<NUM>) the 3D scene the processing system is configured to connect the vertices to form faces, and wherein to detect (<NUM>) the regions of the 3D scene with incomplete information the processing system is configured to identify contiguous faces (323b) including a least one of degenerated faces or low confidence faces (323c);
wherein the degenerated faces each have at least one angle that is less than a threshold value; and
wherein each low confidence face includes at least one vertex generated with inconsistent values between the first 2D image and the second 2D image.