Patent Description:
Increasing resolutions of cameras has increased proliferation of three-dimensional vision systems. In a three-dimensional vision system, images of an object are captured by one or more cameras. The captured images are provided to a computing device, which analyzes the images to generate a three-dimensional graphical reconstruction of the object.

However, combining images of different views of an object to generate the three-dimensional graphical reconstruction of the object is computationally intensive. This use of computational resources increases the time to generate the graphical reconstruction of the object. Increased time to generate the graphical reconstruction of the object limits potential use of three-dimensional vision systems to implementations that are tolerant of delays from image capture to generation of graphical reconstructions of objects.

The journal article <NPL>, is useful for understanding the present invention. It discloses how to reconstruct both the shape and reflectance properties of surfaces from multiple images. The disclosure argues that an object-centered representation is most appropriate for this purpose because it naturally accommodates multiple sources of data, multiple images (including motion sequences of a rigid object), and self-occlusions.

To capture image or video data of an object to be reproduced in a virtual reality (VR) environment or in an augmented reality (AR) environment, multiple cameras are positioned to focus on a target position where the object is positioned. The cameras have specific locations relative to each other and relative to the target position. This allows each camera to capture a different image of the object positioned at the target position. Each camera is coupled to a console that receives images of the object positioned at the target position from each camera. The console processes the received images to generate a graphical representation of the object positioned at the target position.

To generate the graphical representation of the object positioned at the target position, the console processes images from each camera in parallel to determine depth information. For example, different processors apply a patch-match process on a graphics processing unit (GPU) to images captured by different cameras at a common time. To improve the convergence and also reduce the computation time, a coarse-to-fine patch-match process is applied to images captured by different cameras at a common time; in various embodiments, the patch-match process is first applied to a coarse resolution of images captured by different cameras at the common time, and fine resolution of the images captured at the common time is initialized to run the patch-match process with fewer iterations. To improve the accuracy of the determined depth information, the console modifies the depth information determined for images received from each camera. In various embodiments, the console modifies depth information determined for various images by optimizing an energy function based on intensities from stereoscopic information and intensities from shading information for various images. For example, for images received from multiple cameras (e.g., from each camera) at a common time, the console determines global intensities for portions of the images corresponding to a common depth, based on the depth information determined for different images. For example, the console determines global intensities for portions of images received from different cameras as an average intensity for portions of images received from different cameras at a common time that have common depth information or having depth information within a threshold amount of each other. Additionally, the console determines intensities for different portions of an image received from a camera received at the common time based on the computed shading information from the image. The console generates a total energy for the image received from the camera at the common time by combining the global intensities of different portions of the images received from different cameras at the common time with intensities determined from shading information for different portions of the image received from the camera at the common time; in various embodiments, the console sums the global intensities of different portions of the images received from different cameras at the common time with the intensities determined from shading information for different portions of the image received from the camera at the common time. In various embodiments, the console sums the intensity differences between the global intensities of different portions of the images received from different cameras at the common time and the intensities computed from the shading information from the corresponding images. The console may also combine a regularization value with the depth estimation for portions of the image with depth estimation of the corresponding neighboring portions of the image received from the camera; the regularization value accounts for similarities between depth estimation of portions of the image that are adjacent to each other.

For each of the images received from different cameras at a common time, the console modifies the depth information determined for the images by minimizing the total energy for each image received at the common time. The console minimizes the total energy for each image received at the common time in parallel using any suitable process or processes. For example, the console applies a Gauss-Newton method using a graphics processing unit (GPU) to an image received from each camera at the common time to minimize the total energy for each image received at the common time, which modifies depth information determined for images received from various cameras. The console combines modified depth information from multiple images to generate a reconstruction of the object positioned at the target position.

The figures depict examples of the invention for purposes of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles, or benefits touted, of the disclosure described herein.

<FIG> is a block diagram of a system environment <NUM> comprising multiple cameras <NUM>10A-F configured to capture images of an object positioned at a target position <NUM>. Each of the cameras 110A-F is coupled to a console <NUM>. Any suitable number of cameras 110A-F may be included in the system environment <NUM>. Further, different and/or additional components may be included in the system environment <NUM>.

Each camera 110A-F captures images of an object positioned at the target position <NUM>. Accordingly, each camera 110A-F is configured to have a focal point of, or within, the target position <NUM>. Additionally, each camera <NUM>10A-F has a specific position relative to each other camera 110A-F of the system environment <NUM>. For example, camera 110A has a specific position relative to camera 110B as well as a specific position relative to camera 110D. The different positions of the cameras 110A-F relative to each other causes each camera <NUM>10A-F to capture images of different portions of the object positioned at the target position <NUM>. In various embodiments, the cameras <NUM>10A-F are positioned relative to each other so each camera <NUM>10A-F still has overlapping fields of view including different portions of the object positioned at the target position <NUM>.

Each camera 110A-F captures images based on light having different wavelengths reflected by the object positioned at the target position <NUM>. Each camera 110A-F may capture images of light within a common wavelength range reflected by the object positioned at the target position <NUM>. For example, each camera <NUM>10A-F captures infrared light reflected by the object positioned at the target position <NUM>. As another example, each camera <NUM>10A-F captures visible light reflected by the object positioned at the target position. Alternatively, different cameras 110A-F capture light with different ranges of wavelengths reflected by the object positioned at the target position <NUM>. For example, cameras 110A, 110C, 110E capture infrared light reflected by the object positioned at the target position <NUM>, while cameras 110B, 110D, 110F capture visible light reflected by the object positioned at the target position <NUM>. Each camera 110A-F has parameters such as focal length, focus, frame rate, ISO, sensor temperature, shutter speed, aperture, resolution, etc.; one or more parameters of a camera <NUM>10A-F may be modified. In some examples, the cameras 110A-F have a high frame rate and high resolution. The cameras 110A-F capture two-dimensional images of the object positioned at the target position <NUM> in various examples.

One or more illumination sources can be positioned relative to the cameras 110A-F and the target position <NUM>. The one or more illumination sources are positioned to illuminate the target position <NUM>, allowing illumination of an object positioned at the target position. Illumination sources may be positioned at discrete locations relative to one or more of the cameras 110A-F. Alternatively, illumination sources are coupled to one or more of the cameras 110A-F. Example illumination sources include light-emitting diodes (LEDs) that emit light in the visible band (i.e., ~<NUM> to <NUM>), in the infrared (IR) band (i.e., ~<NUM> to <NUM>), in the ultraviolet band (i.e., <NUM> to <NUM>), in some other portion of the electromagnetic spectrum, or in some combination thereof. Different illumination sources may have different characteristics. As an example, different illumination sources emit light having different wavelengths or different temporal coherences describing correlation between light waves at different points in time. Further, light emitted by different illumination sources may be modulated at different frequencies or amplitudes (i.e., varying intensity) or multiplexed in a time domain or in a frequency domain.

The one or more illumination sources are coupled to the console <NUM>, which provides control signals to the one or more illumination sources. For example, the console <NUM> provides an illumination source with a control signal that modifies an intensity of light emitted by the illumination sources. As another example, the console <NUM> provides an illumination source with a control signal that modifies a direction in which the illumination source emits light or modifies focusing of light emitted by the illumination source.

The console <NUM> is a computing device coupled to each of the cameras <NUM>10A-F and configured to receive images captured by one or more of the cameras <NUM>10A-F. Additionally, the console <NUM> is configured to transmit control signals to one or more cameras 110A-F that modify one or more parameters of a camera. For example, a control signal provided to a camera 110A from the console <NUM> modifies a focal point of the camera 110A or modifies a zoom of the camera 110A.

Additionally, the console <NUM> receives images captured from multiple cameras 110A-F and generates a reconstruction of an object positioned at the target position <NUM> and included in images received from multiple cameras <NUM>10A-F. When generating the reconstruction of the object <NUM>, the console <NUM> processes images received from multiple cameras <NUM>10A-F in parallel. As further described below in conjunction with <FIG> and <FIG>, the console <NUM> determines depth information from various images based on correspondences between regions within from images captured by different cameras 110A-F in parallel and determines shading information in images captured by different cameras 110A-F in parallel. Using the shading information, the console <NUM> refines depth information determined from the correspondences in parallel. When refining depth information determined from an image, the console <NUM> optimizes a total energy of the image, with the energy based on intensities determined from shading information of portions of images captured by various cameras <NUM>10A-F having a common depth and intensities of portions of the images captured by various cameras <NUM>10A-F determined from shading information. In various embodiments, the console <NUM> refines depth information from an image by minimizing a total energy of the image based on the intensities of portions of images captured by various cameras 110A-F having a common depth and intensities of portions of the images captured by various cameras 110A-F determined from shading information, as further described below in conjunction with <FIG> and <FIG>. The console <NUM> combines refined depth information from multiple images to generate a reconstruction of the object positioned at the target position <NUM>.

<FIG> is a flowchart of one method for generating a representation of an object from images of the object captured by multiple cameras 110A-F in different locations relative to each other. The method may include different or additional steps than those described in conjunction with <FIG>. Additionally, the method may perform steps in different orders than the order described in conjunction with <FIG>.

Multiple cameras <NUM>10A-F are positioned at locations relative to each other and positioned to capture images of an object positioned at a target position <NUM>. This allows different cameras 110A-F to capture images of different portions of the object positioned at the target position <NUM>. As further described in conjunction with <FIG>, each camera 110A-F has one or more parameters that affect capturing of images by the cameras 110A-F.

Each of at least a set of the multiple cameras 110A-F capture <NUM> images of an object positioned at the target position <NUM>. As further described above in conjunction with <FIG>, the positioning of different cameras 110A-F relative to each other causes different cameras <NUM>10A-F to capture <NUM> images of different portions of the object positioned at the target position <NUM>. Each of the multiple cameras <NUM>10A-F captures <NUM> one or more images of the object positioned at the target position <NUM>.

The console <NUM> receives images of the object positioned at the target position <NUM> from various cameras 110A-F and determines <NUM> depth information of portions of each received image. The console <NUM> determines <NUM> depth information for images received <NUM> from each camera 110A-F in parallel. For example, the console <NUM> includes multiple processors, such as graphics processing unit, and each processor determines <NUM> depth information for an image received <NUM> from a camera 110A-F. For example, each processor included in the console <NUM> corresponds to a camera 110A-F, so a processor determines <NUM> depth information for an image received <NUM> from a camera 110A-F corresponding to the processor. The console <NUM> receives images from multiple cameras 110A-F at a common time, and the console <NUM> determines <NUM> depth information for images received from multiple cameras <NUM>10A-F at the common time in parallel. Determining depth information for images received from multiple cameras <NUM>10A-F in parallel allows the console <NUM> to more quickly determine depth information for multiple images. The console <NUM> may initialize a random depth for each pixel in an image received from each camera 110A-F at a common time and defines nearest neighbor fields of pixels that are within a threshold distance of particular pixels in an image received from a camera <NUM>10A-F. A depth of a pixel in an image received from a camera 110A-F determined to have at least a threshold accuracy is subsequently propagated to adjacent pixels. Subsequently, candidate depths are evaluated, and the depth of the pixel is modified to a candidate depth resulting in an increased accuracy. The preceding steps are iteratively applied to an image received from a camera 110A-F in various embodiments. For example, the console iteratively performs the preceding steps a set number of times (e.g., <NUM> times) for each image. In various examples, the console <NUM> applies a coarse to fine patch-match process to images received from various cameras 110A-F to determine <NUM> depth information corresponding to different pixels in each image. Applying the patch-match process to images captured by different cameras 110A-F in parallel allows the console <NUM> to more efficiently obtain depth information from multiple images.

To improve the accuracy of the determined depth information, the console <NUM> modifies <NUM> the depth information determined <NUM> for images received <NUM> from each camera 110A-F. The console <NUM> may modify <NUM> depth information determined <NUM> for various images by optimizing an energy function based on intensities from stereoscopic information and intensities from shading information for various images. For example, for images received from multiple cameras 110A-F (e.g., from each camera 110A-F) at a common time, the console <NUM> determines global intensities for portions of the images corresponding to a common depth, based on the depth information determined <NUM> for different images. For example, the console <NUM> determines global intensities for portions of images received from different cameras <NUM>10A-F as an average intensity for portions of images received from different cameras 110A-F at a common time that have common depth information or having depth information within a threshold amount of each other. Additionally, the console <NUM> determines intensities for different portions of an image received from a camera <NUM>10A-F received at the common time based on shading information from the image. The console <NUM> sums the intensity differences between the global intensities of different portions of the images received from different cameras <NUM>10A-F at the common time and the intensities computed from the shading information from different cameras 110A-F at the common time. The console <NUM> may also combine a regularization value with a depth estimation for portions of the image with depth estimation of the corresponding neighboring portions of the image received from the camera; the regularization value accounts for similarities between depth estimation of portions of the image that are adjacent to each other.

For each of the images received from different cameras <NUM>10A-F at a common time, the console <NUM> modifies <NUM> the depth information determined <NUM> for the images by minimizing the total energy for each image received at the common time. The console <NUM> minimizes the energy total energy for each image received at the common time in parallel using any suitable process or processes. For example, the console <NUM> applies a Gauss-Newton method via a graphics processing unit (GPU) to an image received from each camera 110A-F at the common time to minimize the total energy for each image received at the common time, which modifies <NUM> depth information determined <NUM> for images received from various cameras 110A-F. However, in other examples, the console <NUM> may apply any suitable process or processes to the total energy determined for images received from different cameras 110A-F at the common time that minimizes a total energy for each received image to modify <NUM> depth information determined <NUM> for each of the images received at the common time.

From the modified depth information for images received from multiple cameras <NUM>10A-F at the common time, the console <NUM> generates <NUM> a representation of the object positioned at the target position at the common time. The console <NUM> applies any suitable method or methods to the modified depth information for each image received at the common time that combines the modified depth information in each image into a single-three dimensional representation of the object at the common time. For example, the console <NUM> performs a Poisson reconstruction from the modified depth information for each image received at the common time to generate <NUM> the representation of the object at the common time. The console <NUM> may subsequently present the representation of the object via a display device, store the representation of the object, transmit the representation of the object to a client device for presentation, or perform any other suitable interaction with the representation of the object. The console <NUM> generates <NUM> multiple representations of the object based on modified depth information for images received from multiple cameras 110A-F at different common times and maintains a series of representations of the object. For example, the series of representations correspond to an appearance of the object during a time interval, allowing the series of representations to depict changes in appearance of the object during the time interval or to depict movement of the object during the time interval. The series of representations of the object may be subsequently displayed via a display device, stored by the console <NUM>, or transmitted from the console <NUM> to a client device. However, the console <NUM> may perform any suitable action regarding the series of representations of the object.

<FIG> is a process flow diagram of generating a representation of an object from images of the object captured by multiple cameras <NUM>10A-F in different locations relative to each other. As further described above in conjunction with <FIG> and <FIG>, multiple cameras 110A-F capture images 305A-F of an object positioned at a target location <NUM>. Different cameras 110A-F capture images 305A-F of different portions of the object positioned at the target position <NUM>. A console <NUM> receives the images 305A-F from multiple cameras 110A-F and determines <NUM> depth information from each image 305A-F in parallel. As further described above in conjunction with <FIG>, the console <NUM> may perform any suitable method or methods to determine <NUM> depth information from an image 305A-F. The console <NUM> includes multiple processors (e.g., graphics processing units), and each processor determines <NUM> depth information from a different received image in parallel. For example, each image 305A-F includes a timestamp, and the console identifies images 305A-F having a common timestamp; different images 305A-F having the common timestamp are processed by different processors of the console <NUM> in parallel to determine <NUM> depth information for different images having the common timestamp in parallel.

The console <NUM> also modifies <NUM> the depth information determined <NUM> from each image 305A-F in parallel. The console <NUM> optionally maintains an energy function that determines an energy of each image 305A-F based on intensity information of portions of an image 305A-F with a particular timestamp and intensity information of portions of other images 305A-F having the particular timestamp. An example of modifying <NUM> depth information determined <NUM> from each image 305A-F is further described above in conjunction with <FIG>. Different processors included in the console <NUM> modify <NUM> the depth information determined <NUM> from each image 305A-F in parallel; for example, each processor modifies <NUM> depth information determined <NUM> from a different image 305A-F. By determining <NUM> and modifying <NUM> depth information from multiple images 305A-F in parallel, the console <NUM> more quickly and efficiently obtains and refines depth information from various images 305A-F.

Using the modified depth information from multiple images 305A-F, the console <NUM> generates <NUM> a representation of the object included in the captured images 305A-F. The representation may be a three-dimensional representation of the object included in the images 305A-F generated <NUM> by combining the images 305A-F and modified depth information for the images 305A-F. As the depth information is determined and modified in parallel for multiple images, overall time for the console <NUM> to generate <NUM> the representation of the object is decreased, while the parallel modification of depth information for multiple images increase accuracy of the representation.

The foregoing description of examples of the invention has been presented for the purpose of illustration; it is not intended to be exhaustive or to limit the patent rights to the precise forms disclosed. Persons skilled in the relevant art can appreciate that many modifications and variations are possible in light of the above disclosure.

Examples of the invention may include or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some manner before presentation to a user, which may include, e.g., a virtual reality (VR), an augmented reality (AR), a mixed reality (MR), a hybrid reality, or some combination and/or derivatives thereof. Artificial reality content may include completely generated content or generated content combined with captured (e.g., real-world) content. The artificial reality content may include video, audio, haptic feedback, or some combination thereof, and any of which may be presented in a single channel or in multiple channels (such as stereo video that produces a three-dimensional effect to the viewer). Additionally, in some examples, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, that are used to, e.g., create content in an artificial reality and/or are otherwise used in (e.g., perform activities in) an artificial reality. The artificial reality system that provides the artificial reality content may be implemented on various platforms, including a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

Some portions of this description describe examples of the invention in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to convey the substance of their work effectively to others skilled in the art. These operations, while described functionally, computationally, or logically, are understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Furthermore, it has also proven convenient at times, to refer to these arrangements of operations as modules, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combinations thereof.

In one example, a software module is implemented with a computer program product comprising a computer-readable medium containing computer program code, which can be executed by a computer processor for performing any or all of the steps, operations, or processes described.

Examples of the invention may also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, and/or it may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory, tangible computer readable storage medium, or any type of media suitable for storing electronic instructions, which may be coupled to a computer system bus. Furthermore, any computing systems referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.

Examples of the invention may also relate to a product that is produced by a computing process described herein. Such a product may comprise information resulting from a computing process, where the information is stored on a non-transitory, tangible computer readable storage medium and may include any embodiment of a computer program product or other data combination described herein.

Claim 1:
A system comprising:
a plurality of cameras (110A-F) each having a specific position relative to each other and configured to capture images (305A-F) of an object positioned at a target position (<NUM>); and
a console (<NUM>) coupled to each of the plurality of cameras (110A-F), the console (<NUM>) configured to:
receive (<NUM>) one or more images (305A-F) of the object from each of the plurality of cameras (110A-F);
determine (<NUM>, <NUM>) depth information for images (305A-F) received from each of the plurality of cameras (110A-F) in parallel;
modify (<NUM>, <NUM>) depth information determined for the images (305A-F) received from each of the plurality of cameras (110A-F) at a common time in parallel based on intensities of portions of each image (305A-F) having a common depth and intensities of portions of each image (305A-F) determined from shading information from each image (305A-F); and
generate (<NUM>, <NUM>) a reconstruction of the object by combining the modified depth information for the images (305A-F) received from each of the plurality of cameras (110A-F);
wherein modify (<NUM>, <NUM>) depth information determined for the images (305A-F) received from each of the plurality of cameras (110A-F) at a common time in parallel based on intensities of portions of each image (305A-F) having a common depth and intensities of portions of each image determined from shading information from each image (305A-F) comprises:
identify images (305A-F) received from multiple cameras (110A-F) at the common time;
determine global intensities for portions of the identified images (305A-F) having common depth information, wherein a global intensity is an average intensity for portions of images (305A-F) received from different cameras (110A-F) at a common time that have common depth information or having depth information within a predetermined threshold amount of each other;
determine intensities for portions of an identified image (305A-F) having different depths based on shading information of the identified image (305A-F);
generate an energy of the identified image (305A-F) by combining said global intensities with said intensities based on shading information; and
minimize the energy for the identified image (305A-F) to modify depths of portions of the identified image (305A-F).