Patent Description:
Virtual reality (VR) video is an important application of VR technology which can bring an immersive experience of videos. Generally, VR video technology can be divided into <NUM>-DoF (degree of freedom) and <NUM>-DoF. <NUM>-DoF technology only allows the user to freely rotate his head to watch in different orientations from a fixed location in the video scene. However, <NUM>-DoF technology allows user to select different orientations and also allows user to freely select his positions in scene.

<NUM>-DoF video shooting only requires a minimal number of cameras facing different directions of the scene. Meanwhile, <NUM>-DoF video shooting requires relatively large camera arrays.

In practice, for <NUM>-DoF VR video shooting, there are a large number of cameras facing the scene, where every two neighboring cameras create a camera pair which are used to generate the depth images. The distance between a pair of cameras is called the baseline. The depth images and the images captured by the cameras can be used to render an image of a virtual view.

In a VR video shooting system, all the real-time images are transmitted to a workstation and the workstation runs a depth estimation algorithm to generate the depth images. Thus, a camera array with a large number of cameras requires a large amount of computation resources. For example, for a football match, more than <NUM> cameras might be needed, and thus one workstation may not be enough to compute all of the depth data for the football match. There is clearly a need to improve the computational efficiency for the generation of 6DoF video.

<CIT> discloses techniques related to 3D image capture with dynamic cameras.

According to examples in accordance with an aspect of the invention, there is provided a method for generating depth data for a six degree of freedom, 6DoF, video of a scene according to claim <NUM>.

Obtaining 6DoF video of a dynamic scene can be difficult as the depth of the objects in a dynamic scene may change at any time. Because of this, a large set of cameras has to be used such that, if an object gets close the cameras, depth data can still be obtained for the close object as, for example, it will still be in the field of view of at least two cameras.

However, using a large number of cameras also means obtaining (and possibly transmitting) a large amount of data. It is likely that most of the time during the 6DoF video, not all cameras in the large set are required to obtain depth data for the 6DoF video. This is particularly true if the objects are not close to the cameras.

Thus, the inventors propose analyzing a first set of depth components in a frame of the 6DoF video and, based on the analysis determining the first set of depth components is complete, generating a second set of depth components with a smaller number of depth components than the first set. Methods of analysis will be discussed below.

The completeness of the depth components refers to how much of the scene is captured in the depth components. If all of the depth infromation of the scene is captured in the depth components, a 6DoF frame can be accurately rendered because there is enough depth information of the scene to enable accurate warping of the images.

Determining the first set of depth components to be complete may be defined by the first set of depth components having enough depth information of the scene to render a frame of the 6DoF without any missing depth information of the scene in the frame. Similarly, determining the first set of depth components to be overcomplete is defined by the first set of components having more than enough depth information of the scene. In other words, an overcomplete first set of depth components has redundant or duplicated depth information.

Assessing completeness may be performed by assessing continuity of the depth components, identifying gaps or occlusion in the depth components, identifying missing depth information, identifying artefacts etc..

The first set of images and the first set of depth components may be used to render a first frame of the 6DoF video. The second set of images and the second set of depth components may be used to render a second frame of the 6DoF video, wherein the second frame is subsequent to the first frame.

The images are obtained from a group of cameras. A group of cameras is selected by selecting a subset of cameras from all the available cameras imaging the scene. A group of cameras may comprise one or more pairs of cameras thereby to enable the generation of depth components from the images obtained by both cameras in a camera pair. A camera pair may be formed by selecting a camera and the closest available camera in the group of camera pairs.

A depth component may comprise a depth map (e.g. generated from a pair of images) or any other type of depth information for the scene (e.g. 3D mesh, point cloud etc.). In other words, a depth component could be any form of depth information of the scene comprising a plurality of depth values.

For example, in a depth map, each depth value may correspond to a pixel of the depth map. In a 3D mesh, each depth value may correspond to a vertex, edge and/or face of the 3D mesh. In a point cloud, each depth value may correspond to a point in the point cloud.

It is not necessary for a camera pair to be comprised of two neighboring cameras. In fact, it may be preferable to have camera pairs comprising of two cameras far from each other as, the larger the baseline (i.e. distance between the cameras) of two cameras is, the higher the accuracy of the depth image is. Thus with a fixed number of selected cameras, the accumulation (i.e. sum) of the baselines of the selected cameras may be maximized when meeting the requirement that each pixel can be seen at least <NUM> cameras.

For example, the selected cameras from left to right may be numbered as <NUM>,<NUM>,<NUM>,<NUM>. If the camera pairs are <NUM>-<NUM>, <NUM>-<NUM> and <NUM>-<NUM>, then the accumulation of the baselines is larger than that of the camera pairs <NUM>-<NUM>, <NUM>-<NUM> and <NUM>-<NUM> and thus the accuracy of the depth components may be greater.

Analyzing the first set of depth components may comprise determining an indication of how close one or more of the objects in the scene are to a first group of cameras used to obtain the first set of images.

The inventors propose determining an indication of proximity (or indication of closeness) for the objects in the scene. The indication indicates how close one or more of the objects are to the cameras which are currently imaging the scene (i.e. the first group). If the object is closer to the cameras than the distance for which the current first group of cameras can obtain depth data, a second group of camera pairs is selected to obtain the next frame of the 6DoF video, where the second group of cameras is capable of obtaining depth data for the closest object to the cameras.

The indication is an estimate of how physically close an object is to the first group of cameras. The indication may comprise, for example, a single value of depth or a single non-numerical indication (i.e. close, far etc.). Alternatively, the indication may comprise information on depth of the closest object and information on where the closest object is in the scene.

The indication may also comprise information on the depth of one or more of the objects and, in some embodiments, of the positions of one or more of the objects.

A depth component may comprise a depth map generated by performing depth estimation on at least two images.

A depth component may comprise a depth map/depth image, wherein each depth map is generated from images obtained from a camera pair. A depth map may be obtained by performing depth estimation (e.g. via depth disparity) on two texture images obtained from the camera pair (i.e. an image from each camera in a camera pair).

In an embodiment, the indicator of proximity may be based on the pixel of a depth map with the lowest depth value.

In the context of the invention, the terms depth map and depth image are equivalent and interchangeable.

The method may further comprise determining whether a camera-facing surface of any one of the objects in the scene is not visible in the field of view of at least two cameras in the first group of camera, wherein generating a second group of depth components is further based on the determination.

A camera-facing surface of an object may be a surface of the object which is imaged by at least one camera in a camera group (i.e. visible in the field of view (FOV) of at least one camera). Ensuring all camera-facing surfaces of the objects are imaged by at least two cameras (i.e. within the FOV of at least two cameras) will enable the depth components to be generated with more accuracy.

For example, if it is determined that one or more camera-facing surfaces are not visible in the depth components, the second group of cameras may be selected to comprise all of the cameras available to image the scene. Alternatively, one or more cameras can be identified which can image the camera-facing surfaces not currently visible in the FOV of at least two cameras.

Analyzing the first set of depth components may further comprise determining whether any object in the scene is at least partly occluded in the first set of depth components.

Analyzing the first set of depth components to determine object occlusion may comprise identifying where the occlusion is and which cameras could be used to obtain subsequent images for the next (second) frame to avoid the occlusion in the generated subsequent depth components.

For example, if object occlusion is determined in the depth components, the second group of cameras may be selected to comprise all of the cameras available. Alternatively, one or more cameras can be identified to remedy the occlusion, wherein the identified cameras are included in the second group.

Analyzing the first set of depth components may further comprise determining whether the first set of depth components has any visual artifacts and/or depth artifacts.

Visual artifacts and depth artifacts may occur due to a lack of cameras imaging a particular part of the scene, object in the scene or part of an object. Artifacts may also occur due to faulty cameras (i.e. a camera not imaging the scene as intended due to, for example, the lens being foggy or the sensor in the camera malfunctioning), thus, different or additional cameras may be needed in the second group.

Analyzing the depth components to determine whether they have any visual artifacts and/or depth artifacts may further comprise identifying where the artifact is in the depth components and which cameras could be used to obtain subsequent second set of images for the next (second) frame to remedy the artifact(s) in the subsequent depth components.

Depth estimation usually uses a feature match algorithm between two camera images. If the feature match algorithm fails to match a pixel in the first image with a pixel in the second image, the algorithm outputs a large matching error. Thus, artefacts may be identified using the matching errors output by the feature match algorithm during depth estimation. Additionally, the region(s) of the depth map with the artefact(s) may be identified by checking which pixels in the two images have a large matching error.

The method may further comprise selecting a group of cameras configured to obtain the second set of images based on the analysis of the first set of depth components.

The inventors propose analyzing the first set of depth components and, based on the analysis, selecting which cameras to use to obtain a second set of images. The second set of images may have a lower number of images than the first set. The second set of images is used to generate the second set of depth components.

The group of cameras can be selected in order to reduce the number of images in the second set of images and thus also reduce the number of depth components in the second set of depth components.

Selecting the group of cameras may comprise selecting camera pairs such that every object in the scene visible to at least one camera is included in the field of view of at least two cameras in the group of cameras. The cameras may be selected such that depth can be estimated for every (visible) part of every object.

Selecting a group of cameras may comprise selecting a pre-defined group of cameras from a set of pre-defined groups of cameras, wherein each pre-defined group of cameras is associated with a minimum distance from the pre-defined group of cameras at which an object in the scene is guaranteed to be within the field of view of at least two cameras in the pre-defined group of cameras.

From the knowledge of the distance(s) between cameras and the field of view(s) of the cameras in a pre-defined group of cameras, it is possible to derive the closest/minimum distance an object can be to a pre-defined group before the pre-defined group cannot properly obtain depth information for the object. In other words, a minimum distance can be derived for each pre-defined group of cameras based on the distance between the cameras and the field of views of the cameras in the pre-defined group.

Thus, each pre-defined group of cameras corresponds to a minimum distance (from the pre-defined group) at which an object is guaranteed to be within the field of view of two cameras (in the pre-defined group) such that depth information can be obtained for the object.

If the analysis determines the first set of depth components to be undercomplete, the second set of depth components has a larger number of depth components than the first set of depth components.

Determining the first set of depth components to be undercomplete is defined by the first set of depth components not having enough depth information of the scene to render a frame of the 6DoF. In other words, an undercomplete first set of depth components has missing depth information of the scene in the frame.

Determining the first set of depth components to be overcomplete may be based on the indication of closeness indicating that the closest object in the scene is further from the first group of cameras than a minimum distance at which an object in the scene is guaranteed to be within the field of view of at least two cameras in a second group of cameras, wherein the second group of cameras comprises less cameras than the first group of cameras.

Determining the first set of depth components to be undercomplete may be based on the indication of closeness indicating that the closest object in the scene is closer to the first group of cameras than a minimum distance at which an object in the scene is guaranteed to be within the field of view of at least two cameras in the first group of cameras.

The invention also provides a computer program product according to claim <NUM>.

The invention also provides a system for generating depth data for a six degree of freedom, 6DoF, video of a scene according to claim <NUM>.

The processor may be configured to analyze the first set of depth components by determining an indication of how close one or more of the objects in the scene are to the first group of cameras.

The processor may be further configured to determine whether a camera-facing surface of any one of the objects in the scene is not visible in the field of view of at least two cameras in the first group of camera and wherein the processor is further configured to generate a second group of depth components based on the determination.

The processor may be further configured to select a group of cameras configured to obtain the second set of images based on the analysis of the first set of depth components.

The processor may be configured to select a group of cameras by selecting a pre-defined group of cameras from a set of pre-defined groups of cameras, wherein each pre-defined group of cameras is associated with a minimum distance from the pre-defined group of cameras at which an object in the scene is guaranteed to be within the field of view of at least two cameras in the pre-defined group of cameras.

The system may further comprise a group of cameras comprising a plurality of cameras.

The invention provides a method for generating depth data for a six degree of freedom, 6DoF, video of a scene. The method comprises obtaining a first set of images of the scene, generating a first set of depth components based on the first set of images and analyzing the first set of depth components to determine completeness of the depth components. A second set of images of the scene are further obtained and a second set of depth components are generated based on the second set of images, wherein, if the analysis determines the first set of depth components to be overcomplete, the number of depth components in the second set is selected to be smaller than the number of depth components in the first set.

<FIG> shows an array of cameras <NUM>. Each camera <NUM> has a corresponding field of view <NUM>. The fields of view <NUM> are only shown for a camera pair <NUM> made of cameras <NUM> three and four. The shaded area <NUM> shows the area of a scene for which depth data can be obtained based on images obtained by the camera pair <NUM>.

In general, the number of camera pairs <NUM> used to obtain depth data of a scene does not only depend on the size of the space where you want to shoot the video, but also depends on how close all of the objects in the scene are to the camera pairs <NUM>. If the objects are very close to the camera pairs <NUM>, some cameras <NUM> may have to be moved closer to other cameras <NUM> in order to properly estimate the depth data. In other words, the baseline of the camera pairs <NUM> should be short when the objects are close. However, if the objects are far from the camera pairs <NUM>, then the baseline could be longer. All else being equal, longer baselines may be preferred, as a longer baseline may be associated with greater accuracy in depth estimation, when using feature-matching / disparity.

In a dynamic scene (e.g. a football match) the background is usually very far from the array of cameras <NUM>. However, the players may be moving such that they are sometimes close to the array of cameras <NUM> and sometimes they are far from them. Thus, it is not necessary to always have all of the cameras <NUM> close to each other. However, during VR shooting, all the cameras <NUM> are fixed and, when considering the worst case, enough cameras <NUM> should be used and they should be close enough to each other to obtain depth data for objects which are close.

Once the cameras <NUM> are set up, the number of cameras <NUM> and their relative positions are fixed. However, it has been realized that, in a dynamic scene, it is not necessary to use all the images obtained with the cameras <NUM> to estimate the depth data at all times. Thus an algorithm could select a sub-set of the obtained images which could then be used to estimate the depth data according to the detected scene. This could save computational power and improve the computational efficiency.

In summary, the invention according to the claims proposes a method to dynamically select the necessary images from an array of cameras <NUM> to estimate the depth data of a scene and thus reduce the required computational power.

The array of cameras <NUM> may be part of a <NUM>-DoF VR video shooting system which further includes a workstation and cables for transmitting all the images from the cameras <NUM> to the workstation. The workstation may comprise a processor which is configured to generate the depth data for each camera pair <NUM> for a first frame. The workstation may then select the necessary images from particular camera pairs <NUM> which are needed further estimate the depth data for the following frames. Selecting the necessary images is based on analyzing the depth data for the previous frame.

Analyzing the depth data (e.g. via an algorithm) for the previous frame may comprise detecting and removing the redundant camera pairs <NUM> according to the previous estimated depth data. Optionally, more camera pairs <NUM> may be added if the analysis detects that the currently selected camera pairs <NUM> are not sufficient to estimate the depth data. The depth data may be depth images or depth maps.

The selection of camera pairs <NUM> may be based on the previous depth data. The selection of camera pairs <NUM> could be further based on the detection of occlusion of an object in the scene. Additionally, the selection of camera pairs <NUM> may be based on checking the failure of depth estimation at particular areas of the scene. If there is a failure to estimate depth and/or occlusion at a particular area when estimating the depth data based on the images from a particular camera pair <NUM>, then there may be a need for more (or different) camera pairs <NUM> which are able to obtain depth data for the particular area. Methods of analyzing the depth data will be further detailed below.

The selection of camera pairs <NUM> defines which images (i.e. from which camera pairs <NUM>) are used for depth estimation. Thus, the analysis of depth estimation may be used to choose which images are used for depth estimation for a second frame based on the depth data of a first frame. In this case, all images may be obtained from all of the cameras <NUM> for the second frame but only a particular sub-set of images is chosen based on the selected camera pairs <NUM>. In general, the depth data for a frame is used to choose which images are used for depth estimation in the subsequent frame.

<FIG> shows an array of cameras <NUM> with six camera pairs <NUM>(a-f). In order to estimate depth of an object, the object should be within the field of view <NUM> of at least two cameras <NUM> in a camera pair <NUM>. The depth of a pixel can be estimated by disparity calculation between the two images captured by a camera pair <NUM>.

The shaded area <NUM> defines the area for which depth can be correctly estimated based on the six camera pairs 104a-f. The dotted line <NUM> defines how close an object can be to the six camera pairs 104a-f before the estimated depth data is no longer complete. By using the six camera pairs 104a-f, an object can be relatively close to the camera array whilst ensuring that each part of the object can be seen at least one camera pair <NUM>.

<FIG> shows an array of cameras <NUM> with four camera pairs 104a, 104c, 104e and 104f. Similarly to <FIG>, the shaded area <NUM> defines the area of the scene for which depth can be correctly estimated based on the given camera pairs 104a, 104c, 104e and 104f. The line <NUM> defines how close an object can be to the four camera pairs 104a, 104c, 104e and 104f before the estimated depth data is no longer complete.

By only using four camera pairs 104a, 104c, 104e and 104f, an object can no longer be as close to the array of cameras as when six camera pairs 104a-f are used (see <FIG>), but depth data only needs to be estimated for four camera pairs 104a, 104c, 104e and 104f, instead of for six camera pairs 104a-f.

<FIG> shows an array of cameras <NUM> with three camera pairs <NUM>, <NUM> and 104i. Similarly to <FIG> and <FIG>, the shaded area <NUM> defines the area of the scene for which depth can be correctly estimated based on the given camera pairs <NUM>, <NUM> and 104i. The line <NUM> defines how close an object can be to the three camera pairs <NUM>, <NUM> and 104i before the estimated depth data is no longer complete.

By only using three camera pairs <NUM>, <NUM> and 104i, an object can no longer be as close to the array of cameras as when four or six camera pairs <NUM> are used (see <FIG> and <FIG>), but depth data only needs to be estimated for three camera pairs <NUM>, <NUM> and 104i instead of for four or six camera pairs <NUM>.

Thus, based on <FIG>, <FIG>, there is a clear relationship between the number of camera pairs <NUM> for which depth data is obtained and how close an object can be to the array of cameras <NUM>. For example, it is clear that when an object moves further from the array of cameras <NUM>, the necessary computational resources to estimate the depth data are reduced. If depth data in a dynamic scene is always estimated for the six camera pairs 104a-f shown in <FIG>, this would likely be a waste of computational resources as it is unlikely that there will always be an object that close to the array of cameras <NUM>.

In practice, the distance of objects to array of cameras <NUM> is known (e.g. based on depth data from a previous frame), thus there is a need for a way of finding which camera pairs <NUM> to select (i.e. which images to use for depth estimation for the next frame).

At the beginning of the <NUM>-DoF video, the system may have no idea of the scene it will see. Thus, it may be advantageous to use a large number of camera pairs <NUM> to estimate the depth data for the first frame (e.g. use every two neighboring cameras for the first frame, as shown in <FIG>). After that, the depth data for the first frame can be used to select a minimum number of camera pairs <NUM> which ensures that each part of the scene can be seen at least one camera pair <NUM>.

Additionally, the larger the baseline of a camera pair <NUM> (i.e. distance between the two cameras) is, the more accurate the depth data is. Thus, the accumulation (i.e. sum) of all the baselines for all the camera pairs <NUM> used may be maximized when selecting the camera pairs <NUM>. Any kind of optimization method (or even a brute force method) may be used to select the cameras to meet the two conditions (i.e. each part of the scene imaged by at least one camera pair <NUM> and the sum of baselines maximized). After the camera pairs <NUM> are selected, the selected camera pairs <NUM> can be used to estimate the depth data for the next frame. This method can be continuously used for each of the following frames. This method is applicable when some objects are moving away from the cameras <NUM>.

Once the camera pairs <NUM> are selected and used for depth data estimation, if some objects are moving toward some cameras, more camera pairs <NUM> may need to be selected for the next frame. Optionally, the depth data for all selected camera pairs <NUM> may be stitched. Stitching the depth data from multiple camera pairs <NUM> involves all of the depth data being joined together (i.e. stitched) into a singular component (e.g. a depth image). Stitching algorithms will be known, for example, for stitching depth images and/or depth maps. In some cases, all the depth data may not be able to be stitched as, for example, one depth image by a stitching algorithm.

If an error is detected during the stitching of depth data, one or more camera pairs may be added which correspond to the depth gap. Alternatively, all the camera pairs <NUM> (e.g. <FIG>) can be used to estimate depth data for a second frame and then use the estimated depth data for the second frame to re-select the camera pairs <NUM> for a third frame. Selecting the camera pairs <NUM> may be based on the depth data obtained by the camera pairs <NUM> and/or a stitched depth component generated from the depth data.

<FIG> show an array of cameras <NUM> imaging two objects <NUM> and <NUM> with two and three camera pairs <NUM> respectively. The array of cameras <NUM> for <FIG> only contains four cameras <NUM> for simplicity. In <FIG>, the object <NUM> is partly occluding the object <NUM> for the selected camera pairs 104a and 104b. The area <NUM> shows the area of the scene which is occluded due to the object <NUM>. A part of object <NUM> is thus occluded by the object <NUM> for both camera pairs 104a and 104b. If object <NUM> was not in the scene, the depth data for all of object <NUM> could be estimated by both of camera pairs 104a and 104b. However, due to the occlusion caused by object <NUM>, the depth data can no longer be estimated for all of object <NUM> using these camera pairs.

In <FIG>, the camera pair 104c is also selected. Object <NUM> does not occlude object <NUM> for the camera pair 104c and thus depth data can be obtained by camera pair 104c. Thus, object occlusion may also need to be considered when selecting camera pairs <NUM>.

<FIG> shows a flow chart according to an embodiment of the current invention. At the start of the <NUM>-DoF video (i.e. for a first frame of the video), all camera pairs may be used, step <NUM>. The images from all the cameras pairs are received, step <NUM>, and used to estimate depth data (e.g. depth images/depth maps) for the scene, step <NUM>. The images from every two neighboring cameras may be used to generate depth data as shown in <FIG>. The depth data for each selected camera pair may be stitched, step <NUM>, with a stitching algorithm.

The stitched depth data may be analyzed, step <NUM>, to determine if there are any "errors". Alternatively, the depth data (before stitching <NUM>) may be analyzed <NUM>. If the analysis <NUM> identifies one or more errors, it may mean the selected camera pairs are not sufficient to obtain a complete set of depth data for then scene. In other words, the depth data may be undercomplete. An error may comprise an object not being imaged by at least two cameras in the camera array, an object being occluded or the depth data not being complete for a camera pair (e.g. lens of a camera being dirty/broken). If an error occurs, then all the camera pairs may be re-selected, step <NUM> and used for the next frame.

If there are no errors detected in the stitched depth data, this may mean the depth data obtained for the scene in the current frame is overcomplete (i.e. more depth data than needed for the scene/some of the depth data is redundant or duplicated). If this is the case, an optimization method <NUM> may be used to select the camera pairs which meet three conditions: the amount of selected cameras is minimum, the part of the scene in the real world corresponding to each pixel in the depth data is physically seen by at least two cameras (occlusion considered) and the accumulation of the baselines of the selected camera pairs is maximum. The optimization method <NUM> is used to select camera pairs, step <NUM>, for the next frame. For each frame, the output may be the stitched depth image.

Clearly, the selection of camera pairs above is used to obtain image pairs for depth estimation. However, the selection of camera pairs may involve selecting which cameras obtain images of the scene or selecting the images of which cameras to use for depth estimation. More than two cameras may be selected for depth estimation (e.g. three cameras may be selected for more accurate depth data instead of a camera pair).

The skilled person would be readily capable of developing a processor for carrying out any herein described method. Thus, each step of a flow chart may represent a different action performed by a processor, and may be performed by a respective module of the processor.

As discussed above, the system makes use of processor to perform the data processing. The processor can be implemented in numerous ways, with software and/or hardware, to perform the various functions required. The processor typically employs one or more microprocessors that may be programmed using software (e.g., microcode) to perform the required functions. The processor may be implemented as a combination of dedicated hardware to perform some functions and one or more programmed microprocessors and associated circuitry to perform other functions.

Claim 1:
A computer-implemented method for generating depth data for a six degree of freedom, 6DoF, video of a scene, the method comprising:
obtaining (<NUM>) a first set of images of the scene;
generating (<NUM>) a first set of depth components based on the first set of images, wherein each depth component comprises a plurality of depth values;
analyzing (<NUM>) the first set of depth components to determine completeness of the depth components;
obtaining (<NUM>) a second set of images of the scene, wherein each set of images are obtained from a group of cameras among all the available cameras imaging the scene, which are fixed; and
generating (<NUM>) a second set of depth components based on the second set of images, wherein:
if the analysis (<NUM>) determines the first set of depth components to be overcomplete, the number of depth components in the second set is selected to be smaller than the number of depth components in the first set, wherein the overcomplete first set of depth components has more than enough depth information of the scene to render a 6DoF frame, and
if the analysis determines the first set of depth components to be undercomplete, the number of depth components in the second set is selected to be larger than the number of depth components in the first set, wherein a portion of the depth information of the scene is missing from the undercomplete first set of depth components to render a frame of the 6DoF scene.