Patent ID: 12243162

DETAILED DESCRIPTION

Aspects of the present disclosure are directed generally to methods of determining the depth of a scene, such as surgical scene, and using the depth information to reconstruct a virtual camera perspective of the scene. In several of the embodiments described below, for example, a method includes augmenting depth data of the scene captured with a depth sensor with depth data from one or more images of the scene. For example, the method can include capturing (i) the depth data of the scene with the depth sensor and (ii) images of the scene with a plurality of cameras. The method can further include generating a point cloud representative of the scene based on the depth data from the depth sensor and identifying a missing region of the point cloud, such as a region occluded from the view of the depth sensor. The method can then include generating depth data for the missing region based on the images from the cameras. The images can be light field images containing information about the intensity of light rays emanating from the scene and also information about a direction the light rays are traveling through space. The method can further include merging (i) the depth data for the missing region derived from the images with (ii) the depth data from the depth sensor to generate a merged point cloud representative of the scene.

In one aspect of the present technology, the merged point cloud can have a greater accuracy and/or resolution than the point cloud generated from the depth data from the depth sensor alone. In another aspect of the present technology, depth information is determined quickly for as much of the scene as possible using a depth sensor, and light field processing is used only for the relatively small regions of the scene where depth information cannot be or cannot accurately be determined using the depth sensor (e.g., the missing regions). Accordingly, the present technology can provide real time or near real time depth and image processing while also providing improved accuracy. That is, the combined depth determination approach of the present technology can provide (i) improved latency compared to light field processing alone and (ii) improved accuracy compared to depth sensor processing alone.

Specific details of several embodiments of the present technology are described herein with reference toFIGS.1-9C. The present technology, however, can be practiced without some of these specific details. In some instances, well-known structures and techniques often associated with camera arrays, light field cameras, image reconstruction, depth sensors, and the like have not been shown in detail so as not to obscure the present technology. The terminology used in the description presented below is intended to be interpreted in its broadest reasonable manner, even though it is being used in conjunction with a detailed description of certain specific embodiments of the disclosure. Certain terms can even be emphasized below; however, any terminology intended to be interpreted in any restricted manner will be overtly and specifically defined as such in this Detailed Description section.

The accompanying Figures depict embodiments of the present technology and are not intended to be limiting of its scope. The sizes of various depicted elements are not necessarily drawn to scale, and these various elements can be arbitrarily enlarged to improve legibility. Component details can be abstracted in the Figures to exclude details such as position of components and certain precise connections between such components when such details are unnecessary for a complete understanding of how to make and use the present technology. Many of the details, dimensions, angles, and other features shown in the Figures are merely illustrative of particular embodiments of the disclosure. Accordingly, other embodiments can have other details, dimensions, angles, and features without departing from the spirit or scope of the present technology.

The headings provided herein are for convenience only and should not be construed as limiting the subject matter disclosed.

I. Selected Embodiments of Imaging Systems

FIG.1is a schematic view of an imaging system100(“system100”) configured in accordance with embodiments of the present technology. In the illustrated embodiment, the system100includes an image processing device102that is operably/communicatively coupled to one or more display devices104, one or more input controllers106, and a camera array110. In other embodiments, the system100can comprise additional, fewer, or different components. In some embodiments, the system100can include features that are generally similar or identical to those of the mediated-reality imaging systems disclosed in U.S. patent application Ser. No. 16/586,375, titled “CAMERA ARRAY FOR A MEDIATED-REALITY SYSTEM,” which is incorporated herein by reference in its entirety.

In the illustrated embodiment, the camera array110includes a plurality of cameras112(identified individually as cameras112a-112n) that are each configured to capture images of a scene108from a different perspective. In some embodiments, the cameras112are positioned at fixed locations and orientations relative to one another. For example, the cameras112can be structurally secured by/to a mounting structure (e.g., a frame) at predefined fixed locations and orientations. In some embodiments, the cameras112can be positioned such that neighboring cameras share overlapping views of the scene108. In some embodiments, the cameras112in the camera array110are synchronized to capture images of the scene108substantially simultaneously (e.g., within a threshold temporal error). In some embodiments, all or a subset of the cameras112can be light-field/plenoptic/RGB cameras that are configured to capture information about the light field emanating from the scene108(e.g., information about the intensity of light rays in the scene108and also information about a direction the light rays are traveling through space).

In the illustrated embodiment, the camera array110further comprises (i) one or more projectors114configured to project a structured light pattern onto/into the scene108, and (ii) one or more depth sensors116configured to estimate a depth of a surface in the scene108. In some embodiments, the depth sensor116can estimate depth based on the structured light pattern emitted from the projector114.

The image processing device102is configured to (i) receive images (e.g., light-field images, light field image data) captured by the camera array110and depth information from the depth sensor116, and (ii) process the images and depth information to synthesize an output image corresponding to a virtual camera perspective. In the illustrated embodiment, the output image corresponds to an approximation of an image of the scene108that would be captured by a camera placed at an arbitrary position and orientation corresponding to the virtual camera perspective. More specifically, the depth information can be combined with the images from the cameras112to synthesize the output image as a three-dimensional rendering of the scene108as viewed from the virtual camera perspective. In some embodiments, the image processing device102can synthesize the output image using any of the methods disclosed in U.S. patent application Ser. No. 16/457,780, titled “SYNTHESIZING AN IMAGE FROM A VIRTUAL PERSPECTIVE USING PIXELS FROM A PHYSICAL IMAGER ARRAY WEIGHTED BASED ON DEPTH ERROR SENSITIVITY,” which is incorporated herein by reference in its entirety.

The image processing device102can synthesize the output image from a subset (e.g., two or more) of the cameras112in the camera array110, but does not necessarily utilize images from all of the cameras112. For example, for a given virtual camera perspective, the image processing device102can select a stereoscopic pair of images from two of the cameras112that are positioned and oriented to most closely match the virtual camera perspective. In some embodiments, the image processing device102(and/or depth sensor116) is configured to estimate a depth for each surface point of the scene108and to generate a point cloud and/or three-dimensional (3D) mesh that represents the surface of the scene108. For example, in some embodiments the depth sensor116can detect the structured light projected onto the scene108by the projector114to estimate depth information of the scene108. Alternatively or additionally, the image processing device102can perform the depth estimation based on depth information received from the depth sensor116. As described in detail below, in some embodiments the image processing device102can estimate depth from multiview image data from the cameras112with or without utilizing information collected by the projector114or the depth sensor116.

In some embodiments, functions attributed to the image processing device102can be practically implemented by two or more physical devices. For example, in some embodiments a synchronization controller (not shown) controls images displayed by the projector114and sends synchronization signals to the cameras112to ensure synchronization between the cameras112and the projector114to enable fast, multi-frame, multi-camera structured light scans. Additionally, such a synchronization controller can operate as a parameter server that stores hardware specific configurations such as parameters of the structured light scan, camera settings, and camera calibration data specific to the camera configuration of the camera array110. The synchronization controller can be implemented in a separate physical device from a display controller that controls the display device104, or the devices can be integrated together.

The image processing device102can comprise a processor and a non-transitory computer-readable storage medium that stores instructions that when executed by the processor, carry out the functions attributed to the image processing device102as described herein. Although not required, aspects and embodiments of the present technology can be described in the general context of computer-executable instructions, such as routines executed by a general-purpose computer, e.g., a server or personal computer. Those skilled in the relevant art will appreciate that the present technology can be practiced with other computer system configurations, including Internet appliances, hand-held devices, wearable computers, cellular or mobile phones, multi-processor systems, microprocessor-based or programmable consumer electronics, set-top boxes, network PCs, mini-computers, mainframe computers and the like. The present technology can be embodied in a special purpose computer or data processor that is specifically programmed, configured or constructed to perform one or more of the computer-executable instructions explained in detail below. Indeed, the term “computer” (and like terms), as used generally herein, refers to any of the above devices, as well as any data processor or any device capable of communicating with a network, including consumer electronic goods such as game devices, cameras, or other electronic devices having a processor and other components, e.g., network communication circuitry.

The invention can also be practiced in distributed computing environments, where tasks or modules are performed by remote processing devices, which are linked through a communications network, such as a Local Area Network (“LAN”), Wide Area Network (“WAN”), or the Internet. In a distributed computing environment, program modules or sub-routines can be located in both local and remote memory storage devices. Aspects of the invention described below can be stored or distributed on computer-readable media, including magnetic and optically readable and removable computer discs, stored as in chips (e.g., EEPROM or flash memory chips). Alternatively, aspects of the invention can be distributed electronically over the Internet or over other networks (including wireless networks). Those skilled in the relevant art will recognize that portions of the present technology can reside on a server computer, while corresponding portions reside on a client computer. Data structures and transmission of data particular to aspects of the present technology are also encompassed within the scope of the invention.

The virtual camera perspective can be controlled by an input controller106that provides a control input corresponding to the location and orientation of the virtual camera perspective. The output images corresponding to the virtual camera perspective are outputted to the display device104. The display device104is configured to receive output images (e.g., the synthesized three-dimensional rendering of the scene108) and to display the output images for viewing by one or more viewers. The image processing device102can beneficially process received inputs from the input controller106and process the captured images from the camera array110to generate output images corresponding to the virtual perspective in substantially real-time as perceived by a viewer of the display device104(e.g., at least as fast as the frame rate of the camera array110).

The display device104can comprise, for example, a head-mounted display device, a monitor, a computer display, and/or another display device. In some embodiments, the input controller106and the display device104are integrated into a head-mounted display device and the input controller106comprises a motion sensor that detects position and orientation of the head-mounted display device. The virtual camera perspective can then be derived to correspond to the position and orientation of the head-mounted display device104such that the virtual perspective corresponds to a perspective that would be seen by a viewer wearing the head-mounted display device104. Thus, in such embodiments the head-mounted display device104can provide a real-time rendering of the scene108as it would be seen by an observer without the head-mounted display device104. Alternatively, the input controller106can comprise a user-controlled control device (e.g., a mouse, pointing device, handheld controller, gesture recognition controller) that enables a viewer to manually control the virtual perspective displayed by the display device104.

FIG.2is a perspective view of a surgical environment employing the system100for a surgical application in accordance with embodiments of the present technology. In the illustrated embodiment, the camera array110is positioned over the scene108(e.g., a surgical site) and supported/positioned via a swing arm222that is operably coupled to a workstation224. In some embodiments, the swing arm222can be manually moved to position the camera array110while, in other embodiments, the swing arm222can be robotically controlled in response to the input controller106(FIG.1) and/or another controller. In the illustrated embodiment, the display device104is embodied as a head-mounted display device (e.g., a virtual reality headset, augmented reality headset). The workstation224can include a computer to control various functions of the image processing device102, the display device104, the input controller106, the camera array110, and/or other components of the system100shown inFIG.1. Accordingly, in some embodiments the image processing device102and the input controller106are each integrated in the workstation224. In some embodiments, the workstation224includes a secondary display226that can display a user interface for performing various configuration functions, a mirrored image of the display on the display device104, and/or other useful visual images/indications.

II. Selected Embodiments of Augmenting Depth Data from a Depth Sensor

FIG.3is an isometric view of a portion of the system100illustrating four of the cameras112and the depth sensor116in accordance with embodiments of the present technology. Other components of the system100(e.g., other portions of the camera array110, the image processing device102, etc.) are not shown inFIG.3for the sake of clarity. In the illustrated embodiment, each of the cameras112has a field of view330and is oriented such that the field of view330is aligned with a portion of the scene108. Likewise, the depth sensor116can have a field of view332aligned with a portion of the scene108. In some embodiments, a portion of some or all of the field of views330,332can overlap.

In the illustrated embodiment, a portion of a spine309of a patient (e.g., a human patient) is located in/at the scene108. Often, the spine309(or other surfaces located in the scene108) will have a complex 3D geometry such that is difficult to accurately determine the depth of its surfaces, and therefore difficult to accurately model with a point cloud, 3D mesh, and/or other mathematical representation. For example, if a portion of the surface of the scene108(e.g., a portion of the spine309) is occluded from the field of view332of the depth sensor116, the depth sensor116will be unable to determine the depth of the occluded region. Likewise, it can be difficult to accurately determine the depth along steep surfaces of the scene108. More specifically, for a structured light system to recover depth at a given location the structured light projector (e.g., the projector114) must illuminate that location with a pixel or block of pixels of structured illumination. Also, the imagers (e.g., the depth sensor116) measuring the projection must have a pixel/block that sees that illumination. Both conditions must be met to make a measurement of the location. In practice, it is typically not possible to achieve a fill rate of 100%—where every pixel of depth has a valid value. This is because real scenes have complex geometries that cause occlusion of the projector, the imagers/sensors, or both. Accordingly, if the system100uses only the depth sensor116to determine depth, a depth model generated by the system100can have missing regions (e.g., holes) corresponding to the portions (e.g., surfaces) of the scene108where depth information is unavailable.

In some embodiments, the system100is unable to adequately generate—or unable to accurately generate—an output image of the scene108for such portions of the scene108that have inadequate and/or inaccurate depth information.FIG.4, for example, is a schematic view of a point cloud440generated by the depth sensor116of the system100for the surfaces of the spine309shown inFIG.3in accordance with embodiments of the present technology. Referring toFIGS.1-4together, the point cloud440generally comprises a plurality (e.g., hundreds, thousands, millions, or more) of data points corresponding to a distance of the surfaces of the spine309and/or other features in the scene108relative to the sensor116(e.g., the depth of the spinal surfaces). The point cloud440therefore maps/represents the 3D surface of the spine309and can be used by the image processing device102to synthesize the images from the cameras112into the output image of the scene108rendered from any desired virtual perspective, as described in detail above. In the illustrated embodiment, the point cloud440includes one or more missing regions442corresponding to portions (e.g., surfaces) of the scene108where depth information is inadequate and/or not reliable (e.g., regions where the depth sensor116is occluded). Accordingly, the system100may not be able to render an accurate output image for those portions of the scene108.

In some embodiments, the image processing device102can process image data from one or more of the cameras112to determine the depth of the spinal surfaces at one or more of the locations where depth information from the depth sensor116is inadequate, unreliable, and/or inaccurate. That is, the image data from the cameras112can be used to “fill in” the missing regions442of the point cloud440. More specifically,FIG.5is a flow diagram of a process or method550for augmenting depth data captured with the depth sensor116using image data captured by the cameras112in accordance with embodiments of the present technology. Although some features of the method550are described in the context of the embodiments shown inFIGS.1-4for the sake of illustration, one skilled in the art will readily understand that the method550can be carried out using other suitable systems and/or devices described herein.

At block551, the method550includes (a) capturing depth data of the scene108(e.g., of the spine309) from the depth sensor116and (b) image data of the scene from one or more of the cameras112. The depth data can include, for example, data about a structured light pattern projected onto/into the scene108(e.g., from the projector114). The image data can be light field data including data about the intensity of light rays emanating from the scene108and also information about a direction the light rays are traveling. In some embodiments, the depth sensor116and the cameras112can capture the depth data and the image data simultaneously or substantially simultaneously and/or in real time or near real time. In other embodiments, the depth data can be captured before the image data.

At block552, the method550includes generating a point cloud of the scene108, such as the point cloud440, based on the depth data from the depth sensor116. In some embodiments, the image processing device102can receive the depth data from the depth sensor116and generate the point cloud based on the depth data. In some embodiments, the method550can further include generating a 3D mesh instead of or in addition to a point cloud. In other embodiments, at block552the method550can include generating other mathematical representations of the physical geometry of the scene108.

At block553, the method550includes back projecting the point cloud and/or depth data associated with the point cloud to individual ones of the cameras112and/or to the image processing device102. Back projecting the point cloud to the cameras112allows an image of the scene108to be reconstructed. More specifically, back projection correlates a 2D pixel location in the images from the cameras112with a 3D position from the point cloud. By back projecting the point cloud to each of the images from the cameras112, each pixel in the images can be associated with a 3D point—or not if the 3D position cannot be determined for the reasons discussed in detail above. An even simpler classifier is to label each pixel in the 2D image as having a valid 3D correspondence or not. In some embodiments, this classification can be used to create a binary mask for each of the cameras112that indicates which pixels have a valid 3D point.

At block554, the method550includes identifying regions of missing data in the point cloud. For example, the method550can include identifying the missing regions442of the point cloud440where depth data is missing or incomplete. In some embodiments, identifying the missing data can include filtering the point cloud data and searching for holes that are greater than a predetermined threshold (e.g., a user-specified threshold) using, for example, an inverse Eulerian approach. In some embodiments, identifying missing data can include scanning the point cloud to determine regions with sparse or non-existent points. In some embodiments, a mesh can be generated for the point cloud (e.g., at block552), and holes can be identified in the mesh using, for example, a method that identifies triangles in the mesh having at least one edge that is not shared by another triangle. In yet other embodiments, the missing regions442can be identified by searching for regions of the images from the cameras112where no valid 3D correspondence exists (e.g., by examining the binary mask for each image). In some embodiments, blocks553and554can be executed using the same algorithm and/or as part of the same computational process.

In some embodiments, at block554, the method550can additionally or alternatively include identifying regions of invalid depth data, low confidence depth data, and/or other potentially problematic regions of the point cloud. For example, the depth sensor116can be configured to tag the depth data it captures with validation or confidence levels, and the method550can include identifying regions of the point cloud and/or mesh with validation or confidence levels that are below a predetermined threshold (e.g., a user-specified threshold). Such invalid or low confidence regions can be regions of the point cloud or mesh having discontinuities, sparse depth data, badly behaved normal values, and the like. In some embodiments, the method550may not identify single missing pixels as missing or invalid regions, and/or conversely may identify as missing/invalid regions of missing pixels with some “valid” pixels interspersed.

In some embodiments, at block554, the method550can further include determining depth data for areas surrounding the missing regions442of the point cloud. This surrounding depth data can help inform/predict the depth of the missing regions442if it is assumed that there are not large discontinuities between the missing regions442and the surrounding areas, such that the missing depth values can be expected to be close to the surrounding depths.

At block555, the method550includes extracting/identifying image data corresponding to the missing or invalid regions of the point cloud or mesh. For example, the image processing device102can determine which of the cameras112have their field of view330aligned with the region of the scene108that corresponds to the missing region. In some embodiments, the image processing device102can make this determination based on a priori information about (i) the positions and orientations of the cameras (and thus the extent of their fields of view330), (ii) the back projection of the depth data to the cameras112(block553), (iii) processing of the point cloud or mesh, and/or (iv) other data. Moreover, in some embodiments the system100can identify and extract image data from only those of the cameras112that are determined to have adequate optical coverage of the missing regions. In some embodiments, at least some of the cameras112can have at least partially overlapping fields of view330such that it is very likely that at least one of the cameras112has a field of view330aligned with the region of the scene108that corresponds to the missing region—even when other ones of the cameras112are occluded. Accordingly, in one aspect of the present technology the system100is configured to robustly capture image data about the missing regions even where substantial occlusions exist in the scene108. In some embodiments, blocks553-555can be executed using the same algorithm and/or as part of the same computational process.

At block556, the method550includes processing the extracted image data to generate depth data for the missing or invalid regions. For example, the image processing device102can generate depth data for the missing regions using the disparity from the cameras112that have the missing regions within their field of view330(e.g., that are facing the missing regions). In other embodiments, other suitable image processing techniques (e.g., computational algorithms) for determining depth from light field data can be used. In some embodiments, determining depth by processing the image data from the cameras112can be more computationally expensive (e.g., slower) than determining depth using the depth sensor116because of the complex nature of computational algorithms for processing depth information from light field data. As a result, image data from less than all of the cameras112may be used to generate depth data for the missing or invalid regions. In some embodiments, depth information about the areas surrounding the missing or invalid regions (e.g., captured at block554) can be used to accelerate processing of the extracted image data. Specifically, many depth processing algorithms iterate through depths to search for the true values. Accordingly, by limiting the depth range based on the depth of the surrounding areas, a smaller range of depths, disparities, planes, and so on have to be searched through. Thus, the search can avoid local minima that may exist outside this expected region/range—accelerating processing.

At block557, the method550includes merging/fusing the depth data for the missing or invalid regions with the original depth data (e.g., captured at block551) to generate a merged point cloud.FIG.6, for example, is a schematic view of a merged point cloud640in which image-based depth data644has been filled into the missing regions442of the point cloud440shown inFIG.4in accordance with embodiments of the present technology. Accordingly, the merged point cloud640can provide a more accurate and robust depth map of the scene108that facilitates better reconstruction and synthesis of an output image of the scene108rendered from any desired virtual perspective, as described in detail above.

At block558, the method550can optionally include generating a three-dimensional mesh based on the merged point cloud. The 3D mesh can be used to reconstruct/synthesize the output image of the scene108. In some embodiments, the method550can return to block551to update the depth information of the scene108. In some embodiments, the method550can proceed to back project the merged point cloud to the cameras112(block553).

As noted above, determining depth by processing light field image data can be more computationally expensive than determining depth using a depth sensor. Indeed, if depth information for an entire scene were determined entirely through light field image processing, it would be difficult/impracticable to render output images in real time or near real-time because even very fast systems cannot measure and process the significant volume of data fast enough. However, in one aspect of the present technology depth information is determined quickly for as much of the scene as possible using a depth sensor, and light field processing is used only for the relatively small regions of the scene where there is inadequate and/or unreliable depth information from the depth sensor. Accordingly, the present technology can provide real time or near real time depth and image processing while also providing improved accuracy. That is, the combined depth determination approach of the present technology can provide (i) improved latency compared to light field processing alone and (ii) improved accuracy and resolution compared to depth sensor processing alone.

In some embodiments, the latency of the system100can be further improved by updating the depth information only for missing or invalid regions of the point cloud for which increased accuracy is desired. For example,FIG.7is an isometric view of a portion of the system100illustrating two of the cameras112, the display device104, and the generated point cloud440(FIG.4) in accordance with embodiments of the present technology. Other components of the system100(e.g., other portions of the camera array110, the image processing device102, etc.) are not shown inFIG.7for the sake of clarity.

In the illustrated embodiment, the display device104is a head-mounted display device104(e.g., a headset) configured to be worn by a user (e.g., a surgeon) and having a field of view736that is aligned with only a portion of the scene108(e.g., a portion of the spine309shown inFIG.3). The head-mounted display device104can include a display705configured to display the rendered output image of the scene108to the user. The display705can be opaque or partially transparent. In some embodiments, the field of view736of the head-mounted display device104corresponds to a foveated region that represents the relatively narrow field of view that the eyes of the user can perceive.

The system100(e.g., the image processing device102) can track the position and orientation of the field of view736relative to the scene108and can employ the method550(FIG.5) to update only the missing regions442of the point cloud440that are within the field of view736—without updating regions outside of the field of view736. In some embodiments, the system100can identify the cameras112that have the best optical coverage of the portion of the scene108within the field of view736. When the user changes the position and/or orientation of the head-mounted display device104—and thus the field of view736—the system100can seamlessly update (e.g., fill-in) the missing regions442that are within the field of view736in real time or near real time. In one aspect of the present technology, the latency of the image presented to the user via the head-mounted display device104is decreased because the missing regions442that are outside the foveated region of the user are not updated.

Referring again toFIG.1, in some embodiments the cameras112can have a higher resolution than the depth sensor116such that more depth detail of the scene108can be extracted from the cameras112than from the depth sensor116. Accordingly, even where depth information from the depth sensor116exists and is at least adequate to determine the general depth of the scene108, it can be advantageous to also include image data from the cameras112to increase the depth resolution and, correspondingly, the resolution of the image output to the user via the display device104. Therefore, in some embodiments the system100can process image data for particular local regions of the scene108to supplement or replace the depth data captured by the depth sensor116for those local regions. In some embodiments, a background process running on the image processing device102can update the local regions of the scene108automatically if, for example, the depth data from the depth sensor116is of poor quality in those regions. In other embodiments, the user can select certain areas in which to improve the resolution. In yet other embodiments, the system100can improve the resolution by processing light field data corresponding to all or a portion of the foveated region736of the user as shown inFIG.7.

In other embodiments, depth data captured by the depth sensor116can be supplemented or replaced with depth information obtained from means other than processing image data from the cameras112. For example,FIG.8is a flow diagram of a process or method860for augmenting depth data captured with the depth sensor116with depth data from one or more medical scans of a patient in accordance with embodiments of the present technology. Although some features of the method860are described in the context of the system100shown inFIG.1for the sake of illustration, one skilled in the art will readily understand that the method860can be carried out using other suitable systems and/or devices described herein. Moreover, while the method860is described in the context of augmenting depth data of the anatomy of a patient with medical scans of the patient, the method860can be practiced to update/augment depth data for other scenes and/or based on other data from other imaging/scanning techniques.

At block861, the method860includes capturing depth data of the scene108(e.g., live data) from the depth sensor116, such as data about a structured light pattern projected onto/into the scene108. The scene108can include, for example, a portion of a patient undergoing surgery. As one example, the portion of the patient can be a portion of the patient's spine exposed during spinal surgery. Block862of the method860can proceed generally similarly or identically to block552of the method550ofFIG.5to, for example, generate a point cloud representation of the depth of the scene108.

At block863, the method860includes registering the point cloud with medical scan data (e.g., patient data). In some embodiments, the medical scan can be a computerized tomography (CT) scan of the patient's spine that provides a complete 3D data set for at least a portion of the scene108. The registration process matches points in the point cloud to corresponding 3D points in the medical scan. The system100can register the point cloud to the medical scan data by detecting positions of fiducial markers and/or feature points visible in both data sets. For example, where the volumetric data comprises CT data, rigid bodies of bone surface calculated from the CT data can be registered to the corresponding points/surfaces of the point cloud. In other embodiments, the system100can employ other registration processes based on other methods of shape correspondence, and/or registration processes that do not rely on fiducial markers (e.g., markerless registration processes). In some embodiments, the registration/alignment process can include features that are generally similar or identical to the registration/alignment processes disclosed in U.S. Provisional Patent Application No. 62/796,065, titled “ALIGNING PRE-OPERATIVE SCAN IMAGES TO REAL-TIME OPERATIVE IMAGES FOR A MEDIATED-REALITY VIEW OF A SURGICAL SITE,” filed Jan. 23, 2019, which is incorporated herein by reference in its entirety, and which is attached hereto as Appendix A.

At block864, the method860includes identifying missing/invalid regions of the point cloud. In some embodiments, block864can proceed generally similarly or identically to block554of the method550ofFIG.5. In one aspect of the present technology, the medical scan data includes 3D depth data corresponding to the missing or invalid regions of the point cloud. For example,FIG.9Ais a schematic view of a point cloud970corresponding to a portion of a patient's spine and including missing regions972.FIG.9Bis a schematic view of corresponding CT scan data974of the patient's spine in accordance with embodiments of the present technology. Referring toFIGS.9A and9Btogether, the CT scan data974can include 3D volumetric depth data976corresponding to at least a portion of the missing regions972in the point cloud970.

At block865, the method860includes merging/fusing the 3D data of the medical scan with the original depth data (e.g., captured at block861) to generate a merged point cloud that includes data points for the missing or invalid regions. In general, the medical scan data can replace and/or supplement the data in the point cloud. For example, the medical scan data can replace data in regions of the point cloud where captured data is poor, and supplement (e.g., fill in) the missing regions of the point cloud. Accordingly, the merged point cloud can provide a more accurate and robust depth map of the scene108that facilitates better reconstruction and synthesis of an output image of the scene108rendered from any desired virtual perspective, as described in detail above.

More specifically, in some embodiments data from the medical scan is filled-in only for the missing regions of the point cloud.FIG.9C, for example, is a schematic view of a merged point cloud980in which the CT scan data974shown inFIG.9Bhas been filled in for the missing regions972of the point cloud970shown inFIG.9A. In some embodiments, the appropriate regions of the CT scan data974corresponding to the missing regions972of the point cloud970can be found by comparing nearest neighbors between the registered CT scan data974and the point cloud970. That is, for example, points in the medical scan that have no neighbor (e.g., are below a threshold) in the registered point cloud can be identified for merger into/with the point cloud data. In other embodiments, as much of the original depth data (e.g., the point cloud970) as possible can be replaced with the registered medical scan data (e.g., the CT scan data974). In some embodiments, a nearest neighbors algorithm can be used to determine which regions of the original depth data to remove and replace. In yet other embodiments, the medical scan data and the point cloud can be directly merged with a volumetric (e.g., voxel) representation, such as a truncated signed distance function (TSDF).

At block866, the method860can optionally include generating a three-dimensional mesh based on the merged point cloud. The 3D mesh can be used to reconstruct/synthesize the output image of the scene108. In some embodiments, the method860can return to block851to update the depth information of the scene108. In some embodiments, when the medical scan data and original depth data are directly merged using a TSDF, the 3D mesh can be generated using a marching cubes or other suitable algorithm.

In some embodiments, the medical scan data is known a priori and thus does not require significant processing. Accordingly, in one aspect of the present technology the method860can quickly update (e.g., supplement and/or replace) the original depth based on the medical scan—allowing real time or near real time processing and generation of an output image of the scene108.

In other embodiments, the medical scan data can act as an initial state for a depth optimization process, where further refinement is possible. For example, the medical scan data can be registered to the live data to fill-in holes as described in detail with reference toFIGS.8-9C. However, in some embodiments the cameras112can have a higher resolution/accuracy than the medical scan data. Accordingly, the merged depth information from the depth sensor116and the medical scan data can be used to initialize a 3D reconstruction process using images from the cameras112. The depth information from the images can then be merged with or replace the depth information from the medical scan. In some embodiments, the depth processing of the image data is accelerated because a depth range for the missing/invalid regions is known based on the medical scan data—thus minimizing the range of iterations needed to determine true depth values from the image data.

III. Further Examples

The following examples are illustrative of several embodiments of the present technology:

1. A method of determining the depth of a scene, the method comprising:capturing depth data of the scene with a depth sensor;capturing image data of the scene with a plurality of cameras;generating a point cloud representative of the scene based on the depth data;identifying a region of the point cloud;generating depth data for the region based on the image data; and merging the depth data for the region with the depth data from the depth sensor to generate a merged point cloud representative of the scene.

2. The method of example 1 wherein the region of the point cloud is a missing region of the point cloud in which the point cloud includes no data or sparse data.

3. The method of example 2 wherein identifying the missing region of the point cloud includes determining that the missing region of the point cloud has fewer than a predetermined threshold number of data points.

4. The method of example 2 or example 3 wherein identifying the missing region of the point cloud includes identifying a hole in the point cloud that is larger than a user-defined threshold.

5. The method of any one of examples 2-4 wherein generating the depth data for the missing region is further based on a portion of the depth data captured by the depth sensor that surrounds the missing region.

6. The method of any one of examples 1-5 wherein the depth data for the region has a greater resolution than the depth data captured with the depth sensor.

7. The method of any one of examples 1-6 wherein the method further comprises generating a three-dimensional mesh representative of the scene based on the merged point cloud.

8. The method of any one of examples 1-7 wherein the scene is a surgical scene.

9. The method of any one of examples 1-8 wherein the plurality of cameras each have a different position and orientation relative to the scene, and wherein the image data is light field image data.

10. The method of any one of examples 1-9 wherein the method further comprises:processing the image data and the merged point cloud to synthesize an output image of the scene corresponding to a virtual camera perspective; andtransmitting the output image to the display for display to a user.

11. The method of example 10 wherein the display is a head-mounted display worn by the user, and wherein identifying the region of the scene is based on at least one of a position and an orientation of the head-mounted display.

12. A system for imaging a scene, comprising:multiple cameras arranged at different positions and orientations relative to the scene and configured to capture image data of the scene;a depth sensor configured to capture depth data about a depth of the scene; anda computing device communicatively coupled to the cameras and the depth sensor, wherein the computing device has a memory containing computer-executable instructions and a processor for executing the computer-executable instructions contained in the memory, and wherein the computer-executable instructions include instructions for—receiving the image data from the cameras;receiving the depth data from the depth sensor;generating a point cloud representative of the scene based on the depth data;identifying a region of the point cloud;generating depth data for the region based on the image data; and merging the depth data for the region with the depth data from the depth sensor to generate a merged point cloud representative of the scene.

13. The system of example 12 wherein the region of the point cloud is a missing region of the point cloud in which the point cloud includes no data or sparse data.

14. The system of example 12 or example 13 wherein the region of the point cloud is user-selected.

15. The system of any one of examples 12-14, further comprising a display, wherein the computing device is communicatively coupled to the display, and wherein the computer-executable instructions further include instructions for—processing the image data and the merged point cloud to synthesize an output image of the scene corresponding to a virtual camera perspective; andtransmitting the output image to the display for display to a user.

16. The system of example 15 wherein identifying the region of the scene is based on at least one of a position and an orientation of the display.

17. A method of determining the depth of a scene, the method comprising:capturing depth data of the scene with a depth sensor;generating a point cloud representative of the scene based on the depth data;identifying a region of the point cloud;registering the point cloud with three-dimensional (3D) medical scan data; and merging at least a portion of the 3D medical scan data with the depth data from the depth sensor to generate a merged point cloud representative of the scene.

18. The method of example 17 wherein the region of the point cloud is a missing region of the point cloud in which the point cloud includes no data or sparse data.

19. The method of example 18 wherein the scene is a medical scene including a portion of a patient, wherein the missing region of the point cloud corresponds to the portion of the patient, and wherein the portion of the 3D medical scan data portion corresponds to the same portion of the patient.

20. The method of any one of examples 17-19 wherein the 3D medical scan data is a computed tomography (CT) data.

IV. Conclusion

The above detailed description of embodiments of the technology are not intended to be exhaustive or to limit the technology to the precise form disclosed above. Although specific embodiments of, and examples for, the technology are described above for illustrative purposes, various equivalent modifications are possible within the scope of the technology as those skilled in the relevant art will recognize. For example, although steps are presented in a given order, alternative embodiments can perform steps in a different order. The various embodiments described herein can also be combined to provide further embodiments.

From the foregoing, it will be appreciated that specific embodiments of the technology have been described herein for purposes of illustration, but well-known structures and functions have not been shown or described in detail to avoid unnecessarily obscuring the description of the embodiments of the technology. Where the context permits, singular or plural terms can also include the plural or singular term, respectively.

Moreover, unless the word “or” is expressly limited to mean only a single item exclusive from the other items in reference to a list of two or more items, then the use of “or” in such a list is to be interpreted as including (a) any single item in the list, (b) all of the items in the list, or (c) any combination of the items in the list. Additionally, the term “comprising” is used throughout to mean including at least the recited feature(s) such that any greater number of the same feature and/or additional types of other features are not precluded. It will also be appreciated that specific embodiments have been described herein for purposes of illustration, but that various modifications can be made without deviating from the technology. Further, while advantages associated with some embodiments of the technology have been described in the context of those embodiments, other embodiments can also exhibit such advantages, and not all embodiments need necessarily exhibit such advantages to fall within the scope of the technology. Accordingly, the disclosure and associated technology can encompass other embodiments not expressly shown or described herein.