Camera array for a mediated-reality system

A camera array for a mediated-reality system includes a plurality of hexagonal cells arranged in a honeycomb pattern in which a pair of inner cells include respective edges adjacent to each other and a pair of outer cells are separated from each other by the inner cells. A plurality of cameras are mounted within each of the plurality of hexagonal cells. The plurality of cameras include at least one camera of a first type and at least one camera of a second type. The camera of the first type may have a longer focal length than the camera of the second type.

BACKGROUND

Technical Field

The disclosed embodiments relate generally to a camera array, and more specifically, to a camera array for generating a virtual perspective of a scene for a mediated-reality viewer.

Description of the Related Art

In a mediated reality system, an image processing system adds, subtracts, or modifies visual information representing an environment. For surgical applications, a mediated reality system may enable a surgeon to view a surgical site from a desired perspective together with contextual information that assists the surgeon in more efficiently and precisely performing surgical tasks.

DETAILED DESCRIPTION

Overview

A camera array includes a plurality of hexagonal cells arranged in a honeycomb pattern in which a pair of inner cells include respective edges adjacent to each other and a pair of outer cells are separated from each other by the inner cells. A plurality of cameras is mounted within each of the plurality of hexagonal cells. The plurality of cameras includes at least one camera of a first type and at least one camera of a second type. For example, the camera of the first type may have a longer focal length than the camera of the second type. The plurality of cameras within each of the plurality of hexagonal cells are arranged in a triangular grid approximately equidistant from neighboring cameras. In an embodiment, at least one camera of the second type within each of the plurality of hexagonal cells is at a position further from or equidistant from a center point of the camera array relative to cameras of the first type.

FIG. 1illustrates an example embodiment of a mediated-reality system100. The mediated-reality system100comprises an image processing device110, a camera array120, a display device140, and an input controller150. In alternative embodiments, the mediated-reality system100may comprise additional or different components.

The camera array120comprises a plurality of cameras122(e.g., a camera122-1, a camera122-2, . . . , a camera122-N) that each capture respective images of a scene130. The cameras122may be physically arranged in a particular configuration as described in further detail below such that their physical locations and orientations relative to each other are fixed. For example, the cameras122may be structurally secured by a mounting structure to mount the cameras122at predefined fixed locations and orientations. The cameras122of the camera array120may be positioned such that neighboring cameras may share overlapping views of the scene130. The cameras122in the camera array120may furthermore be synchronized to capture images of the scene130substantially simultaneously (e.g., within a threshold temporal error). The camera array120may furthermore comprise one or more projectors124that projects a structured light pattern onto the scene130. The camera array120may furthermore comprise one or more depth sensors126that perform depth estimation of a surface of the scene150.

The image processing device110receives images captured by the camera array120and processes the images to synthesize an output image corresponding to a virtual camera perspective. Here, the output image corresponds to an approximation of an image of the scene130that would be captured by a camera placed at an arbitrary position and orientation corresponding to the virtual camera perspective. The image processing device110synthesizes the output image from a subset (e.g., two or more) of the cameras122in the camera array120, but does not necessarily utilize images from all of the cameras122. For example, for a given virtual camera perspective, the image processing device110may select a stereoscopic pair of images from two cameras122that are positioned and oriented to most closely match the virtual camera perspective.

The image processing device110may furthermore perform a depth estimation for each surface point of the scene150. In an embodiment, the image processing device110detects the structured light projected onto the scene130by the projector124to estimate depth information of the scene. Alternatively, or in addition, the image processing device110includes dedicated depth sensors126that provide depth information to the image processing device110. In yet other embodiments, the image processing device110may estimate depth only from multi-view image data without necessarily utilizing any projector124or depth sensors126. The depth information may be combined with the images from the cameras122to synthesize the output image as a three-dimensional rendering of the scene as viewed from the virtual camera perspective.

In an embodiment, functions attributed to the image processing device110may be practically implemented by two or more physical devices. For example, in an embodiment, a synchronization controller controls images displayed by the projector124and sends synchronization signals to the cameras122to ensure synchronization between the cameras122and the projector124to enable fast, multi-frame, multi-camera structured light scans. Additionally, this synchronization controller may operate as a parameter server that stores hardware specific configurations such as parameters of the structured light scan, camera settings, and camera calibration data specific to the camera configuration of the camera array120. The synchronization controller may be implemented in a separate physical device from a display controller that controls the display device140, or the devices may be integrated together.

The virtual camera perspective may be controlled by an input controller150that provides a control input corresponding to the location and orientation of the virtual imager perspective. The output image corresponding to the virtual camera perspective is outputted to the display device140and displayed by the display device140. The image processing device110may beneficially process received inputs from the input controller150and process the captured images from the camera array120to generate output images corresponding to the virtual perspective in substantially real-time as perceived by a viewer of the display device140(e.g., at least as fast as the frame rate of the camera array120).

The image processing device110may comprise a processor and a non-transitory computer-readable storage medium that stores instructions that when executed by the processor, carry out the functions attributed to the image processing device110as described herein.

The display device140may comprise, for example, a head-mounted display device or other display device for displaying the output images received from the image processing device110. In an embodiment, the input controller150and the display device140are integrated into a head-mounted display device and the input controller150comprises a motion sensor that detects position and orientation of the head-mounted display device. The virtual perspective can then be derived to correspond to the position and orientation of the head-mounted display device such that the virtual perspective corresponds to a perspective that would be seen by a viewer wearing the head-mounted display device. Thus, in this embodiment, the head-mounted display device can provide a real-time rendering of the scene as it would be seen by an observer without the head-mounted display. Alternatively, the input controller150may comprise a user-controlled control device (e.g., a mouse, pointing device, handheld controller, gesture recognition controller, etc.) that enables a viewer to manually control the virtual perspective displayed by the display device.

FIG. 2illustrates an example embodiment of the mediated-reality system100for a surgical application. Here, an embodiment of the camera array120is positioned over the scene130(in this case, a surgical site) and can be positioned via a swing arm202attached to a workstation204. The swing arm202may be manually moved or may be robotically controlled in response to the input controller150. The display device140in this example is embodied as a virtual reality headset. The workstation204may include a computer to control various functions of the camera array120and the display device140, and may furthermore include a secondary display that can display a user interface for performing various configuration functions, or may mirror the display on the display device140. The image processing device120and the input controller150may each be integrated in the workstation204, the display device140, or a combination thereof.

FIG. 3illustrates a bottom plan view of an example embodiment of a camera array120. The camera array120include a plurality of cells202(e.g., four cells) each comprising one or more cameras122. In an embodiment, the cells202each have a hexagonal cross-section and are positioned in a honeycomb pattern. Particularly, two inner cells202-A,202-B are each positioned adjacent to other cells202along three adjacent edges, while two outer cells202-C,202-D are each positioned adjacent to other cells202along only two adjacent edges. The inner cells202-A,202-B are positioned to have respective edges adjacent to each other and may share a side wall, while the outer cells202-C,202-D are separated from each other (are not in direct contact). Here, the outer cells202-C,202-D may each have a respective pair of edges that are adjacent to respective edges of the inner cells202-A,202-B. Another feature of the illustrated cell arrangement is that the outer cells202-C,202-D each include four edges that form part of the outer perimeter of the camera array120and the inner cells202-A,202-B each include three edges that form part of the outer perimeter of the camera array120.

The hexagonal shape of the cells202provides several benefits. First, the hexagonal shape enables the array120to be expanded to include additional cells202in a modular fashion. For example, while the example camera array120includes four cells202, other embodiments of the camera array120could include, for example eight or more cells202by positioning additional cells202adjacent to the outer edges of the cells202in a honeycomb pattern. By utilizing a repeatable pattern, camera arrays120of arbitrary size and number of cameras120can be manufactured using the same cells202. Furthermore, the repeatable pattern can ensure that spacing of the cameras122is predictable, which enables the image processor120to process images from different sizes of camera arrays120with different numbers of cameras122without significant modification to the image processing algorithms.

In an embodiment, the walls of the cells202are constructed of a rigid material such as metal or a hard plastic. The cell structure provides strong structural support for holding the cameras122in their respective positions without significant movement due to flexing or vibrations of the array structure.

In an embodiment, each cell202comprises a set of three cameras122arranged in a triangle pattern with all cameras122oriented to focus on a single point. In an embodiment, each camera122is approximately equidistant from each of its neighboring cameras122within the cell202and approximately equidistant from neighboring cameras122in adjacent cells202. This camera spacing results in a triangular grid, where each set of three neighboring cameras122are arranged in triangle of approximately equal dimensions. This spacing simplifies the processing performed by the image processing device110when synthesizing the output image corresponding to the virtual camera perspective. The triangular grid furthermore allows for a dense packing of cameras122within a limited area. Furthermore, the triangular grid enables the target volume to be captured with a uniform sampling rate to give smooth transitions between camera pixel weights and low variance in generated image quality based on the location of the virtual perspective.

In an embodiment, each cell202comprises cameras122of at least two different types. For example, in an embodiment, each cell202includes two cameras122-A of a first type (e.g., type A) and one camera122-B of a second type (e.g., type B). In an embodiment, the type A cameras122-A and the type B cameras122-B have different focal lengths. For example, the type B cameras122-B may have a shorter focal length than the type A cameras122-A. In a particular example, the type A cameras122-A have 50 mm lenses while the type B cameras122-B have 35 mm lenses. In an embodiment, the type B cameras122-B are generally positioned in their respective cells202in the camera position furthest from a center point of the array120.

The type B cameras122-B have a larger field-of-view and provide more overlap of the scene130than the type A cameras122-A. The images captured from these cameras122-B are useful to enable geometry reconstruction and enlargement of the viewable volume. The type A cameras122-A conversely have a smaller field-of-view and provide more angular resolution to enable capture of smaller details than the type B cameras122-B. In an embodiment, the type A cameras occupy positions in the center of the camera array120so that when points of interest in the scene150(e.g., a surgical target) are placed directly below the camera array120, the captured images will benefit from the increased detail captured by the type A cameras122-A relative to the type B cameras122-B. Furthermore, by positioning the type B cameras122-B along the exterior of the array120, a wide baseline between the type B cameras122-B is achieved, which provides the benefit of enabling accurate stereoscopic geometry reconstruction. For example, in the cells202-A,202-C,202-D, the type B camera122-B is at the camera position furthest from the center of the array120. In the case of a cell202-B having two cameras equidistant from the center point, one of the camera positions may be arbitrarily selected for the type B camera122-B. In an alternative embodiment, the type B cameras122-B may occupy the other camera position equidistant from the center of the array120.

In an embodiment, the camera array120further includes a projector124that can project structured light onto the scene130. The projector124may be positioned near a center line of the camera array120in order to provide desired coverage of the scene130. The projector124may provide illumination and project textures and other patterns (e.g., to simulate a laser pointer or apply false or enhanced coloring to certain regions of the scene150). In an embodiment, the camera array120may also include depth sensors126adjacent to the projector124to use for depth estimation and object tracking.

FIG. 4illustrates a more detailed bottom plan view of an embodiment of a camera array120. In this view, the orientation of the cameras can be seen as pointing towards a centrally located focal point. Furthermore, in this embodiment, the type A cameras122-A are 50 mm focal length cameras and the type B cameras122-B are 35 mm focal length cameras. As further illustrated in this view, an embodiment of the camera array120may include one or more cooling fans to provide cooling to the camera array120. For example, in one embodiment, a pair of fans may be positioned in the outer cells202-C,202-D of the camera array120. In an alternative embodiment, the camera array120may incorporate off-board cooling via tubing that carries cool air to the camera array120and/or warm air away from the camera array120. This embodiment may be desirable to comply with restrictions on airflow around a patient in an operating room setting.

FIG. 5illustrates a perspective view of the camera array120. In this view, a top cover504is illustrated to cover the hexagonal cells202and provide structural support to the camera array120. Additionally, the top cover may include a mounting plate506for coupling to a swing arm202as illustrated inFIG. 2. The top cover504may further include mounting surfaces on the outer cells202-C,202-D for mounting the fans402.

Upon reading this disclosure, those of skill in the art will appreciate still additional alternative structural and functional designs for the disclosed embodiments as disclosed from the principles herein. Thus, while particular embodiments and applications have been illustrated and described, it is to be understood that the disclosed embodiments are not limited to the precise construction and components disclosed herein. Various modifications, changes and variations, which will be apparent to those skilled in the art, may be made in the arrangement, operation and details of the method and system disclosed herein without departing from the scope of the described embodiments.