Patent Publication Number: US-10334227-B2

Title: Quantitative three-dimensional imaging of surgical scenes from multiport perspectives

Description:
RELATED APPLICATIONS 
     This application is a U.S. National Stage Filing under 35 U.S.C. 371 from International Application No. PCT/US2015/023217, filed on Mar. 28, 2015, and published as WO 2015/149046 A1 on Oct. 1, 2015, which claims the benefit of priority to U.S. provisional patent application No. 61/971,749, filed on Mar. 28, 2014, and entitled “QUANTITATIVE THREE-DIMENSIONAL IMAGING OF SURGICAL SCENES”; and to U.S. provisional patent application No. 62/096,525, filed on Dec. 23, 2014, and entitled “QUANTITATIVE THREE-DIMENSIONAL IMAGING OF SURGICAL SCENES FROM MULTIPORT PERSPECTIVES, each of which is incorporated herein by reference in its entirety. 
    
    
     FIELD 
     The invention relates in general to surgical endoscopy systems having associated image sensors, and more particularly, to determining three-dimensional coordinates of physical structures displayed in surgical images. 
     BACKGROUND 
     Quantitative three-dimensional (Q3D) vision provides numerical information about the actual physical (x, y, z) 3D coordinates of target points in a real world scene. With quantitative 3D vision, a person not only can obtain a three-dimensional perception of a real world scene, but also can obtain numerical information about physical dimensions of objects in the scene and physical distances between objects in the scene. In the past, some Q3D systems have been proposed that use time-of-flight related information or phase information to determine 3D information about a scene. Other Q3D systems have used structured light to determine 3D information about a scene. 
     The use of time-of-flight information is disclosed in U.S. Pat. No. 6,323,942, entitled, “CMOS-compatible three-dimensional image sensor IC”, which discloses a three-dimensional imaging system that includes a two-dimensional array of pixel light sensing detectors fabricated on a common IC using CMOS fabrication techniques. Each detector has an associated high speed counter that accumulates clock pulses in number directly proportional to time-of-flight (TOF) for a system-emitted pulse to reflect from an object point and be detected by a pixel detector focused upon that point. The TOF data provides a direct digital measure of distance from the particular pixel to a point on the object reflecting the emitted light pulse. In a second embodiment, the counters and high speed clock circuits are eliminated, and instead each pixel detector is provided with a charge accumulator and an electronic shutter. The shutters are opened when a light pulse is emitted and closed thereafter such that each pixel detector accumulates charge as a function of return photon energy falling upon the associated pixel detector. The amount of accumulated charge provides a direct measure of round-trip TOF. 
     The use of time delay information is disclosed in U.S. Pat. No. 8,262,559, entitled, “Apparatus and method for endoscopic 3D data collection”, which discloses a modulated measuring beam and a light-transmitting mechanism for conducting the measuring beam onto an area to be observed, where the light-transmitting mechanism includes an illuminating lens, in addition to a light-imaging mechanism for imaging a signal beam from the area to be observed at least onto a phase-sensitive image sensor. Time delays, which may correspond to differences in depth in the millimeter range, result in phase information that makes possible the production of an image that depicts depth and distance information. 
     The use of structured light to determine physical coordinates of objects in a visual image is disclosed in U.S. Pat. App. Pub. No. 2012/0190923, entitled “Endoscope”; and in C. Schmalz et al., “An endoscopic 3D scanner based on structured light”, Medical Image Analysis, 16 (2012) 1063-1072. A triangulation method is used to measure the topography of a surface. Structured light in the form of projection rays, which may have a range of different color spectra, are incident upon and are reflected from a surface. The reflected rays are observed by a camera that is calibrated to use the reflected color spectra information to determine 3D coordinates of the surface. More specifically, the use of structured light typically involves shining a light pattern on a 3D surface, and determining physical distances based upon a deformation pattern of the light due to contours of the physical object. 
     An imager array camera has been built that includes a plurality of pixel arrays that can be used to compute scene depth information for pixels in the array. High resolution (HR) images are generated from multiple low resolution (LR) images. A reference viewpoint is selected and an HR image is generated as seen by that viewpoint. A parallax processing technique utilizes the effects of aliasing to determine pixel correspondences for non-reference images with respect to the reference image pixels. Fusion and superresolution are utilized to produce the HR image from the multiple LR images. See e.g., U.S. Pat. No. 8,514,491, entitled “Capturing and Processing Images using Monolithic Camera Array with Heterogeneous Imager”; U.S. Pat. App. Pub. No. 2013/0070060, entitled, “Systems and Methods for Determining Depth from multiple Views of a Scene that Include Aliasing using Hypothesized Fusion”; and K. Venkataraman et al., “PiCam: An ultra-Thin high Performance Monolithic Camera Array”. 
       FIG. 1  is an illustrative drawing showing details of a known imager sensor  180  in accordance with some embodiments. The image sensor  180  includes an arrangement of sensors  184 . Each sensor in the arrangement includes a two dimensional arrangement of pixels having at least two pixels in each dimension. Each sensor includes a lens stack  186 . Each lens stack  186  has a corresponding focal plane  188 . Each lens stack  186  creates a separate optical channel that resolves an image onto a corresponding arrangement of pixels disposed in its corresponding focal  188  plane. The pixels act as light sensors, and each focal plane  188  with its multiple pixels acts as an image sensor. Each sensor with its focal plane  188  occupies a region of the sensor arrangement different from regions of the sensor arrangement occupied by other sensors and focal planes. 
       FIG. 2  is an illustrative drawing showing a simplified plan view of the known arrangement of sensors  184  of  FIG. 1  that includes sensors labeled as sensors S 11  through S 33 . The imager sensor arrangement  184  is fabricated on a semiconductor chip to include a plurality of sensors S 11  through S 33 . Each of the sensors S 11  through S 33  includes a plurality of pixels (e.g., 0.32 megapixels) and is coupled to peripheral circuitry (not shown) that includes independent read-out control and pixel digitization. In some embodiments, the sensors S 11  through S 33  are arranged into a grid format as illustrated in  FIG. 2 . In other embodiments, the sensors are arranged in a non-grid format. For example, the sensors may be arranged in a circular pattern, zigzagged pattern, scattered pattern, or irregular pattern including sub-pixel offsets. 
     Each individual pixel of the sensors  184  of  FIGS. 1-2  includes a microlens pixel stack.  FIG. 3  is an illustrative drawing of a known microlens pixel stack of the sensors of  FIGS. 1-2 . The pixel stack  800  includes a microlens  802 , which is positioned above an oxide layer  804 . Typically beneath the oxide layer  804  there may be a color filter  806 , which is disposed above a nitride layer  808 , which is disposed above a second oxide layer  810 , which sits atop a silicon layer  812  that includes the active area  814  (typically a photodiode) of the individual pixel. The primary role of the microlens  802  is to gather the light incident on its surface and to focus that light onto the small active area  814 . The pixel aperture  816  is determined by the spread of the microlens. 
     Additional information concerning the above-described known imager sensor arrangement architecture is provided in U.S. Pat. No. 8,514,491 B2 (filed Nov. 22, 2010), and in U.S. Patent Application Pub. No. US 2013/0070060 A1 (filed Sep. 19, 2012). 
     SUMMARY 
     In one aspect, systems and methods comprising a quantitative three-dimensional (Q3D) endoscope which is inserted through a first port providing access to a body cavity to a first position at which a first Q3D image of a first portion of an anatomical structure is captured from a perspective of said endoscope. A first Q3D model is produced based upon the captured first image. The endoscope, or a second endoscope, is inserted through a second port providing access to the body cavity to a second position at which a second image of a second portion of the anatomical structure is captured. A second quantitative Q3D model is produced based upon the captured second image. The first Q3D model and the second Q3D model are combined together to produce an expanded Q3D model of the anatomical structure. The expanded Q3D model can be stored for further manipulation or can be displayed in a stereoscopic or 3D manner. The further manipulation may consist of quantitative measurements of distances between other instruments or between instruments and anatomic structures. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       Aspects of the present disclosure are best understood from the following detailed description when read with the accompanying figures. It is emphasized that, in accordance with the standard practice in the industry, various features are not drawn to scale. In fact, the dimensions of the various features may be arbitrarily increased or reduced for clarity of discussion. In addition, the present disclosure may repeat reference numerals and/or letters in the various examples. This repetition is for the purpose of simplicity and clarity and does not in itself dictate a relationship between the various embodiments and/or configurations discussed. 
         FIG. 1  is an illustrative drawing showing details of a known imager sensor. 
         FIG. 2  is an illustrative drawing showing a simplified plan view of a known sensor array of the imager sensor of  FIG. 1 . 
         FIG. 3  is an illustrative drawing of a known microlens pixel stack of a pixel within a sensor of the sensor array of  FIG. 2 . 
         FIG. 4  is an illustrative drawing showing a perspective view of a surgical scene through a viewer  312  in accordance with some embodiments. 
         FIG. 5  is an illustrative block diagram of a teleoperation surgery system to perform minimally invasive surgical procedures using one or more mechanical arms in accordance with some embodiments. 
         FIG. 6  is an illustrative perspective view of a patient-side system of the system of  FIG. 5  in accordance with some embodiments. 
         FIG. 7A  is an illustrative drawing of a first endoscope that includes a first image capture system in accordance with some embodiments. 
         FIG. 7B  is an illustrative drawing of a second endoscope that includes a second image capture system in accordance with some embodiments. 
         FIG. 8  is illustrative block diagram showing control blocks associated with the first endoscope that includes the first image capture system of  FIG. 7A  and showing the system in operation, in accordance with some embodiments. 
         FIG. 9  is an illustrative flow diagram representing a process to determine a quantitative three dimensional location of a physical target in accordance with some embodiments. 
         FIG. 10  is an illustrative flow diagram showing certain details of a process generally corresponding to module  FIG. 9  to systematically select targets in accordance with some embodiments. 
         FIG. 11  is an illustrative drawing of an example sensor imager array that includes multiple sensors and that is disposed to have a field of view that encompasses an illustrative three dimensional physical world scene that includes three illustrative objects in accordance with some embodiments. 
         FIG. 12  is an illustrative drawing representing projections of the multiple physical objects of  FIG. 11  onto multiple sensors in accordance with some embodiments. 
         FIG. 13  is an illustrative drawing indicating selection of a region of interest from within a real-world scene in accordance with some embodiments. 
         FIG. 14  is an illustrative drawing showing detail as to relative geometric offset of the projected images in sensors multiple sensors in accordance with some embodiments. 
         FIG. 15  is an illustrative drawing showing the projected images in certain example sensors within the region of interest (ROI) shifted to the right to align with the projected images in a designated reference sensor within the ROI in accordance with some embodiments. 
         FIG. 16  is an illustrative drawing showing projections of a selected target point onto multiple sensors in accordance with some embodiments. 
         FIG. 17  is an illustrative drawing showing a portion of an imager array that includes the multiple sensors of  FIG. 16  and the selected target point T disposed at location in physical space in accordance with some embodiments. 
         FIG. 18  is an illustrative elevation view of the projection of the currently selected target point T onto the multiple image sensors of  FIG. 16  in accordance with some embodiments. 
         FIG. 19  is an illustrative drawing showing the disposition of a currently selected target relative to the multiple sensors as described above with reference to  FIG. 17  and also showing y-direction pixel offsets for the candidate pixel in each of the sensors in accordance with some embodiments. 
         FIG. 20  is an illustrative flow diagram representing a first process to use Q3D information during a surgical procedure in accordance with some embodiments. 
         FIG. 21  is an illustrative drawing showing menu selections displayed on a display screen in accordance with the process of  FIG. 20  in accordance with some embodiments. 
         FIGS. 22A-22B  are illustrative drawings representing certain details of receiving user input in accordance with the process of  FIG. 20  in accordance with some embodiments. 
         FIG. 23  is an illustrative flow diagram representing a second process to use Q3D information during a surgical procedure in accordance with some embodiments. 
         FIG. 24  is an illustrative drawing showing menu selections displayed on a display screen in accordance with the process of  FIG. 23  in accordance with some embodiments. 
         FIG. 25A-25C  are illustrative drawings showing an endoscope disposed at multiple different positions to capture image information for use in producing Q3D models of an anatomical structure from the different perspectives in accordance with some embodiments. 
         FIGS. 26A-26C  are illustrative drawings representing the first, second, and third Q3D models produced using the first, second, and third image information of  FIGS. 25A-25C  in accordance with some embodiments. 
         FIG. 27  is an illustrative drawing showing example alignment of the respective Q3D models of  FIGS. 26A-26C  in accordance with some embodiments. 
         FIGS. 28A-28C  are illustrative drawings representing example rotations ( FIG. 28A ), translation ( FIG. 28B ), and resulting alignment ( FIG. 28C ) of the example second and third Q3D models in accordance with some embodiments. 
         FIG. 29  is an illustrative drawing showing an example composite Q3D model in which the multiple Q3D models of  FIGS. 25A-25C  are stitched together in accordance with some embodiments. 
         FIGS. 30A-30C  are illustrative drawings showing an endoscope in three different example “jitter” positions to capture three different image viewpoints of the target region in accordance with some embodiments. 
         FIG. 31  is an illustrative flow diagram of a process to capture a multi-perspective Q3D models in accordance with some embodiments. 
         FIG. 32  is an illustrative drawing showing details of a processor that displays a Q3D model in a 3D perspective on a 3D display in accordance with some embodiments. 
     
    
    
     DESCRIPTION OF EMBODIMENTS 
     The following description is presented to enable any person skilled in the art to create and use a surgical endoscopy system having multiple image sensors, each sensor including a pixel array that is separate from pixel arrays of other sensors, so as to determine three-dimensional coordinates of physical structures within a field of view of the image sensors. Various modifications to the embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the inventive subject matter. Moreover, in the following description, numerous details are set forth for the purpose of explanation. However, one of ordinary skill in the art will realize that the inventive subject matter might be practiced without the use of these specific details. In other instances, well-known machine components, processes and data structures are shown in block diagram form in order not to obscure the disclosure with unnecessary detail. Identical reference numerals may be used to represent different views of the same item in different drawings. Flow diagrams in drawings referenced below are used to represent processes. A computer system may be configured to perform some of these processes. Modules within flow diagrams representing computer implemented processes represent the configuration of a computer system according to computer program code to perform the acts described with reference to these modules. Thus, the inventive subject matter is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein. 
     BRIEF OVERVIEW 
     In accordance with some embodiments, an imager that includes that includes a sensor array is associated with an endoscope. The image sensor array includes multiple sensors, and each sensor includes an array of pixels. A portion of the endoscope is inserted into a human body cavity, and a target object in a field of view of the image sensor array is illuminated using a light source. A physical location and/or dimensions of the target object is determined based upon images of the target object projected onto individual sensors of the array. 
       FIG. 4  is an illustrative drawing showing a perspective view of a surgical scene through a viewer  312  in accordance with some embodiments. A viewing system having two viewing elements  401 R,  401 L can provide a good 3D viewing perspective. Numerical values representing physical dimension and/or location information for physical structures in the surgical scene are shown overlaid onto the surgical scene image. For example, a numerical distance value “d_Instr_Trgt” is shown displayed within the scene between instrument  400  and target  410 . 
     Teleoperation Medical System 
     Teleoperation refers to operation of a machine at a distance. In a minimally invasive teleoperation medical system, a surgeon may use an endoscope that includes a camera to view a surgical site within a patient&#39;s body. Stereoscopic images have been captured, which allow the perception of depth during a surgical procedure. A camera system, which is mounted on an endoscope and which includes an imager sensor array, provides quantitative three-dimensional information plus color and illumination data that can be used to generate three-dimensional images in accordance with some embodiments. 
       FIG. 5  is an illustrative block diagram of a teleoperation surgery system  100  to perform minimally invasive surgical procedures using one or more mechanical arms  158  in accordance with some embodiments. Aspects of system  100  includes telerobotic and autonomously operating features. These mechanical arms often support an instrument. For instance, a mechanical surgical arm (e.g., the center mechanical surgical arm  158 C) may be used to support an endoscope with a stereo or three-dimensional surgical image capture device  101 C, such as an endoscope associated a Q3D image sensor array. The mechanical surgical arm  158 C may include a sterile adapter, or a clamp, clip, screw, slot/groove, or other fastener mechanism to mechanically secure an endoscope that includes the image capture device  101 C to the mechanical arm. Conversely, the endoscope with image capture device  101 C may include physical contours and/or structures complementary to those of the mechanical surgical arm  158 C so as to securely interfit with them. 
     A user or operator O (generally a surgeon) performs a minimally invasive surgical procedure on patient P by manipulating control input devices  160  at a master control console  150 . The operator can view video frames of images of a surgical site inside a patient&#39;s body through a stereo display device  164 , which includes the viewer  312  described above with reference to  FIG. 4 . A computer  151  of the console  150  directs movement of teleoperationally controlled endoscopic surgical instruments  101 A- 101 C via control lines  159 , effecting movement of the instruments using a patient-side system  152  (also referred to as a patient-side cart). 
     The patient-side system  152  includes one or more mechanical arms  158 . Typically, the patient-side system  152  includes at least three mechanical surgical arms  158 A- 158 C (generally referred to as mechanical surgical arms  158 ) supported by corresponding positioning set-up arms  156 . The central mechanical surgical arm  158 C may support an endoscopic camera  101 C suitable for capture of Q3D information for images within a field of view of the camera. The mechanical surgical arms  158 A and  158 B to the left and right of center may support instruments  101 A and  101 B, respectively, which may manipulate tissue. 
       FIG. 6  is an illustrative perspective view of the patient-side system  152  in accordance with some embodiments. The patient-side system  152  comprises a cart column  170  supported by a base  172 . One or more mechanical insertion surgical arms/links  158  are respectively attached to one or more set-up arms  156  that are a part of the positioning portion of the patient-side system  152 . Situated approximately at a central location on base  172 , the cart column  170  includes a protective cover  180  that protects components of a counterbalance subsystem and a braking subsystem from contaminants. 
     Excluding a monitor arm  154 , each mechanical surgical arm  158  is used to control instruments  101 A- 101 C. Moreover, each mechanical surgical arm  158  is coupled to a set-up arm  156  that is in turn coupled to a carriage housing  190  in one embodiment of the invention. The one or more mechanical surgical arms  158  are each supported by their respective set-up arm  156 , as is illustrated in  FIG. 6 . 
     The mechanical surgical arms  158 A- 158 D may each include one or more displacement transducers, orientational sensors, and/or positional sensors  185  to generate raw uncorrected kinematics information to assist in initial acquisition by a tracking system and tracking of instruments. The instruments may also include a displacement transducer, a positional sensor, and/or orientation sensor  186  in some embodiments of the invention. Moreover, one or more instruments may include a marker  189  to assist in acquisition and tracking of the instruments. 
     Additional information about a teleoperation medical system is provided in U.S. Patent Application Pub. No. US 2012/0020547, (filed Sep. 30, 2011). 
     Endoscopic Imager System 
       FIG. 7A  is an illustrative drawing of a first endoscope with a first image capture system  101 C in accordance with some embodiments. The image capture system  101 C includes an endoscope that includes elongated portion  202 , which includes a first end portion  204  and a second end portion  206  and a tip portion  208  of the first end portion  204 . The first end portion  204  is dimensioned to be inserted into a human body cavity. A sensor array  210 , which includes multiple image sensors (not shown), is coupled at the tip portion  208  of the first end portion  204 . In accordance with some embodiments, each sensor in the sensor array  210  includes an array of pixels. The elongated portion  202  has a length sufficient to position the tip portion  208  close enough to a target object within the body cavity so that the object can be imaged by the imager sensor array  210 . In accordance with some embodiments, the second end portion  206  may include physical contours and/or structures (not shown), as generally described above, so as to securely interfit with a mechanical arm (not shown). The elongated portion  202  also includes one or more electronic signal paths  212  to electronically communicate information with the imager sensor array  210 . A light source  214  is disposed to illuminate the object to be imaged. In accordance with some embodiments, the light source  214  can be unstructured light, white light, color filtered light, or light at some selected wavelength, for example. In accordance with some embodiments the light source  214  is located at tip  208 , and in other embodiments it is optionally located separately from endoscope  101 C. 
       FIG. 7B  is an illustrative drawing of a second endoscope with a second image capture system  101 C 2 , in accordance with some embodiments. Aspects of the second image capture system  101 C 2  that are essentially the same as those of the first endoscope with the first image capture system  101 C are indicated by identical reference numerals and are not described again. An input to a light pipe input, such as a rod lens, is disposed at the tip portion  208  of the first end portion  204 . A light pipe body extends within the elongate portion  202  so as to transmit an image received as the light pipe input to the imager sensor array  210 , which is physically displaced from the tip portion  208 . In some embodiments, the imager sensor array  210  is displaced far enough from the tip portion  208  so that the imager sensor array  210  is located outside the body cavity during observation of objects within the cavity. 
       FIG. 8  is illustrative block diagram showing control blocks associated with the first endoscope  101 C with the first image capture system  101 C of  FIG. 7A  and showing the system in operation, in accordance with some embodiments. Images captured by the imager sensor array  210  are sent over a data bus  212  to a video processor  104 , which communicates via bus  105  with a controller  106 . The video processor  104  may comprise a camera control unit (CCU) and a video signal detector (VSD) board. The CCU programs or controls various settings of the imaging sensor  210 , such as brightness, color scheme, white balance, etc. The VSD processes the video signal received from the imaging sensor. Alternatively, the CCU and VSD are integrated into one functional block. 
     In accordance with some embodiments a processor system that includes one or more than one processor is configured to perform processor functions. In some embodiments the processor system includes multiple processors configured to operate together to perform the processor functions described herein. Thus, reference herein to at least one processor configured to perform one or more functions includes a processor system in which the functions may be performed by one processor alone or by multiple processors working together. 
     In one implementation, the controller  106 , which includes a processor and a storage device (not shown) computes the physical quantitative 3D coordinates of the points in a scene adjacent the tip  208  of the elongate portion  202  and drives both the video processor  104  and a 3D display driver  109  to compose 3D scenes, which then can be displayed on a 3D display  110 . In accordance with some embodiments, Q3D information about a surgical scene is generated, such as numerical indicia of dimensions of surface contours of objects in a scene or distances from objects within the surgical scene, for example. As explained more fully below, the numerical Q3D depth information can be used to annotate a stereoscopic image of a surgical scene with distance information or surface contour information. 
     Data buses  107  and  108  exchange information and control signals among the video processor  104 , the controller  106 , and the display driver  109 . In some embodiments, these elements can be integrated with the image sensor array  210  inside the body of the endoscope. Alternatively, they can be distributed internally and/or externally to the endoscope. The endoscope is shown positioned, via a cannula  140 , to penetrate body tissue  130  in order to provide visualize access to a surgical scene that includes a target  120 . Alternatively, the endoscope and one or more instruments may also pass through a single opening—a single incision or natural orifice—to reach a surgical site. The target  120  can be an anatomic target, another surgical instrument, or any other aspect of the surgical scene inside a patient&#39;s body. 
     An input system  112  receives the 3D visual representation and provide it to processor  106 . The input system  112  may include a storage device coupled to an electronic communication bus (not shown) that receives a 3D model such as a CRT or MRI from a system (not shown) that generates the 3D model. Processor  106 , for example, can be used to compute the alignment intended between the Q3D model and the 3D visual representation. More particularly, without limitation, input system  112  may include a processor configured to establish an Ethernet communication connection between system  152  and an imaging system (not shown), such as a MRI, CT or ultrasound imaging system. Other imaging systems may be used. Other types of communication connections may be used, such as Bluetooth, WiFi, optical, etc. Alternatively, system  152  and the imaging system may be integrated in one larger system. The result of the alignment process may be saved in the storage device associated with processor  106 , provided for further manipulation to external devices or system or displayed as shown in  FIG. 25 . 
     Example of Q3D Information Added to an Image of a Scene 
     Referring once again to  FIG. 4  is an illustrative drawing showing a perspective view of a viewer  312  of the master control console  150  of  FIG. 5  in accordance with some embodiments. In accordance with some embodiments, to provide a three-dimensional perspective, the viewer  312  includes stereo images for each eye. As shown, a left image  400 L and a right image  400 R of the surgical site include any instruments  400  and a target  410  respectively in a left viewfinder  401 L and a right viewfinder  401 R. The images  400 L and  400 R in the viewfinders may be provided by a left display device  402 L and a right display device  402 R, respectively. The display devices  402 L, 402 R may optionally be pairs of cathode ray tube (CRT) monitors, liquid crystal displays (LCDs), or other type of image display devices (e.g., plasma, digital light projection, etc.). In the preferred embodiment of the invention, the images are provided in color by a pair of color display devices  402 L,  402 R; such as color CRTs or color LCDs. To support backward compatibility with existing devices, stereoscopic display devices  402 L and  402 R may be used with a Q3D system. Alternatively, the Q3D imaging system can be connected to 3D monitors, 3D TVs, or to autostereoscopic displays, such as a display that does not require use of 3D effect eye glasses. 
     A viewing system having two viewing elements  401 R,  401 L can provide a good 3D viewing perspective. The Q3D imaging system supplements this viewing perspective with physical dimension information for physical structures in the surgical scene. The stereo viewer  312  used in conjunction with a Q3D endoscopy system, can display Q3D information overlayed onto the stereo image of the surgical scene. For example, as shown in  FIG. 4 , the numerical Q3D distance value “d_Instr_Trgt” between instrument  400  and target  410  can be displayed within stereo viewer  312 . 
     An explanation of a video stereo viewing system that can be used to overlay physical location and dimension information onto a 3D perspective of a surgical scene is provided in U.S. Patent Application Pub. No. US 2012/0020547, (filed Sep. 30, 2011), paragraphs [0043]-[0053] and corresponding drawings, which is expressly incorporated herein by reference. 
     Processing Quantitative Three-Dimensional Physical Information 
       FIG. 9  is an illustrative flow diagram representing a process to determine a quantitative three-dimensional location of a physical target in accordance with some embodiments. The process is described with reference to the endoscope with image capture system  101 C of the embodiment of  FIG. 8 . Module  401  configures the controller  106  to acquire video data from imaging sensors S ij . It will be appreciated that although the image sensor array  210  “images” an entire field of view, different sensors and different pixels within different sensors in image sensor array  210  may be illuminated by image projections from different object points within the field of view. The video data, for example, may include color and light intensity data. Each pixel of each sensor may provide one or more signals indicative of the color and intensity of an image projected onto it. Module  402  configures the controller to systematically select targets from a selected region of interest in a physical world view. Module  403  configures the controller to commence the computation of the target 3D coordinates (x, y, z) with an initial (x 0 , y 0 , z 0 ) set. The algorithm then checks the coordinates for consistency, by using image diversity data from all sensors S ij  that receive a projected image of the target. The coordinate computation is refined at decision module  404  until an acceptable accuracy is reached. Decision module  404  also configures the controller to determine whether the currently computed physical location is sufficiently accurate. In response to a determination that the currently computed location is not accurate enough, control flows back to module  403  to try a different possible physical location. In response to a determination that the currently computed location is sufficiently accurate, module  405  configures the controller to determine whether the entire region of interest has been scanned. In response to a determination that the entire region of interest has not been scanned, control flows back to module  402  and a different target is selected. In response to a determination that the entire region of interest has been scanned, control flows to module  406 , which configures the controller to assemble a three-dimensional model of the imaging volume of interest. Assembly of a 3D image of a target based upon three-dimensional information indicating the physical position of structures of the target is known to persons of ordinary skill in the art and need not be described herein. Module  407  configures the controller to store the 3D model developed using the physical position information determined for multiple targets for further review and manipulation. For example, the 3D model could be used at a later time for surgical applications, such as sizing an implant for the particular dimensions of a patient&#39;s organ. In yet a different example, when a new surgical instrument  101  is installed on the robotic system  152 , it may be necessary to call back the 3D model and display it on display  110  in order to reference the new instrument to the previous surgical scene. Module  407  may also store the result of the alignment between the 3D visual representation and the Q3D model. Module  408  configures the controller to use the physical position information determined for multiple targets to display a quantitative 3D view. An example of a Q3D view is the distance value “d_Instr_Trgt” shown in  FIG. 4 . 
     It is noted that a stereoscopic display creates the illusion of viewing in three dimensions. However, an actual 3D display presents a 3D image, such as a holographic image or an image projected onto a curved surface. Typically, a 3D display allows the view to move to change viewing perspective. 
       FIG. 10  is an illustrative flow diagram showing certain details of a process generally corresponding to module  402  of  FIG. 9  in accordance with some embodiments. Module  402 . 1  configures the controller to capture images of a physical world scene from all sensors in the sensor array  210 . Module  402 . 2  configures the controller to specify a region of interest from within the captured scene. Module  402 . 3  configures the controller to search for a best match as between scene images within the region of interest so as to identify pixel locations in different sensors that are illuminated by projections of the same target. As explained later, the best matching may be achieved, without limitation, by shifting the individual images from sensors S ij  until maximizing two-dimensional cross-correlation function between the shifted image and a reference image. The reference image, for example, may be the scene image received from sensor S 11 . Module  402 . 4  configures the controller to identify candidate pixels illuminated by projections from the same target. Module  402 . 5  configures the controller to compute two or more pixel coordinates (N x , N y ) coordinates for the selected target to determine whether the candidate pixels are illuminated by a projection from the same target. Decision module  402 . 6  determines whether the computed 2D pixel coordinate values indicate that the candidate pixels are illuminated by a projection from the same target. The image diversity caused by viewing the same scene with multiple sensors S ij  plays a role in correctly identifying (N x , N y ) associated with a specific target in the various individual images S ij . For example, in accordance with some embodiments, assuming a simplified scenario where only three sensors are used, S 11 , S 12  and S 13 , if the triplet of 2D pixel coordinates [(Nx 11 , Ny 11 ), (Nx 12 , Ny 12 ), (Nx 13 , Ny 13 )] are not corresponding to projections of the same target onto [S 11 , S 12  and S 13 ] then the quantities ŷ 12  and ŷ 13  (which are estimates of the projection shift in the y direction) will yield different values. According the equations presented later, ŷ 12  and ŷ 13  should be equal if pixel coordinates (Nx 11 , Ny 11 ), (Nx 12 , Ny 12 ), (Nx 13 , Ny 13 ) come from projections of the same target. 
     
       
         
           
             
               
                 
                   
                     
                       y 
                       ^ 
                     
                     12 
                   
                   = 
                   
                     
                       Ny 
                       11 
                     
                     
                       
                         Ny 
                         11 
                       
                       - 
                       
                         Ny 
                         12 
                       
                     
                   
                 
               
               
                 
                   ( 
                   
                     402.5 
                     ⁢ 
                     
                       - 
                     
                     ⁢ 
                     1 
                   
                   ) 
                 
               
             
             
               
                 
                   
                     
                       y 
                       ^ 
                     
                     13 
                   
                   = 
                   
                     2 
                     · 
                     
                       
                         Ny 
                         11 
                       
                       
                         
                           Ny 
                           11 
                         
                         - 
                         
                           Ny 
                           13 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   
                     402.5 
                     ⁢ 
                     
                       - 
                     
                     ⁢ 
                     2 
                   
                   ) 
                 
               
             
           
         
       
     
     If ŷ 12  and ŷ 13  are not approximately equal then control flows back to module  402 . 4  and to refine the best candidates for target projections onto sensor planes S ij . As mentioned, the above is just a simplified implementation of the algorithm. In general, as shown in  FIG. 10  module  402 . 6 , the norm of the difference between ŷ ij  and ŷ i,j+1  should be less than an acceptable tolerance ε in order for module  402  to complete its iterations. A similar restriction should be met for the corresponding estimates for the x axis, {circumflex over (x)} i,j  and x i,j+1 . In response to a determination that the computed 2D pixel coordinate values (N x , N y ) do indicate that the candidate pixels are illuminated by a projection from the same target, then control flows to module  403 . 
     It will be appreciated that each pixel directly captures color and intensity information from a world scene. Moreover, in accordance with the above process, each pixel is associated with the (x, y, z) coordinates of the physical object in the world view that is projected onto the pixel. Thus, color information, illumination intensity information and physical location information, i.e. the location of the physical object that projected the color and illumination, can be associated with a pixel in a non-transitory computer readable storage device. The following Table 1 illustrates this association. 
     
       
         
           
               
               
               
               
             
               
                 TABLE 1 
               
               
                   
               
             
            
               
                 Pixel Identifier 
                 Color Value 
                 Intensity Value 
                 Location (x, y, z) 
               
               
                   
               
            
           
         
       
     
     Examples of Determining Q3D Information 
     Example of Projection Matching 
       FIG. 11  is an illustrative drawing of an example sensor array  210  that includes an array of sensors sensors S 11 -S 33  that is disposed to have a field of view that encompasses an illustrative three-dimensional physical world scene that includes three illustrative objects in accordance with some embodiments. Each sensor S ij  in the array includes a two-dimensional arrangement of pixels having at least two pixels in each dimension. Each sensor includes a lens stack that creates a separate optical channel that resolves an image onto a corresponding arrangement of pixels disposed in a focal plane of the lens stack. Each pixel acts as a light sensor, and each focal plane with its multiple pixels acts as an image sensor. Each sensor S 11 -S 33  with its focal plane occupies a region of the sensor array different from regions of the sensor array occupied by other sensors and focal planes. Suitable known image sensor arrays are disclosed in U.S. Patent No. U.S. Pat. No. 8,514,491 (filed Nov. 22, 2010) and in U.S. Patent Application Pub. No. US 2013/0070060 (filed Sep. 19, 2012), which are described above. 
     In accordance with some embodiments, the sensors are characterized by a N x  and N y , their total number of pixels in the x and y directions, and by field of view angles, θ x  and θ y . In some embodiments, the sensor characteristics for the x and y axes are expected to be the same. However, in alternative embodiments, the sensors have asymmetric x and y axis characteristics. Similarly, in some embodiments, all sensors will have the same total number of pixels and the same field of view angle. The sensors are distributed across the sensor array  210  in a well-controlled manner. For example, the sensors may be at δ distance apart on the two-dimensional grid shown. The sensor placement pitch δ may be symmetric or asymmetric across such grid. 
     In the embodiment shown in  FIG. 11 , the sensors are arranged in a rectangular grid in which sensors S 11 -S 13  occupy a top row, sensors S 21 -S 23  occupy a middle row, and sensors S 31 -S 33  occupy a bottom row. Each sensor includes N rows of pixels and N columns of pixels. Light rays, indicated by dashed lines, produced by a light source are reflected from each of a triangular-shaped first object, a spherical-shaped second object, and a rectangular-shaped third object, to each sensor of the imager array. For illustration purposes, only rays to sensors S ij , S 11  and S 13  in the top row are shown. The light source may be non-structured white light or ambient light, for example. Alternatively, the light source may provide light at a selected wavelength, such as in the visible or infrared spectrums, or the light may be filtered or split to provide a selected wavelength (e.g., color) or range of wavelengths (e.g., range of colors), for example. It will be appreciated that light rays are similarly reflected from each of the objects to sensors S 21 -S 33 . However, in order to simplify the explanation, these other light rays are not shown. 
     In accordance with modules  401  and  402 . 1 , sensors of the sensor array  210  separately capture images from a world view.  FIG. 12  is an illustrative drawing representing projections of the three objects of  FIG. 11  onto the sensors S ij  (only S 11 , S 12 , and S 13  are shown) in accordance with some embodiments. A person of ordinary skill in the art will appreciate that the reflected light rays incident upon the sensors project images of the objects that are in the field of view. More specifically, the rays of light reflected from the objects in the field of view that are incident upon multiple different image sensors of the imager array produce multiple perspective projections of the objects from three dimensions to two dimensions, i.e. a different projection in each sensor that receives the reflected rays. In particular, the relative location of projections of the objects is shifted from left to right when progressing from S 11  to S 12  to S 13 . Image sensor pixels that are illuminated by incident light rays produce electrical signals in response to the incident light. Accordingly, for each image sensor, a pattern of electrical signals is produced by its pixels in response to the reflected rays that indicates the shape and location of the image projection within that image sensor. 
     In accordance with module  402 . 2 , a region of interest is selected from the world scene.  FIG. 13  is an illustrative drawing indicating selection of a region of interest from within the scene. In this example, the triangular-shaped first object, spherical-shaped second object, and rectangular-shaped third object all are in the selected region of interest. This step can be achieved by accepting input from an operator, or it can be automatically performed using a computer configured by software in a prescribed manner, or by combination of operator inputs and automatic software-controlled selection. For example, in some embodiments, the world scene may show an internal cavity of the human anatomy, and the objects may be internal body organs, or surgical instruments, or portions thereof. A surgeon may receive real time visual imagery from within the internal cavity and may see tissue regions of the human anatomy and a portion of the surgical instruments projecting within the body cavity. The surgeon may specify those objects within the field of view for which location information is to be determined through well-known techniques, such as a telestration video marker. Alternatively or in addition to such operator request, an automated process such as an edge detection algorithm can be used to specify a region of interest (ROI). 
     In accordance with module  402 . 3 , a best match is determined between scene images within the region of interest so as to identify pixel locations in different sensors that are illuminated by projections of the same target.  FIG. 14  is an illustrative drawing showing additional detail about relative geometric offset of the projected images in sensors S 11 , S 12 , and S 13  in accordance with some embodiments. In accordance with some embodiments, an image from sensor S 13  is considered to be reference image, and the projections of the objects in the selected ROI are offset to the right by an amount σ 23  pixels in sensor S 12  relative to their location in sensor S 13 . Similarly, the projections of the objects in the selected ROI are offset to the right by an amount σ 0  pixels in sensor S 11  relative to their location in sensor S 13 . It will be appreciated that since the FOV viewing axes of sensors S 12 , S 11  each is offset to the right of the FOV viewing axis of sensor S 13  (such viewing axes being perpendicular to plane of the sensors), the projected images from ROI are offset to the left in the sensors S 13  and S 11  relative to sensor S 11 . 
       FIG. 15  is an illustrative drawing showing the projected images in sensors S 11  and S 12  within the ROI shifted to the right to align with the projected images in sensor S 13  within the ROI in accordance with some embodiments. In the current example, sensor S 13  is designated to act as a reference sensor. It will be appreciated that other sensors can be chosen for use in determining alignment and geometric dimensions. Projections of the objects within the selected ROI are identified in the designated sensor, e.g., sensor  513 , and projections in the other sensors, e.g., in sensors S 11  and S 12 , are shifted until they align with the projection in the designated sensor. In this manner, the corresponding projections of objects within the selected ROI can be identified within the other sensors, together with their offsets relative to the location of the projections in the designated sensor. 
     In particular, for example, the projections of the three example objects are shifted to the right by an amount σ 23  pixels in sensor S 12 , and the projections of the three example objects are shifted to the right by an amount σ 13  pixels in sensor S 13 . In this illustrative example, in order to simplify the explanation, it is assumed that the projections are offset in the y direction only and not in the x direction, although the same principles apply for x direction projection offsets as between sensors. Moreover, although this example shows a linear offsets, a person of ordinary skill in the art can apply other transformations such as rotation, for example, to align projections that have relative offsets in different sensors. 
     In accordance with some embodiments for example, two-dimensional (2D) cross-correlation techniques or principal component analysis (PCA), can be used to align the projections within the ROI in S 13  with the projections within the ROI in S 12  and to align the projections within the ROI in S 13  with the projections within the ROI in S 11 . In general, the intent is to best match or align the images from sensors S ij  with respect to the image from the sensor designated as reference. More specifically, the projected images within the ROI in S 12  are shifted and cross-correlated with the projected images within the ROI in S 13  until a highest correlation coefficient is achieved. Likewise, the projected images within the ROI in S 11  are shifted and cross-correlated with the projected images within the ROI in S 13  until a highest correlation coefficient is achieved. Thus, alignment of the projections of the ROI is used to identify the locations of the projections of the ROI in sensors S 11  and S 12  by determining the offset between the projection of the ROI in S 13  and the projection of the ROI in S 12  and by determining the offset between the projection of the ROI in S 13  and the projection of the ROI in S 11 . 
     Example of Candidate Pixel Selection and Refinement 
     In accordance with module  402 . 4 , candidate pixels are identified within different sensors, which according to the best match process, are illuminated by projections from the same target. Once the projections of objects within the ROI have been identified in each of the sensors S 11 , S 12 , and S 13 , the physical (x, y, z) projections of individual target points within the ROI can be determined relative to the imager array. In accordance with some embodiments, for each of a multiplicity of target points within the ROI, one or more pixels within each of multiple sensors is identified that is illuminated by a projection from the target point. For each such target point, a physical (x, y, z) target point location is determined based at least in part upon the geometric relationships among pixels disposed in different sensors that are determined to be illuminated by projections from the target point. 
     It will be appreciated that a sequence of target points can be chosen automatically by systematically traversing the ROI (e.g., right to left with a certain step size and up to down with a step size), and a physical (x, y, z) target point location can be determined for each selected target point. Since S 11  and S 12  are best matched to S 13 , the traversing is performed inside the shifted regions of interest. Selecting a target involves identifying a pixel in each of sensors S 11 , S 12 , and S 13  that is illuminated by a projection of the target. Thus, candidate pixels in each of S 11 , S 12 , and S 13  are identified as being the ones illuminated by a projection of the selected target point. 
     In other words, in order to select a target point T, a pixel is selected in each of the sensors S 11 , S 12 , and S 13  that is illuminated by a projection of the target point T. It will be appreciated that the (x, y, z) physical location of the target T is unknown at the moment of its selection. Moreover, it will be appreciated that inaccuracy of the above-described alignment process can result in inaccuracy in the determination of which pixels in each sensor are illuminated by the projection of a selected target T. Thus, as explained with reference to  FIGS. 17, 18, and 19 , a further determination is made as to the accuracy of the determination as to the pixels in each of S 11 , S 12 , and S 13  that are illuminated by the projection of a currently selected target T. 
     Continuing with the above example, assume that the triangular-shaped first object is the currently selected target point.  FIG. 16  is an illustrative drawing showing projections of the selected triangle shaped target point onto sensors S 11 , S 12 , and S 13  in accordance with some embodiments. From these projections, the 2D pixel coordinates for target T are determined, [(Nx 11 , Ny 11 ), (Nx 12 , Ny 12 ), (Nx 13 , Ny 13 )]. For simplification,  FIG. 16  shows only the y-axis pixel coordinates. Using these 2D pixel coordinates, expressions ( 402 . 5 - 1 ) and ( 402 . 5 - 2 ) are applied and ŷ 12  and ŷ 13  computed as part of module  402 . 5 . As part of module  402 . 6 , the norm |ŷ 12 −ŷ 13  is computed and compared to the acceptable tolerance C. Similarly, the x-axis pixel coordinates and location estimates are computed and compared against acceptable tolerances. If the condition of module  402 . 6  is met, then the process proceeds to module  403 . Else, it returns to module  402 . 4  to further refine target candidates. 
     Referring to  FIG. 17 , there is shown a portion of an imager array that includes sensors S 11 , S 12 , and S 13  and the selected triangular-shaped first object target point T disposed at location (x, y, z) in physical space. Sensors within an imager array have a known spacing between them, δ ij . The physical position spacing between S 11  and S 12  is δ 12 , and the physical position spacing between S 12  and S 13  is δ 23 . In some embodiments the spacing between all sensors S ij  is identical, equal to δ, a constructional specification. Sensors S ij  also have a known field of view angle θ. 
     As explained above, in some embodiments, each sensor is constructed as a 2D imaging element with pixels arranged in a rectangular pattern of rows and columns. Alternatively, pixels can be arranged in a circular pattern, zigzagged pattern, scattered pattern, or an irregular pattern including sub-pixel offsets, for example. The angle and the pixel characteristics of these elements may be identical or, alternatively, may be different from sensor to sensor. However, these characteristics are assumed to be known. In order to simplify the explanation, it is assumed that the sensors are identical, although they may, however, be different. 
     For simplicity, let us assume that all sensors S ij  have N×N pixels. At a distance z from sensor S 11 , the N-pixel width of the sensor expands out to a y-dimension field of view of S 11  indicated by FOV 1 . Likewise, at a distance z from sensor S 12 , the y-dimension field of view of sensor S 12  is indicated by FOV 2 . Also, at a distance z from sensor S 13 , the y-dimension field of view of sensor S 13  is indicated by length FOV 3 . The lengths FOV 1 , FOV 2 , and FOV 3  overlap each other, signifying that sensors S 11 , S 12 , and S 13  achieve a 3-way sampling diversity of target T physically located at some (unknown) distance z. Of course, if the sensors are identically built, as assumed in this example, length FOV 1 , FOV 2 , and FOV 3  will be identical as well. It will be appreciated that the three lengths FOV 1 , FOV 2 , and FOV 3  all have the same magnitude and are coplanar in that they are at the same (unknown) z-distance from the imager array, although for the purpose of illustration they are portrayed as if they were stacked adjacent to each other. 
     Referring to  FIG. 18 , there is shown an illustrative elevation view of the projection of the currently selected target point T onto the image sensors S 11 , S 12 , and S 13 . For the sake of simplicity, it is assumed that the sensors include geometrically rectangular pixel arrays of size N×N pixels. It is also assumed that the x coordinates of the target T projections are all equal. In other words, it is assumed that for the projections of target T onto S 11 , S 12 , and S 13 , n x1 =n x2 =n x3 . To simplify the explanation, it is also assumed that the geometric field of view angle θ is the same horizontally as it is vertically, θ x =θ y . A person of skill in the art would know how to modify the process presented below so that to compute the x, y, and z physical coordinates of target T if any of the above assumptions would change. 
     An image of the target T is projected to a physical point within sensor S 11  at geometric coordinates (n x1 , n y1 ), in the plane of the image sensor S 11 . More specifically, the projection of target point T onto sensor S 11  is located n y1  pixels along the y axis, and n x1  pixel along the x axis, taken from the origin. An image of the target T is projected to a physical point within sensor S 12  at geometric coordinates (n x2 , n y2 ) in the plane of the image sensor S 12 . An image of the target T is projected to a physical point within sensor S 13  at geometric coordinates (n x3 , n y3 ) in the plane of the image sensor S 13 . It will be appreciated that pixel locations (n x1 , n y1 ) within each sensor are determined relative to origin (0, 0) reference coordinates provided for the sensor. As shown in  FIG. 17  or  FIG. 19 , a global system of coordinates (x, y, z) is defined and used to reference the target. For example, the origin of such system of coordinates may be placed, without limitations, at the geometrical center of sensor S 11 . 
     Referring to both  FIG. 16  and  FIG. 18 , it can be seen that they pixel distance of the projection of the target is different in each sensor. The projection of a currently selected target T is disposed n y1  pixels to the left of the origin in S 11 . The projection of the selected target T is disposed n y2  pixels to the left of the origin in S 12 . The projection of the selected target T is disposed n y3  pixels to the left of the origin in S 13 . As mentioned above, to simplify the explanation, it is assumed that the projection of the target falls at the same x pixel distance from the origin in all three sensors. 
     Referring to  FIG. 19 , there is shown the disposition of the currently selected target T relative to sensors S 11 , S 12  and S 13  as described above with reference to  FIG. 17  and also showing y-direction pixel offsets for the candidate pixel in each of the sensors. It will be understood that the drawings of  FIG. 19  present physical structures and an analytical framework for determining the (x, y, z) physical coordinates of the selected target point T. At an (unknown) distance z from the imager array plane, the y-direction field of view for each sensor extends over a length marked as FOV i . This length, FOV i , corresponds to the maximum pixel width of the sensor, which is N pixels, in some embodiments. Given that the working assumption was that the sensor has a field of view that is symmetric in the x and y directions, the length would also be FOV i  vertically, along the x axis. 
     Recall that the candidate pixel selections are made based at least in part upon a correlation process that can have a level of uncertainty than can result in inaccuracy in determination of the physical location of the selected target. Thus, a further check of the accuracy of the target projection candidate selections, in accordance with some embodiments, is made as follows. 
     Example of Determining Target&#39;s Physical (x, y) Location and Checking Accuracy of Target Projection Candidate Selection 
     In accordance with module  402 . 5 , two or more two-dimensional (N x , N y ) coordinate values are computed for the selected target to determine whether the candidate pixels actually are illuminated by a projection from the same target. Based on the assumptions discussed above and placing the origin of the 3D system of coordinates at the center of sensor S 11 , the imager array and currently selected target T in the example in  FIG. 19  have the following relationships: 
     
       
         
           
             
               
                 
                   z 
                   = 
                   
                     
                       N 
                       · 
                       δ 
                     
                     
                       2 
                       · 
                       
                         ( 
                         
                           
                             n 
                             
                               y 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               1 
                             
                           
                           · 
                           
                             n 
                             
                               y 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               2 
                             
                           
                         
                         ) 
                       
                       · 
                       
                         tan 
                         ⁡ 
                         
                           ( 
                           
                             θ 
                             2 
                           
                           ) 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
             
               
                 
                   y 
                   = 
                   
                     
                       
                         
                           2 
                           ⁢ 
                           
                             n 
                             
                               y 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               1 
                             
                           
                         
                         - 
                         N 
                       
                       
                         2 
                         ⁢ 
                         
                           ( 
                           
                             
                               n 
                               
                                 y 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 1 
                               
                             
                             - 
                             
                               n 
                               
                                 y 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 2 
                               
                             
                           
                           ) 
                         
                       
                     
                     · 
                     δ 
                   
                 
               
               
                 
                   ( 
                   2 
                   ) 
                 
               
             
             
               
                 
                   x 
                   = 
                   
                     
                       ( 
                       
                         
                           
                             2 
                             ⁢ 
                             
                               n 
                               
                                 x 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 1 
                               
                             
                           
                           N 
                         
                         - 
                         1 
                       
                       ) 
                     
                     · 
                     z 
                     · 
                     
                       tan 
                       ⁡ 
                       
                         ( 
                         
                           θ 
                           2 
                         
                         ) 
                       
                     
                   
                 
               
               
                 
                   ( 
                   3 
                   ) 
                 
               
             
           
         
       
         
         
           
             Where: 
             N is the pixel dimension of the imaging sensors; 
             n x1  is the position of a target point T expressed in number of pixels from the origin of the S 11  plane in the x direction; 
             n y1  is the position of the target point T expressed in number of pixels from the origin of the S 11  plane in the y direction; 
             n y2  is the position of the target point T expressed in number of pixels from the origin of the S 12  plane in the y direction; and 
             n y2  is the position of the target point T expressed in number of pixels from the origin of the S 12  plane in the y direction; 
             θ is the angle of the field of view. 
             Moreover, if performing the same math using sensors S 11  and S 13  and given that the separation between S 11  and S 13  is 28, we obtain: 
           
         
       
    
     
       
         
           
             
               
                 
                   z 
                   = 
                   
                     
                       2 
                       · 
                       N 
                       · 
                       δ 
                     
                     
                       2 
                       · 
                       
                         ( 
                         
                           
                             n 
                             
                               y 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               1 
                             
                           
                           - 
                           
                             n 
                             
                               y 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               3 
                             
                           
                         
                         ) 
                       
                       · 
                       
                         tan 
                         ⁡ 
                         
                           ( 
                           
                             θ 
                             2 
                           
                           ) 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   4 
                   ) 
                 
               
             
             
               
                 
                   y 
                   = 
                   
                     
                       
                         
                           
                             2 
                             ⁢ 
                             
                               n 
                               
                                 y 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 1 
                               
                             
                           
                           - 
                           N 
                         
                         
                           2 
                           ⁢ 
                           
                             ( 
                             
                               
                                 n 
                                 
                                   y 
                                   ⁢ 
                                   
                                       
                                   
                                   ⁢ 
                                   1 
                                 
                               
                               - 
                               
                                 n 
                                 
                                   y 
                                   ⁢ 
                                   
                                       
                                   
                                   ⁢ 
                                   3 
                                 
                               
                             
                             ) 
                           
                         
                       
                       · 
                       2 
                     
                     ⁢ 
                     δ 
                   
                 
               
               
                 
                   ( 
                   5 
                   ) 
                 
               
             
             
               
                 
                   x 
                   = 
                   
                     
                       
                         ( 
                         
                           
                             
                               2 
                               ⁢ 
                               
                                 n 
                                 
                                   x 
                                   ⁢ 
                                   
                                       
                                   
                                   ⁢ 
                                   3 
                                 
                               
                             
                             N 
                           
                           - 
                           1 
                         
                         ) 
                       
                       · 
                       z 
                       · 
                       
                         tan 
                         ⁡ 
                         
                           ( 
                           
                             θ 
                             2 
                           
                           ) 
                         
                       
                     
                     + 
                     
                       2 
                       ⁢ 
                       δ 
                     
                   
                 
               
               
                 
                   ( 
                   6 
                   ) 
                 
               
             
           
         
       
         
         
           
             Where: 
             n x3  is the position of a target point T expressed in number of pixels from the origin of the S 13  plane in the x direction; and 
             n y3  is the position of the target point T expressed in number of pixels from the origin of the S 13  plane in the y direction. 
           
         
       
    
     Thus, determination of the physical x coordinate of the selected target T can be determined based upon expressions (3) or (6). A determination of the physical y coordinate of the selected target T can be determined based upon expressions (2) or (5). A determination of the physical z coordinate of the selected target T can be determined based upon equations (1) or (4). 
     More generally, in accordance with module  402 . 6 , a determination is made as to whether the computed 2D coordinate values indicate that the candidate pixels are illuminated by a projection from the same target. It will be appreciated that a more reliable determination of the physical (x, y, z) coordinates of the target T can be obtained through the use of two formulations for each coordinate. For example, the y coordinate for the target T can be determined using both formulations (2) and (5). If the resulting y coordinate values computed using the two formulations differ by more than some acceptable tolerance value, ε y , then a determination can be made that the matching process failed to resolve the offset between projections in the different sensors with sufficient accuracy, and as result that the candidate pixels do not correspond in that they do not receive projections from the same target T. In the event of a failure of the y computations to match, another iteration of the matching process may be performed in an effort to make an improved selection of candidate pixels within the sensors that each corresponds to a selected target T. It will be appreciated that the computed y values are unlikely to be equal since the different perspective projections onto different sensors can differ due to parallax effects, for example. Therefore, an acceptable tolerance value is prescribed according to the intended application. For surgical imaging applications, an ε of 0.1-0.3 mm typically offers an acceptable Q3D accuracy. A person of skill in the art may define different acceptable tolerance levels without departing from the spirit of this invention. 
     Given the assumed sensor symmetry around the x and y axes, persons skilled in the art will appreciate that the same kind of determination can be made for the x coordinates of the target T using formulations similar to those in (2) and (5), but using n xi  instead of n yi . Formulations (3) and (6) cannot be used part of  402 . 5  and  402 . 6  because they require knowledge of the z coordinate. However, the essence of modules  402 . 5  and  402 . 6  is to determine the correct target projections on the planes of sensors S 11 , S 12  and S 13 . For this purpose formulations (2) and (5), adjusted for x and y axes, are sufficient. The complete set of coordinates (x, y, z) is computed part of modules  403  and  404 , as described below. 
     Example of Determining Target&#39;s Physical z Location 
     As illustrated in  FIG. 19 , in accordance with modules  403  and  404 , an initial estimate for the z coordinate, z 0 , is used to initiate the computation process. This initial value is defined automatically, according to the medical application. The medical application defines the intended world view to be visualized. The initial value z 0  starts at the edge of the field of view closest to the endoscope. Referring to  FIG. 8 , for a Q3D application involving surgical endoscopy, z 0  can be 1-5 mm off the distal end  208  of the Q3D endoscope  202 , for example. Such initial estimate generally is sufficient for this application as it is unlikely to have any tissues or surgical instruments reside in such close proximity to the Q3D endoscope. Next, value z 0  is plugged into formulations (3) and (6). Given that the x coordinate of the target is unique, if z 0  were the true and correct z coordinate of the target then formulations (3) and (6) would yield identical values, or approximately equal, within an acceptable level of tolerance, ε x .
 
| x   (3)   −x   (6) |&lt;ε x   (7)
 
     If (3) and (6) are outside an acceptable tolerance ε x , then the iteration continues and a new estimate for z is tried, z 1 . In accordance with some embodiments, the new estimate is defined automatically. For example, z 1 =z 0 +Δ, where Δ is the size of the iteration step. In general, at k th  iteration z k =z k-1 +Δ. The iterative process stops when condition (7) is met. A smaller Δ yields increased accuracy in determining the correct target coordinates, but it would also require more computational time to complete the process, hence an increased latency. An increased latency may result in delays between surgical instrument movement and its visualization by the operating surgeon. In other words, the surgeon may perceive the system as lagging behind commands. For a surgical viewing space of 20-30 cm of depth, a Δ of 0.1-0.3 mm may be sufficient. Of course, a person skilled in the art would know to balance the size of A against the computational required to complete the iterative process. 
     The above explanation has been simplified for presentation reasons and, therefore, it included only three sensors, S 11 , S 12 , and S 13 . In general, more sensors can be used to increase the accuracy of Q3D coordinate computations but also to reduce the overall number of iterations. For example, if more than three sensors are used, preferably an array of 3×3 sensors, then methods such as the steepest gradient may be employed to trend the direction of estimation errors made by modules  402 . 5  and  403 . The iterative step size and direction can then be adjusted to match the progression towards the local extreme of the 3D error gradient surface. 
     Guiding Endoscopic Surgery with Q3D Information 
       FIG. 20  is an illustrative flow diagram representing a first process  2000  to use Q3D information during a surgical procedure in accordance with some embodiments. Computer program code configures the computer  151  to perform the process  2000 . Module  2002  configures the computer to receive user input to select at least two objects within a surgeon&#39;s field of view when looking into the viewer  312 . Module  2004  configures the computer to display a menu on a computer console in response to receipt of a user selection. Decision module  2006  configures the computer to determine whether user input to the menu is received to display a distance. In response to a determination that user input is received to display a distance, module  2008  configures the computer to display a numerical distance within the video image in the surgeon&#39;s field of view. Decision module  2010  configures the computer to wait for a prescribed time interval for receipt of user input to select distance display and to end operation of decision module  2006  in response to no receipt of user input within a “time out” interval. 
     Decision module  2012  configures the computer to determine whether user input to the menu is received to enter a proximity alarm limit. In response to a determination that user input is received to enter a proximity threshold, module  2014  configures the computer to use Q3D information to monitor proximity between two or more objects within the surgeon&#39;s field of view. Decision module  2016  determines whether the proximity threshold has been crossed. In response to a determination that the proximity threshold has been crossed, module  2018  configures the computer to activate an alarm. The alarm may include a sound, a visual queue such as a blinking light, locking of instrument movement to avoid collision, or other haptic feedback. In response to a determination that the proximity threshold has not been crossed, control flows back to monitoring module  2014 . Decision module  2020  configures the computer to wait for the prescribed time interval for receipt of user input to enter the proximity threshold and to end operation of decision module  2012  in response to no receipt of user input within the “time out” interval. 
       FIG. 21  is an illustrative drawing showing menu selections displayed on a display screen  2102  in accordance with the process of  FIG. 20  in accordance with some embodiments. The display screen  2102  includes a viewing monitor associated with the computer  151 . Alternatively, the display screen  2102  may include a region of the viewing elements  401 R,  401 L of the viewer  312 . In response to user input, module  2004  causes the display of a menu  2104  that includes a first menu item “Display Distance”  2106  and a second menu item “Set Proximity Alarm”  2108 . In response to user input to select the “Display Distance” menu item  2106 , module  2008  causes a display of Q3D distance between two or more objects. Referring again to  FIG. 4 , there is shown a display of a Q3D distance “d_Instr_Trgt” between an instrument  400  and target displayed using module  2008 . In response to user input to select the “Set Proximity Alarm” menu item  2108 , an “Enter Distance” UI input  2110  is displayed that includes a field in which a user can enter a proximity distance threshold value, e.g., one cm. In an alternative embodiment (not shown), a default proximity threshold may be set in advance for all instruments, and a user may change the proximity threshold using the menu of  FIG. 21 , for example. In the alternative embodiment, a user can choose to elect the default threshold value rather than enter a threshold value. In some embodiments, a user can select both to display the distance and set a proximity alert. 
       FIGS. 22A-22B  are illustrative drawings representing certain details of receiving user input in accordance with the process of  FIG. 20  in accordance with some embodiments.  FIG. 22A  shows example first highlighting areas  2202 L,  2202 R of a target  410 L,  410 R, such as body tissue, which can be created using video marker tool, such as telestration, or using the surgeon console manipulating control input devices  160  of  FIG. 4 .  FIG. 22B  shows example second highlighting areas  2206 L,  2206 R of an instrument tip  400 L,  400 R, which can be created using the video marker tool. In operation in accordance with some embodiments, a user creates the first highlighting areas  2202 L,  2202 R. Next, the user creates second highlighting areas  2206 L,  2206 R of the instrument tip  400 L,  400 R using video marker tool. It will be understood that the order in which items are highlighted is unimportant. The user then actuates a selector (not shown) (e.g., press the ENTER key) to enter the selection. Module  2002  interprets the received user input as selection of the target image  410 L,  410 R and the instrument image  400 L,  400 R. 
       FIG. 23  is an illustrative flow diagram representing a second process  2300  to use Q3D information during a surgical procedure in accordance with some embodiments. Computer program code configures the computer  151  to perform the process  2300 . Module  2302  configures the computer to receive user input to select an object within a surgeon&#39;s field of view when looking in to the viewer  312 . For example, referring again to  FIG. 22A , user input is shown received to create the second highlighting areas  2206 L,  2206 R of the instrument tip  400 L,  400 R using the video marker tool. User input (not shown) is received to actuate a selector (not shown) (e.g., press the ENTER key) to enter the selection of the image of the instrument tip  400 L,  400 R. 
     Returning once again to  FIG. 23 , in response to receipt of a user selection, module  2304  configures the computer to display a menu on a computer console. Decision module  2306  configures the computer to determine whether user input to the menu is received to rotate an image of a selected object. In response to a determination that user input is received to rotate an image, module  2308  configures the computer to display rotate the image to show a different three-dimensional perspective of the object. Decision module  2310  configures the computer to wait for a prescribed time interval for receipt of user input to rotate an image and to end operation of decision module  2306  in response to no receipt of user input within a “time out” interval. 
       FIG. 24  is an illustrative drawing showing menu selections displayed on a display screen  2402  in accordance with the process of  FIG. 23  in accordance with some embodiments. The display screen  2402  includes a viewing monitor associated with the computer  151 . Alternatively, the display screen  2402  may include a region of the viewing elements  401 R,  401 L of the viewer  312 . In response to received user input, module  2304  causes the display of a menu  2404  that includes a third menu item “Rotate Left”  2406  and a fourth menu item “Rotate Right”  2408 . In response to user input to select one or the other of the third or fourth menu items  2406 ,  2408 , module  2308  uses the causes a rotation of the 3D model created and stored pursuant to module  407  of  FIG. 9 . It will be appreciated that the amount of rotation may be limited to a few degrees, less than 30 degrees for example, since the sensor imager array  210  has a limited overall field of view. 
     Multiport 
       FIGS. 25A-25C  are illustrative drawings showing an endoscope  101 C associated with a Q3D image sensor array disposed at multiple different positions to capture Q3D image information for an anatomical structure  2502  from the different perspectives of the multiple different positions in accordance with some embodiments. Referring again to  FIG. 6 , the patient-side system  152  in accordance with some embodiments includes multiple mechanical surgical arms  158 A- 158 D. As shown in  FIG. 5 , different mechanical arms can be associated with a different instruments  101 A- 101 C, which can be inserted via different ports to the inside of a patient&#39;s body cavity to participate in a minimally invasive surgical procedure. As used herein the term “port” may refer to a natural opening or a surgical incision that provides access to a surgical site within a patient&#39;s body cavity. A cannula (not shown) may be inserted to line the contour of a port to act as a protective barrier between an instrument and the patient&#39;s body tissue at the point of entry to the body cavity and to act as a guide for the surgical instruments inserted into the body cavity. Alternatively, a cannula may not be used. 
     In accordance with an additional aspect of the invention, the endoscopic imager system described with reference to  FIGS. 7A-8 , which is associated with an image sensor array  210 , can be alternately coupled to each of the multiple surgical arms  158 A- 158 D. Each arm can be used to insert the endoscope through a different port. Different ports provide entry to the body cavity at different locations, and the endoscope has different perspective views of a scene within the body cavity when coupled to different ones of the surgical arms. Thus, the endoscope can be used to capture Q3D information from different perspectives by alternately coupling it to different arms and inserting it through different ports into the body cavity. The following description is made in the context of using a single endoscope, and it should be understood that similar optional implementations may be achieved by using two or more endoscopes, each entering via a different port, or each entering via the same port and imaging from different locations in space. 
     Referring to  FIG. 25A , the image capture system  101 C (also referred to as the “endoscope”) is coupled to a first surgical arm  158 A, inserted through a first port  2504  that provides access through an anatomical tissue wall  2510  to a body cavity  2512  containing an anatomical structure  2502  to be imaged. The image capture system  101 C, when coupled to the first surgical arm  158 A, captures first image information  2514  within a first field of view (FOV 1 ) from the first viewing perspective. The first image information  2514  is used to produce a first Q3D model  2515 , which is stored in non-transitory machine readable storage device  2516 . The arrow labeled  2517  indicates in an abbreviated form the process of  FIGS. 9-19 , whereby the first image information  2514  is gathered and used to produce the first Q3D model  2515  in accordance with some embodiments. 
     In  FIG. 25B , the image capture system  101 C is coupled to a second surgical arm  158 B inserted through a second port  2506 . The image capture system  101 C, when coupled to the second surgical arm  158 B, captures second image information  2524  within a second field of view (FOV 2 ) from a second perspective. The second image information  2524  is used to produce a second Q3D model  2525 , which is stored in non-transitory machine-readable storage device  2516 . The arrow labeled  2517  indicates in an abbreviated form the process of  FIGS. 9-19 , whereby the second image information  2524  is gathered and used to produce the second Q3D model  2525  in accordance with some embodiments. 
     In  FIG. 25C , the image capture system  101 C is coupled to a third surgical arm  158 C, inserted through a third port  2508 . The image capture system  101 C, when coupled to the third surgical arm  158 C, captures third image information  2534  within a third field of view (FOV 3 ) from a third perspective. The third image information  2534  is used to produce the third Q3D model  2535 , which is stored in non-transitory machine readable storage device  2516 . The arrow labeled  2517  indicates in an abbreviated form the process of  FIGS. 9-19  whereby the third image information  2534  is gathered and used to produce the third Q3D model  2535  in accordance with some embodiments. 
       FIGS. 26A-26C  are illustrative drawings representing the first, second, and third Q3D models  2515 ,  2525 , and  2535  produced using the first, second, and third image information  2514 ,  2524 , and  2534  of  FIGS. 25A-25C , in accordance with some embodiments. It is noted that in this example, each unique Q3D model includes Q3D structure information that is not present in the other Q3D models, and each Q3D model includes information that overlaps with at least one other Q3D model. For example, as shown the triangle structure  2602  is represented only in the first Q3D model  2515 , the rectangle structure  2604  is represented only in the second Q3D model  2525 , and the circle structure  2606  is represented only in the third Q3D model  2535 . However, the elongated rectangle structure  2608 - 1 ,  2608 - 2 ,  2608 - 3  overlaps the three Q3D models. For increased Q3D accuracy, the multiple Q3D models from  FIGS. 26A-26C  may be acquired using a controlled jittering technique, as described more fully below. Controlled jittering may be produced mechanically, such as by controlling robotic arms  158 A- 158 C, or electronically, such as by controlling the optical properties of array lenses. 
       FIG. 27  is an illustrative drawing showing example alignment of the respective Q3D models of  FIGS. 26A-26C . It is noted that the first, second, and third Q3D models  2515 ,  2525 , and  2535  overlap. For example, it can be seen that a first overlapping portion  2808 - 4  of the elongated rectangle structure  2808  is contained in both the first and second Q3D models  2515 ,  2525 . Also, for example, it can be seen that a second overlapping portion  2808 - 5  of the elongated rectangle structure  2808  is contained in both the second and third Q3D models  2525 ,  2535 . 
     U.S. Pat. No. 6,424,885 B1 (filed Aug. 13, 1999), entitled, “Camera Referenced Control in a Minimally Invasive Surgical Apparatus”, invented by Niemeyer et al. (the “&#39;885 Patent”), which is expressly incorporated herein in its entirety by this reference, discloses a system and method using kinematics, which is used in accordance with some embodiments herein, to map the relative positions of instruments  101 A- 101 C coupled to different surgical arms  158 A- 158 C such that relative positions are known. Referring again to illustrative  FIGS. 25A-25C , for example, the first surgical arm  158 A is associated with a first Cartesian coordinate system (x1, y1, z1). The second surgical arm  158 B is associated with a second Cartesian coordinate system (x2, y2, z2). The third surgical arm  158 C is associated with a third Cartesian coordinate system (x3, y3, z3). As explained in the &#39;885 Patent, the relative positions of instruments coupled to the first, second, and third surgical arms are known based upon known kinematic relationships among the coordinate systems associated with each of the arms. In other words, changes in absolute positions of any of the arms (and instruments coupled to the arms) may change the positions of the arms relative to one another. However, the relative positions of the arms to each other are known due to known kinematic relationships among the arms (and instruments coupled to the arms) as explained in the &#39;885 Patent. 
     In the example alignment for the Q3D models, it is assumed that the perspective and corresponding FOV 2  of the image capture system  101 C, when coupled to the second surgical arm  158 B and positioned to capture the second image information  2524  used to produce the second model Q3D  2525 , is taken as the reference perspective. A determination of the reference perspective includes a determination of an orientation (u x2 , u y2 , u z2 ) of the image capture system  101 C relative to the second Cartesian coordinate system (x2, y2, z2) associated with the second surgical arm  158 B. In other words, in accordance with some embodiments, although a relative position of the second surgical arm  158 B is known based upon kinematics of the overall system, an orientation of the endoscope  101 C relative to the second arm  158 B, when the endoscope  101 C is coupled to the second arm  158 B, contributes to determination of the orientation of the second field of view, FOV 2 . Similarly, a first workspace perspective of the image capture system  101 C, when the image capture system  101 C is secured to the first surgical arm  158 A and positioned to capture the first image information  2514  used to produce the first model Q3D model  2515 , includes an orientation (u x1 , u y1 , u z1 ) of the image capture system  101 C relative to the first Cartesian coordinate system (x1, y1, z1) associated with the first surgical arm  158 A. Likewise, a third workspace perspective of the image capture system  101 C, when the image capture system  101 C is secured to the third surgical arm  158 C and positioned to capture the third image information  2534  used to produce the third Q3D model  2535 , includes an orientation (u x3 , u y3 , u z3 ) of the image capture system  101 C relative to the third Cartesian coordinate system (x3, y3, z3) associated with the third surgical arm  158 C. 
     A known kinematics relationship between the first and second surgical arms  158 A and  158 B is used to determine the relative positions of FOV 1  and FOV 2 , which in turn, determines the relative positions represented by the first and second Q3D models  2515 ,  2525 . Similarly, a known kinematics relationship between the second and third surgical arms  158 B and  158 C is used to determine the relative positions of FOV 2  and FOV 3 , which in turn, determines the relative positions represented by the second and third Q3D models  2525 ,  2535 . In accordance with some embodiments, this relative field of view position information, in turn, can be used to align the first Q3D model produced with first image information  2514  within the first field of view FOV 1  with the second Q3D model produced with second image information  2524  within the second field of view FOV 2  as shown in  FIG. 27 . Similarly, this relative field of view position information can be used to align the third Q3D model produced with third image information  2534  within the third field of view FOV 3  with the second Q3D model produced with second image information  2524  within the second field of view FOV 2  as shown in  FIG. 27 . 
     Continuing with the above example, alternatively, in accordance with some embodiments, common fiducial points found in portions of the first, second, and third Q3D models and geometrical transformations (e.g., rotations and translations) can be used to determine the relative positions of FOV 1 , FOV 2 , and FOV 3 .  FIGS. 28A-28C  are illustrative drawings representing example rotations ( FIG. 28A ), translations ( FIG. 28B ), and resulting alignment ( FIG. 28C ) of the example second and third Q3D models in accordance with some embodiments. Referring to  FIG. 28A , the second and third Q3D models  2525 ,  2535  are shown in slight misalignment. First and second fiducial points  2802 ,  2804  are identified as occurring in both the second and third Q3D models  2525 ,  2535 . Continuing with the example, in accordance with some embodiments, rotational and translational transformations of the orientation of the third Q3D model  2535  are made to align the occurrences of the first and second fiducials  2802 ,  2804  in the third Q3D model  2535  with their occurrence in the second Q3D model  2525 . 
     Still referring to the example in  FIG. 28A , the third Q3D model  2535  is rotated, if needed, around its x-axis by employing a rotation matrix M x  defined by rotation angle α. The third Q3D model  2535  is rotated, if needed, around its y-axis, by employing a rotation matrix M y  defined by rotation angle β. Lastly, if needed, the third Q3D model  2535  is rotated around its z-axis, by employing a rotation matrix M z  defined by rotation angle γ. The use of rotation matrices is well known to persons skilled in the art and need not be described in detail herein. As illustrated in  FIG. 28B , the objective and result of these virtual rotations, in accordance with some embodiments, are such that the transformed third Q3D model  2535  is rotationally aligned with the second Q3D model  2525 . 
     The resulting overall rotation matrix M equals:
 
 M=M   x   ·M   y   ·M   z   (8)
 
     or, in trigonometric format: 
     
       
         
           
             
               
                 
                   M 
                   = 
                   
                       
                     
                       [ 
                       
                         
                           
                             
                               cos 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               α 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               cos 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               β 
                             
                           
                           
                             
                               
                                 cos 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 γ 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 sin 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 α 
                               
                               + 
                               
                                 sin 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 γ 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 sin 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 β 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 cos 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 α 
                               
                             
                           
                           
                             
                               
                                 
                                   
                                     
                                       sin 
                                       ⁢ 
                                       
                                           
                                       
                                       ⁢ 
                                       γ 
                                       ⁢ 
                                       
                                           
                                       
                                       ⁢ 
                                       sin 
                                       ⁢ 
                                       
                                           
                                       
                                       ⁢ 
                                       α 
                                     
                                     - 
                                   
                                 
                               
                               
                                 
                                   
                                     cos 
                                     ⁢ 
                                     
                                         
                                     
                                     ⁢ 
                                     γ 
                                     ⁢ 
                                     
                                         
                                     
                                     ⁢ 
                                     sin 
                                     ⁢ 
                                     
                                         
                                     
                                     ⁢ 
                                     β 
                                     ⁢ 
                                     
                                         
                                     
                                     ⁢ 
                                     cos 
                                     ⁢ 
                                     
                                         
                                     
                                     ⁢ 
                                     α 
                                   
                                 
                               
                             
                           
                         
                         
                           
                             
                               
                                 - 
                                 cos 
                               
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               α 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               cos 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               β 
                             
                           
                           
                             
                               
                                 cos 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 γ 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 cos 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 α 
                               
                               - 
                               
                                 sin 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 γ 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 sin 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 β 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 cos 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 α 
                               
                             
                           
                           
                             
                               
                                 
                                   
                                     
                                       sin 
                                       ⁢ 
                                       
                                           
                                       
                                       ⁢ 
                                       γ 
                                       ⁢ 
                                       
                                           
                                       
                                       ⁢ 
                                       cos 
                                       ⁢ 
                                       
                                           
                                       
                                       ⁢ 
                                       α 
                                     
                                     + 
                                   
                                 
                               
                               
                                 
                                   
                                     cos 
                                     ⁢ 
                                     
                                         
                                     
                                     ⁢ 
                                     γ 
                                     ⁢ 
                                     
                                         
                                     
                                     ⁢ 
                                     sin 
                                     ⁢ 
                                     
                                         
                                     
                                     ⁢ 
                                     β 
                                     ⁢ 
                                     
                                         
                                     
                                     ⁢ 
                                     cos 
                                     ⁢ 
                                     
                                         
                                     
                                     ⁢ 
                                     α 
                                   
                                 
                               
                             
                           
                         
                         
                           
                             
                               sin 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               β 
                             
                           
                           
                             
                               
                                 - 
                                 sin 
                               
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               γ 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               cos 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               β 
                             
                           
                           
                             
                               cos 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               γ 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               cos 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               β 
                             
                           
                         
                       
                       ] 
                     
                   
                 
               
               
                 
                   ( 
                   9 
                   ) 
                 
               
             
           
         
       
     
     Referring now to the example in  FIG. 28B , a virtual position translation is determined to move the third Q3D model  2535  to an overlapping position relationship with the second Q3D model  2525  such that the fiducial points  2802 ,  2804 , which occur separately in each of the second and third models  2525 ,  2535 , now overlap, indicating that the models now are aligned, as shown in  FIG. 28C . The geometric position translation is defined by a vector  v , T v  shown in  FIG. 28B . Geometric translations are well known to persons skilled in the art and need not be described in detail herein. 
     It will be understood from the explanation above that similar rotations and translations can be made to align the first and second Q3D models  2515 ,  2525 . 
       FIG. 29  is an illustrative drawing showing an example composite Q3D model  2902  in which the multiple Q3D models of  FIGS. 25A-25C  are stitched together in accordance with some embodiments. As used herein, “stitching” refers to blending of aligned overlapping Q3D models to produce a seamless or substantially seamless composite Q3D model. Techniques to stitch together overlapping image information are well known. See for example Richard Szeliski, “Image Alignment and Stitching: A Tutorial”, Technical Report, MSR-TR-2004-92, Microsoft Research, Microsoft Corporation, last updated Dec. 10, 2006, which discloses image alignment and image stitching algorithms that can be used to stitch together the multiple Q3D models. 
     During image capture in accordance with some embodiments, a viewing perspective of the endoscope  101 C used to capture an image is “jittered” in controlled fashion. In this context, to “jitter” means to adjust or move between two or more real viewpoints, orientations, or combined viewpoints and orientations. At each such viewpoint or orientation, the Q3D endoscope captures image information from a target region in order to enhance the resolution and accuracy of a Q3D model produced based upon the captured image information.  FIGS. 30A-30C  are illustrative drawings showing an endoscope  101 C in three different example jitter positions to capture three different image viewpoints of the target region in accordance with some embodiments. To achieve the different jitter positions, the endoscope  101 C is moved by known small amounts so that the different jitter positions or orientations are known relative to each other. The different endoscope positions differ by small distances, for example smaller than 5 mm. The small relative distances between successive orientations ensure sufficient redundant information in corresponding 3D images. The redundancy is used to improve the accuracy of the Q3D model. 
     The controlled jittering of the endoscope  101 C increases the accuracy of the computation of coordinates (x, y, z) for the points within the space volume corresponding to a target region of interest. The accuracy is increased because additional data, from additionally and slightly changed viewing angles, are provided to the processes described above with reference to  FIGS. 9-19 . Particularly, the controlled jittering helps process of  FIG. 9  to find more accurately the target at steps  402 - 404 . This is achieved by improving the accuracy of finding the best matches described at steps  402 . 3 - 402 . 5  in  FIG. 10 . The fact that the jittering of the endoscope  101 C is controlled offers views of a target scene of interest from slightly different angles. This is equivalent to a virtual increase of the distance between sensors S ij . The controlled jittering can be achieved by the hardware that controls the robotic arm  158 C that supports the endoscope  101 C, as shown for example in reference to  FIG. 5 . Alternatively, the jittering may be electronically introduced if Q3D endoscope  101 C is built using lenses with electronically adjustable optical properties. For example, according to Hassanfiroozi et al., “Liquid crystal lens array for a 3D endoscope”, Proceedings SPIE 9117, May 9, 2014, a liquid crystal lens array can be employed. Such lenses, if used in conjunction with endoscope  101 C, can have their optical properties adjusted by modulation of control voltage signal. Optical jitter can thus be introduced by modifying the control voltage based on a known pattern. For example, the optical axis of such lenses could be adjusted. This action has the equivalent effect of mechanically jittering the orientation of  101 C. Either mechanical jittering, through of motion of third surgical arm  158 C, or electronic jittering, through modulation of a control voltage of liquid crystal lenses, can have the same result of increased accuracy in computing the coordinates (x, y, z) of target points. 
       FIG. 31  is an illustrative flow diagram of a process  3100  to capture a multi-perspective Q3D model in accordance with some embodiments. It will be appreciated that the system  152  can be configured to implement one or more of the blocks or steps in the process  3100 . In block  3102 , a surgical arm is selected. For convenience and clarity of explanation, the following diagram shall be explained with reference to the above example system and Q3D models, although it will be understood that the principles apply more broadly. In block  3104 , the image capture system  101 C is secured to the selected surgical arm. Assume, for example that the first surgical arm  158 A is first selected to have the image capture system secured to it. In block  3106 , the image capture system secured to the selected surgical arm  158 A is inserted through a first port in the body cavity wall to a position having a perspective from which to use the endoscope to produce the first Q3D model  2515  based upon the first image information  2514  within the first field of view FOV 1 , for example. In accordance with some embodiments, in block  3106 , the orientation (u x1 , u y1 , u z1 ) of the image capture system  101 C is determined relative to the first Cartesian coordinate system (x1, y1, z1) associated with the first surgical arm  158 A, and the orientation is stored in a computer-readable storage device (not shown). In block  3108 , information used to generate the first Q3D model  2515  is from the first field of FOV 1 , as represented in  FIG. 25A , and is stored in a computer-readable storage device (not shown). Decision module  3110  determines that there are additional surgical arms and corresponding ports to use to obtain Q3D models, and therefore control flows again to step  3102  and a next surgical arm is selected. 
     Continuing with the above example, assume that in in a second pass through block  3102  the second surgical arm  158 B is selected. 
     In the second pass through block  3104 , the image capture system secured to the selected second surgical arm  158 B is inserted through a second port in the body cavity wall to a position having a perspective from which to use the endoscope to produce the second Q3D model  2525  based upon the second image information  2524  within the second field of view FOV 2 , for example. 
     In accordance with some embodiments, in the second pass through block  3106 , the orientation (u x2 , u y2 , u z2 ) of the image capture system  101 C is determined relative to the second Cartesian coordinate system associated with the second surgical arm  158 B, and the orientation is stored in the computer readable storage device. 
     In the second pass through block  3108 , information used to produce the second Q3D model  2525  is captured from the second field of view FOV 2 , as represented in  FIG. 25B , and is stored in the computer-readable storage device. 
     Still continuing with the above example, decision step  3110  determines that there is another surgical arm and corresponding port to use to obtain Q3D models, and therefore, control flows again to step  3102  and a next surgical arm is selected. Assume that in a third pass through block  3102 , the third surgical arm  158 C is selected. In the third pass through block  3104 , the image capture system secured to the selected third surgical arm  158 C is inserted through a third port in the body cavity wall to a position having a perspective from which to use the endoscope to produce the third Q3D model  2535  based upon the third image information  2534  within the third field of view FOV 3 , for example. In accordance with some embodiments, in the third pass through block  3106 , the orientation (u x3 , u y3 , u z3 ) of the image capture system  101 C is determined relative to the third Cartesian coordinate system associated with the third surgical arm  158 C, and the orientation is stored in the computer-readable storage device. In block  3108 , information used to generate the third Q3D model  2535  is captured from the third field of view FOV 3 , as represented in  FIG. 27C , and is stored in the computer-readable storage device. 
     Next, in accordance with the above example, decision step  3110  determines that there is no additional surgical arm and corresponding port to use to obtain Q3D models, and therefore, control flows to block  3012 . 
     Alternatively, the Q3D endoscope may be manipulated using the same arm, for example  158 A, but positioned in different ports  2504 ,  2506 , or  2508 , in reference to  FIGS. 25A-C . In accordance with some embodiments, a known kinematics relationships between the orientation (u x1 , u y1 , u z1 ) of the endoscope in the first Cartesian coordinate system and the orientation (u x2 , u y2 , u z2 ) of the endoscope  101  in the second Cartesian coordinate system are used to align the first and second Q3D models, as represented in  FIG. 27 . The known kinematics relationships between the orientation (u x2 , u y2 , u z2 ) of the endoscope  101  in the second Cartesian coordinate system and the orientation (u x3 , u y3 , u z3 ) of the endoscope  101  in the third Cartesian coordinate system are used to align the second and third Q3D models, as represented in  FIG. 27 . 
     Alternatively, in accordance with some embodiments and as explained above with reference to  FIGS. 28A-28C , common fiducial points found in portions of the first, second, and third Q3D models, or in respective pairs of models, can also be used to align the models. As described above, geometrical transformations may be used to achieve a best match aligned of the common fiducial points present in different Q3D models. 
     Once the Q3D models are aligned, block  3114  stitches together the first, second and third Q3D models, as represented in  FIG. 29 . Once created in block  3114 , the combined and expanded Q3D model may be stored by module  407  or displayed in 3D by module  408 , both in reference to  FIG. 9 . The Q3D model can be displayed in 3D based on a process described in  FIG. 32 . A number of 3D video algorithms and hardware implementations can be used for this purpose. It should be understood that display in 3D includes both stereoscopic display that creates the illusion of viewing in three dimensions, and true 3D image display that creates an image that occupies three spatial dimensions. This description is made in terms of a stereoscopic 3D display, and persons of skill in the art will understand how to similarly create a true 3D image, mutatis mutandis. 
       FIG. 32  is an illustrative drawing showing details of a processor that displays a Q3D model in a 3D perspective on a 3D display in accordance with some embodiments. The computer module  3202  configures the system of  FIG. 8 , which comprises the video processor  104 , controller  106 , and display driver  109  to separate the Q3D model into a right-eye view  3204 R and left-eye view  3204 L, given an average human interpupillary distance (IPD) of 62-65 mm. To achieve this action, known details of stereoscopy and human IPD are applied. In accordance with some embodiments, module  3206  configures the 3D video controller  106  of 3D display driver  109  to alternately switch between providing video frames to the left-eye  3208 L and providing frames to the right-eye  3208 R at known frame rates in order to give the viewer the impression of a 3D scene. The approach described in  FIG. 32  employs 3D viewing goggles, such as those present on a teleoperation surgery system such as that described in  FIGS. 4-6 . However, other 3D video display mechanism can be used. For example, U.S. Pat. No. 4,562,463 (filed May 15, 1981) issued to Lipton and U.S. Pat. No. 6,008,839 (filed Nov. 27, 1995) issued to Näglele et al. provide additional implementation details, and these patents are incorporated herein in their entirety by this reference. After being displayed, the user can manipulate the combined Q3D model according to description provided in reference to  FIGS. 23 and 24 . The further manipulation may consist of quantitative measurements of distances between other instruments or between instruments and anatomic structures. 
     The foregoing description and drawings of embodiments in accordance with the present invention are merely illustrative of the principles of the invention. For example, rather than use multiple ports to gain multiple different perspectives, a single port could be used with a single endoscope being dynamically repositioned to capture information from different perspectives. For example, an imaging system mounted on distal end of an endoscope that is mounted on a surgical arm wrist mechanism can be dynamically steered to image from various perspectives. Therefore, it will be understood that various modifications can be made to the embodiments by those skilled in the art without departing from the spirit and scope of the invention, which is defined in the appended claims.