Optical imaging system and methods thereof

An optical imaging system utilizes a three-dimensional (3D) light scanner to capture topography information, color reflectance information, and fluorescence information of a target object being imaged, such as a surgical patient. The system also utilizes the topography information of the target object to perform an image mapping process to project the captured fluorescence or other intraoperative images back onto the target object with enhanced definition or sharpness. Additionally, the system utilizes the topography information of the target object to co-register two or more images, such as a color image of the target object with a fluorescence image for presentation on a display or for projection back onto the target object.

TECHNICAL FIELD

Generally, the present invention relates to systems and methods of optical imaging. In particular, the present invention relates to a system and method for performing 3D scanning, optical imaging, multi-modal imaging, and image registration and display. More particularly, the present invention relates to a system and method for utilizing 3D scanning in combination with biomedical imaging, such as optical imaging and nuclear imaging modalities, to perform diagnostic functions.

BACKGROUND OF THE INVENTION

Current imaging techniques, such as those used in the medical field, still fail to deliver robust integration of multiple images, as desired by healthcare professionals, such as surgeons. For example, surface topography measurements are currently conducted using 3D scanning methods, such as that utilized by a structured light three-dimensional (3D) scanner, which do not offer advanced optical imaging capabilities, such as that provided by fluorescence imaging or polarization imaging. While 3D scanning based imaging methods provide surface information, topography data, 3D point cloud data, and the ability to track objects and their movements, 3D scanning only provides limited biomedical applications and diagnostic information.

Alternatively, biomedical optical imaging methods typically involve the use of planar imaging methods, and do not provide surface topography information. For example, the planar imaging information acquired by such biomedical optical imaging methods does not present morphology/topography information. In addition, such planar imaging information is difficult to fuse or combine with other imaging information, such as computerized tomography (CT) or magnetic resonance imaging (MRI) data. The lack of 3D shape information from current optical imaging systems, such as a fluorescence imaging system, makes the registration between optical imaging and other imaging modalities difficult.

For example, typical planar fluorescence imaging is not able to acquire adequate depth/topography information from a target of interest, such as a surgical site. Fluorescence tomography is very slow and is unable to be used for intraoperative imaging, and to track tissue deformation. Furthermore, it is difficult to co-register fluorescence imaging with preoperative imaging data from a CT, an MRI, or positron emission tomography (PET).

Therefore, there is a need for an imaging system that utilizes 3D scanning with biomedical optical imaging techniques to generate surface topography images that are integrated with information that is obtained from specialized optical imaging methods, such as fluorescence imaging or polarization imaging. In addition, there is a need for an imaging system that utilizes 3D scanning to provide surface topography information, tracking of movements and deformation of the target object, and the generation of 3D models for data fusion/image registration, in combination with an optical imaging method that provides diagnostic information. Additionally, there is a need for an imaging system that utilizes 3D scanning, which is capable of use in various applications, including but not limited to surgery, therapeutic monitoring, radiotherapy monitoring, wound healing, telemedicine, security check, and medical training. Furthermore, there is a need for an imaging system that combines virtual reality features with 3D scanning imaging and biomedical optical imaging techniques.

SUMMARY OF THE INVENTION

In light of the foregoing, it is one aspect of the present invention to provide an optical imaging system to image a target object comprising: a controller; a light emitter module coupled to the controller to illuminate the target object with light; and an image detection module coupled to the controller, the image detection module having a field of view to image the target object; wherein the controller controls the light emitter and the image detection module, such that in one mode the image detection module captures a three-dimensional (3D) topography image of the target object, and in a second mode the image detection module captures a specific image of the target object.

It is another aspect of the present invention to provide an optical imaging system to image a non-flat target object comprising: a controller; a light emitter module coupled to the controller to illuminate the target object with light; an image detection module coupled to the controller, the image detection module having a field of view to image the non-flat target object; wherein the controller controls the light emitter and the image detection module, such that the image detection module captures a three-dimensional (3D) topography image of the target object, and wherein the controller identifies the relative distance between a plurality of portions of the non-flat surface and the light emitter module to generate a corrected projection image, whereupon the corrected projection image is projected by the light emitter module.

It is another aspect of the present invention to provide a method of registering images of a target object comprising the steps of: acquiring a first specific image of the target object from a first image detection module; acquiring a second specific image of the target object from a second image detection module; acquiring a distance of the target object relative to two or more image detection modules to obtain a topography image of the target object; calculating a transformation matrix based on the first specific image, the second specific image, and the distance; and registering the first specific image to the second specific image based on the transformation matrix.

It is another aspect of the present invention to provide a method of optical scanning of a target object comprising the steps of: illuminating the target object by a polarized light; processing the light returned from the target object by an analyzer as analyzed light; and detecting the analyzed light as a topography image of the target object.

It is another aspect of the present invention to provide an optical imaging system, wherein a controller is configured to perform the steps comprising: computing transformation matrices between a pre-operative image space, an intra-operative image space and a peripheral image space; and co-registering the pre-operative image spaces, the intra-operative image space, and the peripheral image space.

It is another aspect of the optical imaging system of the present invention oxygen saturation imaging or vasculature imaging of a target object.

It is another aspect of the optical imaging system of the present invention to provide to capture 3D shape, fluorescence imaging, absorption coefficient and scattering coefficient of a target object

It is another aspect of the optical imaging system of the present invention to provide gesture recognition.

It is another aspect of the optical imaging system of the present invention to provide cardiac gating.

It is another aspect of the optical imaging system of the present invention to be utilized with a contrast agent that is dual-labeled (e.g. a contrast agent that is both fluorescent and also labeled with radioisotope for PET or SPECT imaging).

It is another aspect of the optical imaging system of the present invention to be configured, whereby fluorescence imaging (such as 5-Aminolevulinic acid or Indocyanine green) is used in conjunction with preoperative surgical navigation (e.g. MRI, CT, PET, SPECT . . . ) for guiding interventions or surgeries.

It is another aspect of the optical imaging system of the present invention to provide, where fluorescence imaging (such as 5-Aminolevulinic acid or Indocyanine green) is being used in conjunction with preoperative surgical navigation (eg. MRI, CT, PET, SPECT . . . ) for guiding interventions or surgeries for the brain.

It is another aspect of the optical imaging system of the present invention to provide deformation compensation using biomechanical modeling and FEM modeling.

It is another aspect of the optical imaging system of the present invention to provide vasculature imaging, where vein imaging is preferred with a light source and a detector positioned on 2 different or opposite sides of tissues to be imaged (transmission geometry).

It is another aspect of the optical imaging system of the present invention to provide vasculature imaging, where vein imaging is performed with a light source and a detector on the same side of tissues to be imaged (reflectance geometry).

It is another aspect of the optical imaging system of the present invention to provide vasculature imaging, where the vein imaging is done with multispectral imaging.

It is another aspect of the optical imaging system of the present invention to provide projection quality monitoring, projection optimization and least-squares errors are implemented, and the accuracy of projection is updated accordingly.

It is another aspect of the optical imaging system of the present invention to provide a method of optical scanning of a target object comprising the steps of illuminating the target object by a polarized light; processing the light returned from the target object by an analyzer as analyzed light; and detecting the analyzed light as a topography image of the target object, wherein the polarized light includes a plurality of wavelengths, whereby each wavelength corresponds to a depth within the target object; and processing the polarized light by a spectral filter having a band-pass for each one of the plurality of wavelengths; processing the analyzed light by a spectral filter having a band-pass for each one of the plurality of wavelengths; detecting the analyzed light of each one of the plurality of wavelengths as a topography image for each depth within the target object.

It is another aspect of the optical imaging system of the present invention to generate images by subtraction reflectance image based on one of the plurality of wavelengths with another reflectance image based on another one of the plurality of wavelengths following normalization.

DETAILED DESCRIPTION OF THE INVENTION

An optical imaging system of the present invention is generally referred to by numeral100, as shown inFIG. 1of the drawings. In particular, the imaging system100includes a 3D scanning module110and an optical imaging module120, which are in operative communication with one another via any suitable controller130.

The optical imaging module120includes one or more technologies, including but not limited to fluorescence imaging, reflectance imaging, hyperspectral imaging, IR thermal imaging, Cerenkov imaging, polarization imaging, polarization difference/ratio imaging, spectral polarization difference imaging, multiphoton imaging, second harmonic generation imaging, photoacoustic imaging, and fluorescence lifetime imaging.

As such, the system100is capable of acquiring both surface topography measurements and optical imaging information, such as fluorescence imaging information.

It should also be appreciated that the controller130may include the necessary hardware, software or combination thereof to carryout the functions of the present invention to be discussed. Furthermore, the controller130may comprise any suitable portable or standalone computing device that is capable of controlling the operation of the 3D scanning module110and the optical imaging module120. It should also be appreciated that the controller130may include a data interface132that is capable of outputting data, such as imaging data, to a suitable projector or display coupled to the data interface132. Several embodiments of such a configuration are provided in detail below. It should also be appreciated that the data interface132of the controller130may be configured to communicate through a wired or wireless network to the display or projector, as well as to other remote computing devices for transferring image data output by the data interface132thereto. In addition, the controller130may also be capable of receiving suitable commands through the data interface130from input devices, such as a keyboard, or by a computing device in communication with the data interface130.

In another embodiment of the system100, designated as100A, may also include a display module200in operative communication with the 3D scanning module110, the optical imaging module120, and the controller130previously discussed, as shown inFIG. 2. Specifically, the display module200communicates with the controller130through the data interface132via a wired or wireless connection. The display module200includes any suitable display, such as a liquid crystal display (LCD), or any other display, including but not limited to a two-dimensional (2D) display, a three-dimensional (3D) display, a head-mounted display or a projector.

In another embodiment of the system100, designated as1008, the controller130of the optical imaging system100A may be configured with the necessary hardware, software, or combinations thereof, designated as software module210, to execute a process for registering an optical image and surface topography to other imaging modalities, such as positron emission tomography (PET), single photon emission computed tomography (SPECT), magnetic resonance imaging (MRI), 2D and 3D ultrasound, gamma imaging, optical coherence tomography (OCT), X-ray imaging, and the like. Of course, the software module210may be configured so that it is embedded in the controller130or may be provided by a remote computing or storage device in communication with the interface132.

It should be appreciated that the structural features utilized by the embodiments of the optical imaging system100and100A-B discussed above will be presented in detail below.

In particular, the functions provided by the 3D scanning module110and the imaging module120may be embodied in an imaging configuration300A, as shown inFIG. 4A. In particular, the configuration300includes a light emitter module310and an image detection module320, as shown inFIG. 4A. The light emitter module310includes any suitable light emitting device for emitting one or more light rays322, which serves as an excitation/illumination light of a target object324, and/or for projection of images, such as pattern projection for 3D scanning on a target object324. In one aspect, the light emitter module310may comprise a light projector or laser interference module, which is capable of illuminating the target object324. The image detection module320includes any suitable light detection device configured to detect the light326reflected or emitted by the target object324in response to being illuminated by the emitted light rays322output by the light emitter module310. In one aspect, the image detection module320may comprise a charge coupled imaging device (CCD) or a complementary metal oxide (CMOS) imaging device, however any other suitable imaging device, such as one or more photomultiplier tubes (PMT), one or more avalanche photodiodes (APD), or photodiodes may be used. In addition, an emission filter340, is operatively positioned in front of the input of the image detection module320. The emission filter340may be configured as a bandpass filter, a long pass filter or a tunable filter, which passes only the wavelengths of light associated with the fluorescence that is emitted by the target object324in response to being illuminated by the excitation light322that it has absorbed. Furthermore, it should be appreciated that the emission filter340blocks the wavelengths of light associated with the excitation light322. In particular, the emission filter may comprise a filter wheel or other suitable mechanism, which is configured to be selectively actuated so that in a first mode, the light326passes through the emission filter340before it is captured by the image detection module320, and in a second mode the filter wheel340is rotated or moved so that the light326does not pass through the emission filter340and is directly detected by the image detection module320. In one aspect, the emission filter340may comprise a fluorescence emission filter. As such, configuration300provides that a single imaging device captures topography information of the target object324when the filter340is not used by the image detection module320, and captures fluorescence information of the target object324when the emission filter340is used by the image detection module320. Thus, during operation of the detection configuration300, the image detection module320captures topography information of the target object324when the emission filter340is moved out of the detection path of the image detection module320, and captures fluorescence information of the target object324when the emission filter340is moved into the detection path of the image detection module320. In another aspect, the 3D scanning process may be optionally performed without having the emission filter340moved out of the detection path of the image detection module320. For example, the light emitter module310may be configured to emit light that is able to pass through the emission filter340to enable 3D scanning. One example of this process is that the light emitter module310may emit light at a wavelength of about 830 nm, which is able to pass through the band-pass emission filter340that is centered at 830 nm, so that 3D scanning is enabled.

In another aspect, the system100may capture 3D topography and fluorescence information of the target object324sequentially. To achieve this, the system100performs 3D scanning first, and then fluorescence imaging second.

In another aspect, the system may capture 3D topography and fluorescence information of the target object324concurrently or simultaneously. As such, the image detection module320is able to detect both a 3D scanning signal and fluorescence signals during a similar timeframe. Thus, the image frames captured by the image detection module320may be designated for the purpose of 3D scanning and fluorescence imaging, respectively. For example, if the frame rate of the image detection module320is 30 frames-per-second (FPS), during a one second period 15 frames (e.g. odd frames: 1, 3, 5, 7, 9 . . . ) can be used for capturing 3D topography, while the remaining 15 frames (e.g. 2, 4, 6, 8, 10 . . . ) can be used for fluorescence detection. It should be appreciated that any other combination of image frame designation may be used for concurrent/simultaneous scanning, for example two-thirds of the total image frames may be used for 3D scanning, while one-third of the total image frames are used for fluorescence imaging.

It should also be appreciated that the operation of the light emitter310and the image detection module320are coordinated by the operation of frame synchronization by the controller130. For example, if the image detection module320is operating at 30 frames-per-second (FPS), the light emitter310is able to emit a fringe pattern for capturing 3D topography in 15 frames (e.g. odd frames 1, 3, 5, 7, 9 . . . ) in synchronization with the image detection module320. The light emitter module310is configured to emit a fluorescence excitation light for fluorescence imaging of the remaining 15 frames (e.g. even frames 2, 4, 6, 8, 10 . . . ) in synchronization with the detector320.

It is also contemplated that the filter wheel may be provided as part of the synchronization if desired, so that the emission filter340is moving in and out of the light path, in accordance with the operation mode of the light emitter310and the image detection module320. It should also be appreciated that the synchronization and control of different components of the system100, including the light emitting module310and the image detection module and fluorescence detection module510are controlled by the controller130, where the synchronizing clock signal is generated by the controller130.

In one aspect, the system300A is configured to enable color reflectance imaging in addition to 3D scanning and fluorescence imaging of the target object324. Specifically, if the image detection module320is a color imager (e.g. with filter arrays disposed at the imaging sensor level, such as a Bayer pattern filter array), the image detection module320is able to detect color information by itself.

In another aspect, if the image detection module320may comprise a monochromatic imager, whereby the color image of the target object324may be captured, such that the light emitter module310projects a red, green and blue wavelength sequentially, and the image detection module320captures 3 monochromatic images for red, green and blue reflectance, respectively. In addition, the color image is able to be digitally synthesized. It should also be appreciated that color imaging, fluorescence imaging and 3D scanning can be performed sequentially or concurrently. In the concurrent or simultaneous mode, the image frames are able to be divided for color imaging, fluorescence imaging and 3D scanning, respectively. For example, one-half of the total imaging frames may be used for 3D scanning, one-quarter of the total imaging frames may be used for fluorescence imaging, and one-quarter of the total imaging frames may be used color imaging. It should be appreciated that the light emitter module310, the image detection module320and the filter wheel embodying the emission filter340operates in a synchronized manner with the assistance of the controller130for the concurrent/simultaneous mode of operation.

In yet another embodiment, the emission filter340may comprise a tunable filter. The use of the tunable filter allows hyperspectral imaging to be performed by the configuration300A to capture multiple light wavelengths. In yet another embodiment, the emission filter340may comprise a filter wheel that includes a plurality of narrow-band filters. As such, the configuration300A is able to capture multiple light wavelengths of reflectance images or absorption images.

In yet another embodiment, the filter wheel embodying the emission filter340comprises filters that are suitable for imaging oxygen saturation. For example, images of tissue oxygen saturation (STO2) or venous oxygen saturation (SVO2) may be measured. For example, 660 nm and 950 nm filters may be used to capture the oxygen saturation image. The oxygen saturation can be calculated using the equation: StO2=value of oxygen-saturated hemoglobin/total hemoglobin value (unsaturated+saturated). It should also be appreciated that Cerenkov imaging may also be enabled by using the appropriate filter340.

It should also be appreciated that polarizers can be used instead of spectral filters, and also another polarizer may be placed in from of the light emitter module310to enable polarization imaging, and polarization difference/ratio imaging. It should be appreciated that the different imaging modalities previously discussed may be obtained along with 3D scanning either sequentially or concurrently using the interleaved methods, previously described. It should be appreciated that a plurality of imaging modalities may be enabled by the invention. For example, oxygen saturation imaging, color reflectance imaging, auto-fluorescence imaging and near infrared (NIR) imaging based on extrinsic contrast may be enabled simultaneously at the same time.

In another embodiment, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration300B, as shown inFIG. 4B. Specifically, the configuration300B includes the light emitter module310, the image detection module320, the emission filter340, as previously discussed, with the addition of a fluorescence excitation filter410. In particular, the fluorescence excitation filter410is used in conjunction with the light emitter module310, and is positioned so that the light emitted from the light emitter module310passes through the excitation filter410before illuminating the target object324being imaged. The fluorescence excitation filter410is configured as a bandpass filter, a short pass filter, or a tunable filter that only allows light322having rays that are within a range of excitation wavelengths (i.e. excitation light) that are readily absorbable by the target object324to pass therethrough. In particular, the fluorescence excitation filter410may comprise a filter wheel or other suitable mechanism, which is configured to be selectively actuated so that in a first mode, the light322passes through the excitation filter410, so as to be processed by the emission filter410, before it is incident on the target object324, and in a second mode the filter wheel340is rotated or moved so that the light322does not pass through the excitation filter340and is directly emitted from the light emitter module310to be incident on the target object324. In response to the absorption of the light, the target object324is able to emit light326in the form of fluorescence. As such, in one aspect, during operation of the detection configuration300B, the image detection module320captures topography information of the target object324when the excitation filter410is moved out of the path of the emitted light322and the emission filter340is moved out of the detection path of the image detection module320, and captures fluorescence information of the target object324when the excitation filter410is moved into the path of the emitted light322and the emission filter340is moved into the detection path of the image detection module320. In another aspect, during operation of the detection configuration300B, the image detection module320captures topography information of the target object324when the excitation filter410is moved out of the path of the emitted light322, while the emission filter340remains in the light path. Alternatively, the image detection module320captures topography information of the target object324when the emission filter340is moved out of the detection path of the image detection module320, while the excitation filter410remains in the light path. In addition, the detection configuration300B captures fluorescence information of the target object324when the excitation filter410is moved into the path of the emitted light322and the emission filter340is moved into the detection path of the image detection module320.

In another aspect, the system300B may also capture color images, in addition to 3D topography and fluorescence images.

It should be appreciated that 3D scanning may be performed with either the emission filter340or the excitation filter410out of the light detection or light emission path.

In a further embodiment, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration300C, as shown inFIG. 4C. Specifically, the imaging configuration300C includes the light emitter module310, the image detection module320, the emission filter wheel340, as previously discussed, with the addition of an excitation light source450which operates in conjunction with the fluorescence excitation filter410. The excitation light source450may be configured to generate any suitable light, which then passes through the excitation filter410. In particular, the fluorescence excitation filter410is positioned so that the light322emitted from the excitation light source450passes therethrough before striking the target object324being imaged. In one aspect, during operation of the detection configuration300C, the image detection module320captures topography information of the target object324when the emission filter340is optionally moved out of the light326detection path of the image detection module320and the light emitter module310is used to generate light322to illuminate the target object324. Alternatively, the image detection module320captures fluorescence information from the target object324when the emission filter340is moved into the detection path of the image detection module320and the excitation filter410is moved into the path of the light322emitted from the light source450to illuminate the target object324. In the concurrent mode, the topography scanning frames and fluorescence imaging frames may be interleaved sequentially, with the light emitter module310, excitation light source450and image detection module320being synchronized. In another aspect, the filters340and410may also be synchronized with the light emitter module310, excitation light source450and detection module to enable concurrent scanning.

It should also be appreciated that the excitation light source450may comprise a laser light; a light emitting diode (LED), such as a white LED; an incandescent light; a projector lamp; an arc-lamp, such as xenon, xenon mercury, or metal halide lamp; as well as coherent or in-coherent light sources.

The excitation light source450may also comprise a digital (LED-based) projector, and additionally the light source may project spatial frequencies for patterned illumination. For example, a digital projector in conjunction with spectral filters may be used as the light source. In addition, the excitation light source450may emit a continuous or pulsed output, and may generate light that is within any desired spectral window of electromagnetic waves.

It should also be appreciated that the excitation light source450may also include a light diffuser.

In a further embodiment, the 3D scanning module110and the imaging module120may be embodied in another imaging configuration300D, as shown inFIG. 4D. Specifically, the imaging configuration300D includes the light emitter module310, the image detection module320, the emission filter340, as previously discussed, with the addition of another image detection module320A and another emission filter340A. It should be appreciated that the emission filters340and340A are configured as filter wheels, which may be moved into and out of the optical detection path of the respective imaging detection modules320and320A. As such, during operation of the detection configuration300D, the light emitter module310illuminates the target object324of the target object with light322. In addition, to capture topography information of the target object324by the image detection modules320and320A, the emission filters340and340A are optionally moved out of the light326detection path of the respective image detection modules320and320A. Furthermore, to capture fluorescence information of the target object324by the image detection modules320and320A, the emission filters340and340A are moved into the light326detection path of the respective image detection modules320and320A. As such, because the configuration300D utilizes multiple image detection modules320and320A light path obstructions are avoided.

In a further embodiment, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration300E, as shown inFIG. 4E. Specifically, the imaging configuration300E includes the light emitter module310, the image detection module320, the emission filter340, the image detection module320A and the emission filter340A, as previously discussed, with the addition of the excitation filter410. It should be appreciated that the excitation filter410may be configured as a filter wheel so as to be selectively moved into and out of the output path of the light emitter310. As such, during operation of the detection configuration300E, the image detection modules320and320A capture topography information of the target object324when the excitation filter410is optionally moved so that the light322emitted by the light emitter module310to illuminate the target object324is not processed by the excitation filter410, and the emission filters340and340A are optionally moved out of the detection path of the respective image detection modules320and320A to capture the light326. In addition, the image detection modules320and320A capture fluorescence information of the target object324when the excitation filter410is moved into the detection path so that the light is processed by the excitation filter410, and the emission filters340and340A are moved into the detection path of the respective image detection modules320and320A to capture the light326. As such, because the configuration300E utilizes multiple image detection modules320light path obstructions are avoided. It should be appreciated that in another aspect, the excitation filter410may be embodied as a stationary filter without using a filter wheel, and as such the filter410need not move in and out of the detection pathway, similar to embodiments previously discussed.

In a further embodiment, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration300F, as shown inFIG. 4F. Specifically, the imaging configuration300F includes the light emitter module310, the image detection module320, the emission filter340, the image detection module320A and the emission filter340A, previously discussed, with the addition of the excitation light source450which operates in conjunction with the fluorescence excitation filter410. It should be appreciated that the emission filters340and340A are configured as filter wheels, which may be moved into and out of the optical detection path of the respective imaging detection modules320and320A. As such, during operation of the detection configuration300F, the image detection modules320and320A capture topography information of the target object324when the light emitter310is used to emit light322to illuminate the target object324, and the emission filters340and340A are optionally moved out of the detection path of the respective image detection modules320and320A to detect the reflected light326from the target object324. Alternatively, to capture fluorescence information of the target object324, the light source450is used to emit light322, which is processed by the fluorescence excitation filter410that illuminates the target object324, and the emission filters340and340A are moved into the detection path of the respective image detection modules320and320A to detect the emitted light326from the target object324.

In one aspect, the configuration300F may capture 3D topography and fluorescence information sequentially. As such, during the collection of topography information the excitation light source450and fluorescence excitation filter410are not used, while during collection of fluorescence information the light emitter module310is not used. In another aspect, the configuration300F can capture 3D topography and fluorescence information concurrently or simultaneously. As such, the detection modules320and320A may detect both a 3D scanning signal and a fluorescence signal during a similar timeframe. As such, the frames of detection modules320and320A may be designated for the purposes of 3D scanning and fluorescence imaging, respectively. For example, if the frame rate is 30 frames-per-second (FPS), during a one second period, 15 frames (e.g. odd frames: 1, 3, 5, 7, 9 . . . ) may be used for capturing 3D topography, while the remaining 15 frames (e.g. even frames: 2, 4, 6, 8, 10 . . . ) may be used for fluorescence detection. It should also be appreciated that the any other combination of the frame designation may be used for concurrent scanning (e.g. two-thirds of the total image frames are used for 3D scanning, and one-third of the total image frames are used for fluorescence imaging).

In another aspect, one detection modules320can be designated for fluorescence imaging while the other detection modules320A can be designated for 3D scanning. As such, simultaneous 3D scanning and fluorescence imaging can be enabled.

In a further embodiment, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration500A, as shown inFIG. 5A. In particular, the configuration500A includes the light emitter module310an image detection module320, as previously discussed with regard toFIG. 4A, with the addition of a fluorescence detection module510that is used in conjunction with the fluorescence emission filter340. As such, the fluorescence detection module510and the emission filter340are operatively arranged, such that the light326emitted from the target object324in response to being illuminated by the light322from the light emitter module310passes through the emission filter340before being detected by the fluorescence detection module510. In addition, the fluorescence detection module510may be configured as any suitable imaging device sensitive to fluorescent light wavelengths, and may comprise in some embodiments a charge coupled imaging device (CCD) or a complementary metal oxide (CMOS) imaging device. As such, during operation of the detection configuration500A, the light emitter module310emits light322to illuminate the target object324, whereupon the image detection module320captures topography information, such as depth information, of the target object324from the light322reflected from the target object324. In addition, the fluorescence detection module510captures fluorescence information of the target object324from the light322emitted from the target object324in response to being illuminated by the light322of the light emitter module310. Thus, the configuration500A utilizes 2 cameras/sensors/detectors, the image detection module320and the fluorescence detection module510, whereby the image detection module320captures topography information of the target object324, and the fluorescence detection module510captures fluorescence information of the target object324. It should be appreciated that the light emitter module310may divide the alternating frames for topography and fluorescence imaging, respectively. For example, odd imaging frames can be used for topography scanning and even imaging frames may be used for fluorescence imaging. In another aspect, the system500A can also capture color images using the fluorescence detector module510or image detection module320, in addition to 3d topography and fluorescence images.

In a further embodiment, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration500B, as shown inFIG. 5B. In particular, the configuration500B includes the light emitter module310an image detection module320, the fluorescence detection module510and the fluorescence emission filter340, as previously discussed with regard to FIG.5A, in addition to the fluorescence excitation filter410. Specifically, the fluorescence excitation filter410is configured to be used in conjunction with the light emitter module310, and is operatively arranged therewith. As such, excitation filter410is positioned so that the light emitted from the light emitter module310passes through the excitation filter410before illuminating the target object324being imaged. It should also be appreciated that the excitation filter410comprises a filter wheel, which can be selectively moved into or out of the output path of the light emitter module310. As such, during operation of the detection configuration500B, the image detection module320captures topography information of the target object324when the excitation filter410is optionally moved out of the light output path of the light emitter module310so that the emitted light322is not processed by the excitation filter410before illuminating the target object324, whereupon the image detection module320detects the light326reflected by the target object324. In addition, the fluorescence detection module510captures fluorescence information of the target object324when the excitation filter410is moved into the light output path of the light emitter310so that the emitted light322is processed by the excitation filter410before illuminating the target object324, whereupon the fluorescence detection module510detects the light326emitted from the target object324that has been processed by the emission filter340as fluorescence information of the target object324.

In a further embodiment, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration500C, as shown inFIG. 5C. Specifically, the imaging configuration500C includes the light emitter module310, the image detection module320, the fluorescence detection module510, and the emission filter340, as previously discussed inFIG. 5A, in addition to the excitation light source450and the fluorescence excitation filter410. In particular, the fluorescence excitation filter410operates in conjunction with the light source450, and are operatively positioned so that the light emitted322from the light source450passes through the excitation filter410before illuminating the target object324being imaged. As such, during operation of the detection configuration500C, the image detection module320captures topography information of the target object324when the light emitter module310emits light322to illuminate the target object324, whereupon the image detection module320detects the reflected light326from the target object324. Alternatively, the fluorescence detection module510captures fluorescence information of the target object324when excitation light source450generates light322that has been processed by the excitation filter410to illuminate the target object324, whereupon the light326emitted by the target object324in response to being illuminated is processed by the emission filter340before being detected by the fluorescence detection module510as fluorescence information.

In a further embodiment, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration500D, as shown inFIG. 5D. Specifically, the imaging configuration500D includes the light emitter module310, the image detection module320, the image detection module320A, the fluorescence imaging module510, and the emission filter340, as previously discussed inFIG. 5A, with the addition of the image detection module320A. As such, during operation of the detection configuration500D to detect topography information of the target object324, the light emitter module310emits light322to illuminate the target object324, whereupon the image detection modules320and320A capture topography information, such as depth information, of the target object324from the light322reflected from the target object324. In addition, the fluorescence detection module510captures fluorescence information of the target object324from the light326emitted from the target object324in response to being illuminated by the light322of the light emitter module310. Thus, the configuration500D utilizes multiple cameras/sensors/detectors, including the image detection module320and320A and the fluorescence detection module510, whereby the image detection modules320and320A capture topography information of the target object324, and the fluorescence detection module510captures fluorescence information of the target object324. Thus, because the configuration500D utilizes multiple image detection modules320light path obstructions are avoided. It should be appreciated that one of the detection modules320,320A may be optionally designated to perform other specific imaging, such as color reflectance imaging.

In a further embodiment, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration500E, as shown inFIG. 5E. Specifically, the imaging configuration500E includes the light emitter module310, the image detection module320, the fluorescence imaging module510, the image detection module320A, as previous discussed with regard toFIG. 5D, in addition to the excitation filter410. In particular, the excitation filter410is configured to operate in conjunction with the light emitter module310, and is operatively positioned so that the light emitted from the light emitter module310passes through the excitation filter410before illuminating the target object324being imaged. In particular, the excitation filter410may comprise a filter wheel or other suitable mechanism, which is configured to be selectively actuated so that in a first mode, the light322passes through the excitation filter410, so as to be processed by the emission filter410, before it illuminates the target object324, and in a second mode the filter wheel340is rotated or moved so that the light322does not pass through the excitation filter340and is directly emitted from the light emitter module310to illuminate the target object324. As such, during operation of the detection configuration300E, the image detection modules320and320A capture the reflected light326as topography information of the target object324when the excitation filter410is optionally moved so that the light322emitted by the light emitter module310to illuminate the target object324is not processed by the excitation filter410. In addition, the fluorescence detection module510captures fluorescence information of the target object324from the light326emitted from the target object324in response to being illuminated by the light322of the light emitter module310that has been processed by the excitation filter410. As such, because the configuration300E utilizes multiple image detection modules320light path obstructions are avoided. It should be appreciated that the controller130synchronizes and actuates the light emitter module310, the image detection module320, the fluorescence imaging module510, the image detection module320A, and the excitation filter wheel410. It should be also appreciated that the system500E may use the interleaved method previously described to split the projection and image frames, with the aid of the controller130.

In a further embodiment, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration500F, as shown inFIG. 5F. Specifically, the imaging configuration500F includes the light emitter module310, the image detection module320, the image detection module320A, the fluorescence imaging module510, the emission filter340, the light source450and the excitation filter410, as previously discussed with regard toFIG. 5C, with the addition of the image detection module320A. As such, during operation of the detection configuration500F, the image detection modules320and320A capture topography information of the target object324when the light emitter module310emits light322to illuminate the target object324, whereupon the image detection modules320and320A detect the reflected light326from the target object324. It should be appreciated during the topography information collection process, the excitation light source450is not used, while the excitation filter410is not used. Alternatively, the fluorescence detection module510captures fluorescence information of the target object324when excitation light source450generates light322that has been processed by the excitation filter410to illuminate the target object324, whereupon the light326emitted by the target object324in response to being illuminated is processed by the emission filter340before being detected by the fluorescence detection module510as fluorescence information. It should be appreciated that the light emitter module310is not used during the fluorescence information collection process.

In further embodiments, the 3D scanning module110and the imaging module120may be embodied in another imaging configuration600A, as shown inFIG. 6A. Specifically, the imaging configuration600A includes the light emitter module310, the image detection module320, the fluorescence imaging module510, and the emission filter340, as previously discussed with regard toFIG. 5A, in addition to a beam splitter610. In one aspect, the beam splitter610may comprise a dichroic filter or any suitable device capable of reflecting light of one range of wavelengths, and passing light of another range of wavelengths. The image detection module320and the fluorescence imaging module510are positioned at an angle to each other, such as a substantially right angle, while the beam splitter610is positioned at an oblique angle, such as about a 45-degree angle, relative to the image detection module320and the fluorescence imaging module510. As such, during operation of the detection configuration600A, the light emitter module310emits light322to illuminate the target object324, whereupon the light326reflected and emitted by the target object324in response to being illuminated is received by the beam splitter610, where a portion620of the light326is permitted to pass through the beam splitter610for receipt by the image detection module320to capture topography data of the target object324. In addition, another portion622of the light326is reflected by the beam splitter610, whereupon it is directed to pass through the emission filter340for receipt by the fluorescence imaging module510to capture fluorescence information of the target object324. In another aspect, the beam splitter610may be splitting light based on polarization. It should be appreciated that the beam splitter610may include, but is not limited to a plate beam splitter, a cube beam splitter, a non-polarizing beam splitter, a lateral displacement beam splitter, a pent-prism beam splitter, a wire grid polarizing beam splitter, or a polarizing beam splitter. It should also be appreciated that color image can also be captured by either detector module510or the image detection module320.

In further embodiments, the 3D scanning module110and the imaging module120may be embodied in another imaging configuration600B, as shown inFIG. 6B. Specifically, the imaging configuration600B includes the light emitter module310, the image detection module320, the fluorescence imaging module510, the emission filter340, and the beam splitter610, as previously discussed with regard toFIG. 6A, in addition to the excitation filter410. In particular, the excitation filter410is operatively arranged relative to the light emitter310, so that light622output thereby is processed by the excitation filter410. It should be appreciated that the excitation filter410is configured as a filter wheel or other movable device, which can be moved so that the light output by the light emitter310is selectively processed or not processed by the excitation filter410. As such, during operation of the configuration600B, the excitation filter410is moved so that the light322emitted by the light emitter310is not processed by the excitation filter410to illuminate the target object324, whereupon the light326reflected by the target object324in response to being illuminated is received by the beam splitter610. Next, the reflected and emitted light326from the target object324is received by the beam splitter610, where the portion of the light620is permitted to pass through the beam splitter610for receipt by the image detection module320to capture topography data of the target object324. In order to detect fluorescence information, the excitation filter410is moved so that the light emitted by the light emitter310is processed by the excitation filter410to illuminate the target object324, whereupon the light326emitted from the target object324in response to being illuminated is received by the beam splitter610. Next, the portion622of the light emitted light326is reflected by the beam splitter610, whereupon it is directed to pass through the emission filter340for receipt by the fluorescence imaging module510to capture fluorescence information of the target object324.

In further embodiments, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration600C, as shown inFIG. 6C. Specifically, the imaging configuration600C includes the light emitter module310, the image detection module320, the fluorescence imaging module510, the emission filter340, and the beam splitter610, as previously discussed inFIG. 6A, in addition to the light source450and excitation filter410. In particular, the fluorescence excitation filter410is configured for use in conjunction with the excitation filter410, and is operatively positioned so that the light322emitted from the light emitter module310passes therethrough before illuminating the target object324being imaged. As such, during operation of the detection configuration600C, the light emitter module310emits light322to illuminate the target object324, whereupon the light326reflected by the target object324in response to being illuminated is received by the beam splitter610, where a portion620of the light326is permitted to pass through the beam splitter610for receipt by the image detection module320to capture topography data of the target object324. Alternatively, to detect fluorescence information from the target object324, the excitation light source450is operated to generate light322that is processed by the excitation filter410to illuminate the target object324. As such, the light326emitted from the target object324in response to being illuminated is received by the beam splitter610. Next, the portion622of the light emitted light326is reflected by the beam splitter610, whereupon it is directed to pass through the emission filter340for receipt by the fluorescence imaging module510to capture fluorescence information of the target object324.

In further embodiments, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration700A, as shown inFIG. 7A. Specifically, the imaging configuration700A includes the light emitter module310, the image detection module320, the fluorescence imaging module510, the emission filter340, as previously discussed with regard toFIG. 6A, with the addition of the beam splitter610A, the image detection module320A, the fluorescence imaging module510A and the emission filter340A. As such, the image detection module320and the fluorescence imaging module510are positioned at a substantially right angle to each other, while the image detection module320A and the fluorescence imaging module510A are also positioned at a substantially right angle to each other. In addition, the beam splitter620is positioned at an oblique angle, such as about 45 degrees, relative to the image detection module320and the fluorescence imaging module510. Similarly, the beam splitter620A is positioned at an oblique angle, such as about 45 degrees, relative to the image detection module320A and the fluorescence imaging module510A. As such, during operation of the detection configuration700A, the light emitter module310emits light322to illuminate the target object324, whereupon the light326reflected and emitted by the target object324in response to being illuminated is received by the beam splitters610and610A, where a portion620of the light326is permitted to pass through the beam splitters610and610A for receipt by the image detection modules320and320A to capture topography data of the target object324. In addition, another portion622of the light326received by the beam splitters610and610A is reflected by the beam splitters610and610A, whereupon it is directed to pass through the emission filters340and340A for receipt by the fluorescence imaging modules510and510A to capture fluorescence information of the target object324.

In further embodiments, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration700B, as shown inFIG. 7B. Specifically, the imaging configuration700B includes the light emitter module310, the image detection module320, the fluorescence imaging module510, the emission filter340, the beam splitter610A, the image detection module320A, the fluorescence imaging module510A and the emission filter340A, as previously discussed inFIG. 7A, in addition to the excitation filter410. In particular, the excitation filter410is operatively arranged relative to the light emitter310, so that light output thereby is processed by the excitation filter410. It should be appreciated that the excitation filter410is configured as a filter wheel or other movable device, which can be moved so that the light output by the light emitter310is selectively processed or not processed by the excitation filter410. As such, during operation of the configuration700B, the excitation filter410is optionally moved out of the light output path of the light emitter module310so that the light322emitted by the light emitter310is not processed by the excitation filter410, whereupon light326reflected from the target object324in response to being illuminated by the light322emitted by the light emitter310is received by the beam splitters610and610A, where the portion620of the light326is permitted to pass through the beam splitters610and610A for receipt by the respective image detection modules320and320A to capture topography information of the target object324. In addition, to collect fluorescence information from the target object324, the excitation filter410is moved into the light output path of the light emitter so that light emitted thereby is processed by the excitation filter410. As such, the light326emitted from the target object324in response to being illuminated by the processed light322is received by the beam splitters610and610A, whereupon it is directed or reflected to pass through the emission filters340and340A for receipt by the fluorescence imaging modules510and510A to capture fluorescence information of the target object324. It should be appreciated that in another aspect of the present invention, the excitation filter410may be embodied as a stationary filter without using a filter wheel, and as such the filter410need not move in and out of the detection pathway, similar to the embodiments previously discussed.

In further embodiments, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration700C, as shown inFIG. 7C. Specifically, the imaging configuration700C includes the light emitter module310, the image detection module320, the fluorescence imaging module510, the emission filter340, the beam splitter610, the image detection module310A, the fluorescence imaging module510A, the emission filter340A, and the beam splitter610A, as previously discussed with regard toFIG. 7A, with the addition of the excitation light source450and excitation filter410. As such, during operation of the detection configuration700C, the light emitter module310emits light322to illuminate the target object324, whereupon the light326reflected by the target object324in response to being illuminated is received by the beam splitters610and610A, where a portion620of the light326is permitted to pass through the beam splitters610and610A for receipt by the image detection modules320and320A to capture topography data of the target object324. In addition, to collect fluorescence information from the target object324, the excitation light source450is activated so that light322emitted therefrom is processed by the excitation filter410. As such, the light326emitted from the target object324in response to being illuminated by the processed light output through the excitation filter410is received by the beam splitters610and610A. The received light326is then reflected by the beam splitters610and610A, whereupon the portion622of the light326is directed, or otherwise reflected, to pass through the emission filters340and340A for receipt by the fluorescence imaging modules510and510A to capture fluorescence information of the target object324.

In further embodiments, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration800A, as shown inFIG. 8A. In particular, the configuration800A includes the light emitter module310, and the image detection module320, previously discussed, with regard toFIG. 4A, with the addition of a filter wheel810, and a filter wheel820. In particular, the filter wheel810is configured to be used in conjunction with the light emitter module310, and includes the excitation filter410and a polarizer830carried thereon. In addition, the filter wheel820includes both the emission filter340and an analyzer840carried thereon. In particular, the filter wheel810may be selectively moved relative to the light emitter module310, such that in one state, the excitation filter410processes the light322emitted by the light emitter module310that is used to illuminate the target object324, and such that in another state the polarizer830processes the light322emitted from the light emitter module310that is used to illuminate the target object324. Similarly, the filter wheel820may be selectively moved relative to the image detection module320, such that in one state, the emission filter340processes the light326emitted from the target object324in response to being illuminated, and such that in another state the analyzer840processes the light326reflected from the target object324in response to being illuminated. As such, during operation of the detection configuration800A, when topography information of the target object324is desired to be captured, the polarizer portion830of the filter wheel810is moved into a position to process the light322emitted by the light emitter module310, such that the processed light322illuminates that target object324. In addition, the analyzer portion840of the filter wheel820is moved into the detection path of the image detection module320so that light326reflected by the target object324in response to being illuminated is processed by the analyzer840before being detected by the image detection module320. In one aspect, the analyzer840and the polarizer830may be arranged in a co-polarization configuration to detect photons reflected off of the surface of the target object324and to minimize the detection of photons diffused into the target object324. Alternatively, when fluorescence information of a target object324is desired to be captured, the excitation filter410of the filter wheel810is moved to a position to process the light322emitted by the light emitter module310, such that the processed light322illuminates the target object324. In addition, the emission filter340is moved into the detection path of the image detection module320so that light326emitted by the target object324in response to being illuminated by the processed light322light emitter310passes through the emission filter340before being detected by the image detection module320. In one aspect, during fluorescence detection, the analyzer840and the polarizer830may remain in the illumination and detection path, such that the analyzer840and the polarizer830are arranged in a cross-polarization configuration to reduce specular reflection of the target object324.

In further embodiments, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration800B, as shown inFIG. 8B. In particular, the configuration800B includes the light emitter module310, the image detection module320, the filter wheel810that includes both the excitation filter410and the polarizer830carried thereon, as previously discussed with regard toFIG. 8B, with the addition of the analyzer840that is used in conjunction with the image detection module320and the fluorescence detection module510that is used in conjunction with the image detection module320and the emission filter340that is used in conjunction with the fluoresce detection module510. That is, the analyzer840is positioned relative to the detection path of the image detection module320, while the emission filter340is positioned relative to the detection path of the fluorescence imaging module510. As such, during operation of the detection configuration800B, the image detection module320captures topography information of the target object324when the polarizer portion830of the filter wheel810is moved into a position to process the light322emitted by the light emitter module310. In one aspect, the analyzer840and the polarizer830may be arranged in a co-polarization configuration to detect photons reflected off of the surface of the target object324and to minimize the detection of photons diffused into the target object324. Next, the light326reflected from the target object324in response to being illuminated is processed by the analyzer840before being detected by the image detection module320as topography information. In addition, to capture fluorescence information of the target object324, the excitation filter410of the filter wheel810is moved to a position to process the light322emitted by the light emitter310. As such, the light326emitted from the target object322in response to being illuminated by the light322emitted from the light emitter module310is processed by the emission filter340before being detected by the fluorescence imaging module510.

Thus, the use of the polarizer and analyzer pair of the configurations800A and800B, shown inFIGS. 8A-B, allows the process of polarization gating to be performed, so as remove diffused photons that may travel in the target object324as it is illuminated by the light emitter310during the imaging process. This ensures that the surface topography scan is performed accurately. That is, when incoming structured light is linearly polarized, a portion of the photons are reflected at the surface of the target object324with a well preserved polarization state, while the polarization states of the diffused photons will be more randomly distributed. As such, the polarizer/analyzer pair used by the present invention allows polarization gating to be performed, thereby allowing the surface reflected photons to be separated from the diffused photons during the collection to topography information of the target object324.

Measurement Data Representation

It should be appreciated that topography measurements that are obtained by the present invention are represented as a map of depth z, and as a function of spatial position x and y, which may be expressed as fzij={(xi, yi); i=1, 2 . . . L, j=1, 2 . . . M}. In the present invention, 3D (three-dimensional) surface topography information combined with fluorescence imaging information is able to identify a scalar value that is associated with each point on a surface being imaged of the target object324. This association of values is referred to as “point clouds,” which are expressed as Pi=xi, yi, zi, Ii; where (i=1, 2, 3 . . . N), and Iirepresents the fluorescence imaging value at the ith surface point in the data set. Similarly, if both color reflectance information is also captured from the target object324and integrated with fluorescence and topography information also captured from the target object324, the point clouds are expressed as Pi=(xi, yi, zi, Ii, Ri, Gi, Bi), where i=1, 2, 3 . . . N, with vector (Ri, Gi, Bi) representing red, green, and blue color components. If multiple channels of fluorescence information are captured along with color reflectance from the target object324, the point clouds are expressed as Pi=xi, yi, zi, Ii, Ri, Gi, Bi), where i=1, 2, 3 . . . N, with Iirepresenting the fluorescence channel 1, and Firepresenting the fluorescence channel 2, with each channel having a different wavelength.

In another embodiment, the topography data may be represented by polygon meshes in lieu of point clouds, as meshes reduce computational time. The mesh faces may comprise triangles, quadrilaterals, or any other convex polygons. For example, the mesh faces may comprise vertex-vertex meshes, face-vertex meshes, winged-edge meshes and render dynamic meshes.

In other embodiments of the present invention, active illumination of the scene or target object324being imaged may be achieved by configuring the light emitter module310to use a spatially varying light intensity pattern. For example, the light emitter module310may be configured as a digital projector or a light source that is modulated by a spatial light modulator to provide structured illumination. The structured illumination can be either 2D (two-dimensional) or 3D (three-dimensional). The relationship between the image detection module320that captures topography information, the light emitter module310, and the target object324can be expressed by a triangulation equation.

With reference toFIG. 8C, the geometric relationship between the image detection module320, the light emitter module310, and an object surface point on the target object324may be expressed by the triangulation principle using the following equation:

In particular, the light emitter module310may utilize various algorithms or techniques to generate structured light illumination of the target object324. For example, the techniques or methods for projecting structured illumination that may be utilized by the light emitter module310may include, but are not limited to: 1.) sequential projection techniques, such as binary patterns and gray coding, gray-level patterns, phase shift, photometric stereo techniques, and a hybrid method of phase shifting and gray coding; 2.) full-frame spatially varying color patterns, such as that provided by a rainbow 3D camera, continuously varying color coding, stripe indexing (single shot), stripe indexing using colors, stripe indexing using segment patterns, stripe indexing using repeated gray-scale patterns, stripe indexing based on a De Bruijn sequence; and 3.) grid indexing (2D spatial grid patterns), such as a pseudo-random binary array (PRBA), mini-patterns used as code words, color-coded grids, a 2D array of color-coded dots; and 4.) hybrid methods that combine the methods presented above.

The light emitter module310and the image detection module320may be calibrated to establish the relationship between depth and the spatial location of a pixel on the image of the target object324captured by the image detection module320. In one aspect, the light emitter module310may be calibrated for both intensity and geometry, using any suitable target, such as a checkerboard pattern or an array of dots, for example.

Imaging and Co-Registration I

Thus, the embodiments of the present invention are able to intraoperatively capture, and combine through co-registration, surface topography information and fluorescence imaging information of the target object324, as shown inFIG. 9. For example, in some embodiments, the light source310may comprise a NIR (near-infrared) light source to provide fluorescence excitation of the target object324, while a white light source may also be integrated into the embodiments of the present invention to provide surgical illumination. Additionally, the image detection module320may comprise a CCD (charge coupled device) imaging sensor or CMOS (complementary metal oxide) imaging sensor, which is capable of capturing color or grayscale images. Furthermore, in other embodiments, the present invention is capable of co-registration between intraoperative fluorescence imaging information and/or color reflectance imaging information, and any other preoperative imaging data/information, such as positron emission tomography (PET) data, computerized tomography (CT) data, or magnetic resonance imaging (MRI) data of the target object324, as shown inFIG. 10. In addition, such embodiments of the present invention have been utilized to combine, through co-registration, intraoperative fluorescence imaging and/or color reflectance imaging data, with preoperative PET/CT data, along with fluorescence imaging data to facilitate guided surgery and surgical planning. In addition, the image data from each of the imaging modalities may be represented using different colors. For example, CT data may be represented in a blue color, PET data may be represented in a red color, and fluorescence data may be represented in a green color. In one aspect, the 3D model with the multimodal image data may be rotated upon user input to facilitate visualization.

In another embodiment, preoperative MRI data captured by an MRI scanner, and intraoperative fluorescence data, color reflectance data and topography data that is generated by the system100may be registered together in an image to facilitate brain surgery. In one aspect, contrast agents, such as indocyanine green (ICG) or 5-am inolevulinic acid (5-ALA) may be used to guide brain surgery. The preoperative MRI data can enable surgical navigation using tracking technologies, such as optical tracking or electromagnetic tracking, and intraoperative fluorescence imaging, color reflectance and topography data can provide accurate image guidance. The system100is also enabled to integrate surgical navigation and intraoperative imaging with visualization of co-registered data. In one aspect, 5-ALA fluorescence can be used to image the brain tumor in order to correlate with images that are detected by surgical navigation based on preoperative MRI data. In another aspect, ICG fluorescence can image vascularity of tissues to provide useful information in addition to the preoperative MRI. In one aspect, the light source of system100may be operated at a predetermined frequency for a predetermined duration (eg. 10 ms per exposure, 12 exposures per second) to minimize the photo-bleach or photo-toxicity of contrast agents. The registration algorithm comprises surface-based registration, feature-based registration, point-based registration, intensity-based registration, or combinations thereof. The registration can be performed using point cloud representation, or polygon mesh representation, or combinations thereof. In one aspect, the deformation of the brain can be calculated with biomechanical modeling using finite element method (FEM) to facilitate the registration process.

In other embodiments, the visual data may be clearly visualized by making each type of data “transparent” for representation on any suitable display. Furthermore, the transparency level of each type of image data may be tuned and changed for improved visualization.

In yet another embodiment, a contrast agent that is dual-labeled (e.g. a contrast agent that is both fluorescent and also labeled with a radioisotope for PET or SPECT imaging) may be used with the system100. For instance, an antibody can be labeled with both positron-emitting radioisotopes and fluorescent moieties. The dual labeled contrast agent may be injected into a human target324for imaging purposes. As the contrast agent produces both positrons and fluorescence, the location of the target324can be imaged with both PET and fluorescence imaging, with a source of the signal for both imaging modalities coming from the same source. This facilitates correlation between the PET data and the fluorescence data. It should be appreciated the contrast agents may be labeled to be used for more than 2 modalities (e.g. MRI, PET and fluorescence). It should be further appreciated that contrast agents can be dual-labeled for other combinations of imaging modalities (e.g. MRI and fluorescence, SPECT and fluorescence, etc.).

Imaging and Co-Registration II

In other embodiments of the system100, the image detection module320may comprise a high-speed CMOS (complementary metal-oxide sensor) sensor, while an amplitude-splitting technique may be utilized to increase the imaging speed of the embodiments of the present invention. For example, the image detection module320may comprise a high-speed CMOS color sensor, while the fluorescence imaging module510may comprise a NIR (near infrared) sensitive CMOS sensor that is utilized in conjunction with a NIR beam splitter, such as beam splitter610. For topography imaging or scanning, the image detection module320may comprise a fast CMOS sensor (LUPA1300-2, ON Semiconductor, Inc.) that has a frame rate of about 500 frame-per-second (fps). As such, the present invention is able to reduce the amount of time needed to perform topography imaging or scanning over that of past generation imaging systems. For fluorescence imaging, the fluorescence detection module510may comprise a NIR-sensitive CMOS sensor (MT9V032, Aptina). This sensor offers a high quantum efficiency of about 40% at 800 nm, and a high frame rate of about 60 fps. In addition, this sensor uses a NIR-sensitive monochrome sensor, which improves the detection limit to reach a pM level ICG (indocyanine green) concentration applied to the target object324, which is used to facilitate fluorescence imaging. In addition, the beam splitter610used by the embodiments of the present invention may comprise a short-pass dichroic beam splitter (FF756-SDi01-25x36, Semrock) to separate the visible light component reflected from the target object324and the NIR component (fluorescence) emitted from the target object324after being illuminated. In addition, the emission filter340used by the embodiments of the present invention may comprise an NIR (near infrared) emission filter (#84-123, Edumund Optics, 832 nm CWL with 37 nm bandwidth, OD 6) to minimize cross-talk.

The embodiments of the system100of the present invention are advantageous, as they enable concurrent topography imaging/scanning and NIR fluorescence imaging/scanning, while shortening the total overall imaging time. For example, the NIR fluorescence sensor comprising the fluorescence detection module510is capable of capturing fluorescence information, such as NIR fluorescence, from the target object324, so as to provide real-time 2D (two-dimensional) fluorescence guidance during a surgical procedure, while a color imaging sensor, such as that provided by the image detection module310, captures 3D (three-dimensional) topography information and color reflectance information.

In addition, the embodiments of the present invention may use an image detection sensor320having an f-number, such as F/4, to provide a large depth of field and high light collection efficiency. The image detection sensor320may utilize a CMOS light detector. For topography scanning of the target object324, as performed by the system100of the present invention, high-resolution imaging at a fast frame rate is desired to reduce imaging or scanning time of the target object324. With regard to standard CCD imaging sensors, its read-out rate is intrinsically limited by the charge-shifting mechanisms and the pixel numbers; and binning is typically needed to achieve faster frame rates, which substantially reduces the effective resolution. Compared to CCD sensors, the CMOS sensor utilized by the image detection sensor320of the present invention offers high resolution imaging with faster frame rates and direct access to each pixel, without the need for binning. In addition, the low power consumption and low cost of CMOS sensors are also desirable.

Furthermore, some embodiments of the present invention are configured, whereby the excitation light source450includes a filtered LED (light emitting diode) array, while the light emitter module310comprises a compact digital projector. The customized LED array of the excitation light source450is used for illuminating a region of the target object324that is well defined and uniform, such that in some embodiments only a specific surgical field will be illuminated, therefore reducing the background noise. In some embodiments, the excitation light source450of the present invention may use a high power 780 nm LED array. In addition, in some embodiments, the excitation filter410(#84-106, Edmund Optics, 775 nm CWL with 46 nm bandwidth, OD 6) is used to block light emitted by the excitation light source450that is over 800 nm. The compact digital projector (Barco F50), which is used in some embodiments as the light emitting module310provides fast structured illumination at 120 Hz of the target object324for topography scanning. In other embodiments, the emission filter340may comprise a 700 nm low-pass filter (#84-727, Edmund Optics, OD 4), which removes any NIR (near infrared) light component that may interfere with fluorescence emission detection. Alternatively, the emission filter340may comprise a band-pass filter instead of the low-pass filter, so as to create a narrow band structured light illumination by use of the light emitter module310.

The polarizer830and the analyzer840used by the embodiments of the present invention may be used together as a pair to remove photons that are diffused into the biological tissues of the target object324being imaged, leaving only photons reflected off of the surface of the target object324, so as to ensure accurate surface topography acquisition of the target object324. The structured light illumination of the target object324utilized by the present invention is more advantageous than traditional laser point scanning technologies, as the present invention is capable of operating at a much faster speed and does not have any moving parts.

The control module or controller130utilized by the embodiments of the present invention is configured to communicate with the various components of the configurations to enable their synchronized operation. In one aspect, the controller130may comprise a compact digital processing unit that comprises a FPGA (field programmable gate array) with a suitable computing unit, such as a laptop computer. The FPGA may be programmed using any suitable programming language and utilizes the parallel processing capabilities of the FPGA to implement real-time imaging and displaying of the results on a computer monitor, or any other suitable display. The FPGA board receives data from the image detection module310and the fluorescence imaging module510, which is transmitted via any suitable data connection, such as a camera-link data bus or USB for example, to any suitable remote computing device. The laptop computer will receive the image information from the FPGA board for storage, processing and display purposes, and operates to directly control the structured light illumination from the digital projector provided by the light emitter310and excitation light source450. It should be appreciated that other embedded systems, such as an embedded computer using ARM architecture and using a Linux operating system may be used instead of the FPGA as the controller130. It should be further appreciated that the controller130may comprise any suitable computing system, such as a standalone or portable computing system.

Gesture Recognition

Because the system100has the capacity to perform 3D scanning and depth sensing, a depth map of what is being seen through the system100can be generated, whereupon this depth data can be used for detection of postures or gestures such as hand gestures. 3D model-based algorithms, skeletal-based algorithms, or appearance-based models may be used in the gesture recognition algorithm. For example, the gesture recognition may assist the user in controlling the system100.

Cardiac Gating

Cardiac gating may be performed, where the electrocardiogram (ECG) signal may be acquired and used as a reference signal to correlate with the imaging data acquired by the system100. As such, the data may be assigned to the different phases of respiratory cycles, so that the motion artifact can be reduced and the physiology of each cycle can be studied. In one aspect, pulsed oximetry sensor or mechanical force sensors (placed on patient's thoracic cavity) may also be used to generate the gate signal.

Projecting Fluorescence Information onto Tissue Surface of Target Object

Furthermore, the various embodiments of the present invention have overcome several drawbacks of typical planar displays, which have been used in the past to display fluorescence information. For example, such planar displays typically utilized to present fluorescence imaging, may disturb or distract a surgeon or other individual, and in some cases affect his or her coordination, as they must continuously glance between the display and the subject being operated on. Furthermore, typical planar displays, such as a planar projector display, does not resolve depth/topography information, which can lead to erroneous projections of fluorescence information. Additionally, such planar displays make it difficult for the surgeon to track tissue deformation of the target object324, as he or she is operating.

Accordingly, to overcome the drawbacks of current planar displays, which have been used to present fluorescence imaging information, another embodiment of the present invention100, configuration900A, may be utilized, as shown inFIG. 11A. Specifically, the configuration900A includes an image projection system910that is incorporated into the imaging system100, as shown inFIG. 11. In particular, the image detection sensor310, the fluorescence detection sensor510, the projection system910, and the image detection module310are coupled to the controller130. Additionally, the beam splitter610may be used, as shown inFIG. 11A, to allow the fluorescence imaging module510and the image detection module310to detect topography information and fluorescence information in response to being illuminated by the light emitter module310, as previously discussed. As such, during operation of the configuration900A, the light emitter module310generates light322to illuminate the target object324. In addition, the fluorescence detection module510receives the reflected light326in response to being illuminated and processes it as fluorescence information. In addition, the image detection module320receives the emitted light326from the target object324in response to being illuminated and processes it as topography information. Next, after the system processes the detected fluorescence information according to the topography/depth information identified from the target object324, the projection system910projects the fluorescence information back onto the target object324, such as the patient. It should be appreciated that the projection system910and the light emitter module310can either be two separate devices, or may comprise one device performing both roles. In one aspect, one projector can serve as both the projection system910and the light emitter module310, using the interleaving method previously described. As such, 2 channels may be created, with one for topography scanning and the other one for projection of image information back onto the surface of the object324. It should be further appreciated that, using the interleaving method previously described, it is possible for the projector910to also create fluorescence excitation of the target object324. Thus, this feature of the present invention provides the advantageous benefits of simultaneously providing both topography and fluorescence information to the surgeon. In addition, the present invention utilizes the topography information of the patient or target object324to enable the projection of the fluorescence information back onto the patient or target object with reduced latencies in space misalignment. The configuration900A takes the depth profile and 3D shape of the target object324into account, and using projection mapping techniques based on the topography data of the target object324calculates and processes the appropriate projection images for the projector910. Furthermore, the present invention uses a 3D surface profile to track tissue movements of the target object324during an operating procedure to reduce motion artifacts in the projected fluorescence image.

It should also be appreciated that in additional embodiments, the projection system910may be provided in an imaging configuration900B, as a separate component apart from the light emitter module310, as shown inFIG. 11B. As such, the topography information and fluorescence information captured by the image detection module310and the fluorescence detection module510of the configuration900B are transmitted to the projector system900, which projects the fluorescence data back to the target object324to guide surgery.

In particular, the projection system910, as shown inFIGS. 11A-B, may be used with any of the embodiments of the present invention. In particular, the projection system910is capable of receiving image data from multiple data input channels, including the image detection module320, the fluorescence detection module510, and any other image data source or peripheral, such as a data source containing preoperative CT/PET image data, MRI data, SPECT data or ultrasound data. Once the multiple data inputs have been received, the projection system910simultaneously projects the image data from the multiple data input channels back onto the target object324. For example, one data input channel may contain fluorescence information, another data input channel may contain color reflectance data, and another data input channel may contain preoperative CT/PET data. In one embodiment, the projection system910may include a 2-channel projector that is capable of both structured illumination of the target object324and the projection of fluorescence images back onto the target object324, as inFIG. 11A. In one aspect, the image projection and structured illumination processes may be interleaved to use alternating frames (e.g. 1st, 3rd, 5th. . . frames for image projection, and 2nd, 4th, 6th . . . frames for structured illumination).

Thus, topography information is used to calculate the manner in which the fluorescence information or other image data is projected back to the patient or target object324by the projection system910. For example, the projection system910of the present invention can project topography-correlated fluorescence information back to the patient or target object324, or can project topography-corrected fluorescence and PET/CT information back to the patient or target object324to guide surgery. It should also be appreciated that any other type of image data, including but not limited to MRI (magnetic resonance imaging) data, ultrasound image data, gamma imaging data, CT imaging data, and X-ray imaging data, may be delivered to the input channels of the projection system910, for projection onto the patient or target object324based on the topography data captured and projection mapping methods.

In another aspect, the present invention may include a peripheral interface930provided by the controller130, as shown inFIGS. 1-3, which may comprise a wired or wireless interface, which allows the addition of one or more peripherals to be selectively incorporated, attached, or otherwise placed into communication with the imaging and detection system100. For example, the peripherals may include, but are not limited to, one or more sensors and detectors, including a vital sign sensor module, which is configured to monitor various attributes of a patient, including but not limited to temperature, blood pressure, pulse, respiratory rate, ECG, EEG, pulse oximetry, and blood glucose. The peripherals may also include, but are not limited to an ultrasound module, a spectroscopy module (e.g. Raman spectroscopy, absorption spectroscopy, and reflectance spectroscopy), a GPS (global positioning system) module, a microscope module (e.g. a handheld microscope, a fiber-based in-vivo microscope, and a traditional microscope), and a non-microscopic imaging module (hyperspectral imaging, photoacoustic imaging, optical coherence imaging).

In still another aspect, the peripheral may comprise a probe-based instrument, such as a hand-held probe used to acquire or sense any in-vivo target of interest. As such, the hand-held probe may be used for any desired type of microscopy, such as in-vivo microscopy. Furthermore, the peripheral probe may utilize any suitable detection method, including but not limited to color microscopy, reflectance microscopy, fluorescence microscopy, oxygen-saturation microscopy, polarization microscopy, infrared microscopy, interference microscopy phase contrast microscopy, differential interference contrast microscopy, hyperspectral microscopy, total internal reflection fluorescence microscopy, confocal microscopy, non-linear microscopy, 2-photon microscopy, second-harmonic generation microscopy, super-resolution microscopy, photoacoustic microscopy, structured light microscopy, 4Pi microscopy, stimulated emission depletion microscopy, stochastic optical reconstruction microscopy, ultrasound microscopy, and combinations thereof.

In another aspect, the handheld probe comprising the peripheral may comprise an imaging device that does not have microscopic resolution. Thus, in some embodiments, the non-microscopic imaging probe peripheral of the present invention may utilize various imaging techniques, including but not limited to reflectance imaging, fluorescence imaging, Cerenkov imaging, polarization imaging, ultrasound imaging, radiometric imaging, oxygen saturation imaging, optical coherence tomography, infrared imaging, thermal imaging, photoacoustic imaging, spectroscopic imaging, hyper-spectral imaging, fluoroscopy, gamma imaging, and X-ray computed tomography. The physical form of the handheld probe peripheral may comprise an endoscope, a laparoscope, a bronchoscope, and angioscope, and a catheter for angiography.

Algorithms for Co-Registration of Optical Data, and Other Image Data (PET/CT and MRI Data)

A process1000executed by the controller130for co-registration of the intraoperative optical data captured by the imaging system100of the present invention with other image data is shown inFIG. 12. In one embodiment of the present invention, data from a preoperative PET/CT scan may be acquired prior to performing intraoperative imaging using the system100of the present invention. Surface-based registration will be carried out using an iterative closest point algorithm, with the use of k-d dimensional trees to minimize processing time, in accordance with the process discussed in detail below. Initially, at step1010, the controller130acquires preoperative imaging data, such as PET/CT image data. Next, at step1020, the process creates a point cloud or 3D mesh model, which is based on the preoperative imaging data. In addition, intraoperative imaging data is acquired from the optical imaging topography scanning system100of the present invention, such as fluorescence imaging data, in order to generate a point cloud or 3D model, as indicated at step1030. Continuing, at step1040, the process1000calculates a transformation matrix between the intraoperative image space and the preoperative image space. Next, at step1050the intraoperative imaging data, such as fluorescence image data and color reflectance data, is registered to the preoperative image space. Finally, at step1060, the co-registered image data is displayed via the projection system910onto the patient or target object324, as previously discussed, or onto any suitable display, such as an LCD (liquid crystal display).

Thus, the process1000provides that the image registration may be either surface based or point based. In addition, the image registration can be based on point clouds or on the surface models that are rendered from the image data, or combinations thereof. Optionally, the registration process may utilize finite element modeling or biomechanical modeling algorithms to compensate for movement and deformation of the surgical site. It should also be appreciated that the 3D topography data may be represented using point clouds, or polygon meshes. Furthermore, the registration may be performed with point cloud representation, or polygon mesh representation, or combinations thereof.

It should also be appreciated that in addition to the registration techniques discussed above, other registration techniques may be used by the present invention, such as point-based registration, surface-based registration, and combinations thereof. The registration may comprise either intensity-based or feature-based registration. The transformation models used may comprise linear transformation, or non-rigid/elastic transformation. Spatial or frequency domain methods may be used, as well as automatic or interactive methods. In addition, machine learning algorithms may be used to facilitate image registration. For example, the machine learning algorithm may include, but is not limited to decision tree learning, association rule learning, artificial neural networks, inductive logic programming, support vector machines, clustering, Bayesian networks, reinforcement learning, representation learning, similarity and metric learning, sparse dictionary learning, genetic algorithms, deep learning, and combinations thereof.

It should also be appreciated that in addition to the techniques described above, other tracking techniques may be used, such as optical tracking, magnetic tracking, radio frequency tracking, gyroscope tracking, video tracking (pattern recognition), acoustic tracking, mechanical tracking, and combinations thereof. In addition, the tracking methods utilized by the present invention may utilize a rigid body, flexible body or digitizer methods. Furthermore, the tracking technique may be used either independently or in conjunction with registration algorithms, such as surface-based registration. It should also be appreciated that the previously discussed peripheral modules may be tracked using the tracking techniques for registration with preoperative imaging data and intraoperative imaging data. For example, an ultrasound probe may be optically tracked to document its spatial locations.

If the peripheral placed in communication with the peripheral interface930, such as ultrasound or handheld microscope, is used an alternative registration process1100is executed by the controller130, as shown inFIGS. 13A-B. Initially, at step1110of the process1100, preoperative imaging data is acquired by the peripheral interface930of the control system130. Next, a point cloud or 3D model that is based on preoperative imaging data is created, as indicated at step1120. The process1100then acquires, at step1130, intraoperative imaging data from the optical imaging system100, such as fluorescence information, to generate a point cloud or 3D model. At step1140, the process1100calculates a transformation matrix between the intraoperative image space and the preoperative image space. Next, the intraoperative imaging data, such as fluorescence image data, is registered to the preoperative image space, as indicated at step1150. Continuing to step1160, a moveable peripheral device, such as a hand-held imaging device, or sensing data is acquired, such as ultrasound, fiber microscope, and Raman spectroscopy to generate a point cloud, a polygon mesh or 2D/3D model. Next, at step1170, the position of the hand-held imaging device is tracked using tracking technologies such as optical tracking, while a transformation matrix between the hand-held device image space and intraoperative image space is calculated at step1180. Next, the hand-held image space, such as the ultrasound fiber microscope, is registered to the intraoperative imaging space, as indicated at step1190. Finally, at step1200the co-registered data of step1190is displayed by the projection system910onto the patient or target object324, as previously discussed, or may be displayed on any other display unit.

Fluorescence Imaging with Laser Triangulation for Topography Scanning

In another embodiment, the present invention may utilize a triangulation based method to perform topography scanning of the target object324, whereby laser light is used to probe the depth information for topography. As such, a triangulation laser emits a laser spot on the target object324and a camera or imaging sensor is used to search for the location of the laser dot. Depending on the depth in which the laser strikes a given target object324, the laser dot appears at a different place in the depth of the field of view of the camera or imaging sensor. Therefore, the depth information of the target object324can be inferred. Raster scanning the laser dot will generate a point cloud with depth information on the target object324. Optionally, a laser stripe may be used instead of a single laser dot, whereupon the laser stripe is swept across the target object324to speed up the topography scanning process.

Thus, to carry out the triangulation topography scanning process, the imaging system100may be embodied as another configuration1300A, as shown inFIG. 14Amay be used. Specifically, the configuration1300A includes the image detection module320, which may comprise a CCD (charge-coupled device) or CMOS (complementary metal oxide) camera or other imaging sensors. In addition, the configuration1300A includes the light emitter module310, which is configured to emit one or more laser beams, which are used to illuminate the target object324. It should be appreciated the light emitter module310may also be configured to emit laser beams of one or more center wavelengths, so that one wavelength is used for 3D scanning and another wavelength is used for fluorescence imaging. In addition, the image detection module320includes the emission filter340, which is configured as a moveable filter wheel, as previously discussed. As such, to capture fluorescence information of the target object324, the target object324is illuminated by the laser beam322emitted by the light emitter module310. In addition, the filter wheel is moved so that the emission filter340processes the light326emitted by the target object324in response to being illuminated by the laser beam322of the light emitter module310, whereupon the processed light detected by the image detection module320comprises fluorescence information of the target object324. In another mode of operation to capture topography information, the light emitter module310emits one or more laser beams322, which are used to illuminate the target object324. In addition, the filter wheel is moved so that the emission filter340does not process the light326reflected by the target object324in response to being illuminated by the laser beam322of the light emitter module310, whereupon the processed light detected by the image detection module320comprises topography information of the target object324.

In a further embodiment, the alternative imaging system100, may be embodied in an imaging configuration1300B, as shown inFIG. 14B. Specifically, the imaging configuration1300B includes the image detection module320, which may comprise a CCD (charge-coupled device) or CMOS (complementary metal oxide) camera or imaging sensor, the light emitter module310, which is configured to emit one or more laser beams used to illuminate the target object324, and the emission filter340, as previously discussed with regard toFIG. 14A, with the addition of the fluorescence imaging module510. The fluorescence imaging module510is configured for use in conjunction with the emission filter340. As such, to capture fluorescence information of the target object324, the target object324is illuminated by the laser beam322emitted by the light emitter module310. In response to being illuminated by the laser beam322of the light emitter module310, the target object324emits light326that is processed by the emission filter340before being detected by the fluorescence imaging module510as fluorescence information of the target object324. In another mode of operation to capture topography information, the light emitter module310emits one or more laser beams322, which are used to illuminate the target object324, whereupon the light326reflected from the target object324is detected by the image detection module320as topography information of the target object324. It should be appreciated the light emitter module310may be configured to emit laser beams of one or more center wavelengths, so that one wavelength is used for 3D scanning and another wavelength is used for fluorescence imaging.

It should also be appreciated that in various embodiment, a fluorescence light source may be used. Furthermore, in various embodiments, laser diodes or a laser can be used for triangulation purposes. In other embodiments, multiple channels of fluorescence can be acquired, as well as color reflectance information can be acquired.

In further embodiments, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration1350A, as shown inFIG. 15A. Specifically, the imaging configuration1350A includes, the image detection module320, the emission filter340, the image detection module320A and the emission filter340A, and the excitation light source450which operates in conjunction with the fluorescence excitation filter410, as previously discussed with regard toFIG. 4F, with the addition of the light emitter module310being configured as a triangulation laser. Specifically, the triangulation laser310is configured to emit a plurality of laser beams322suitable for carrying out a triangulation process of the target object324. It should be appreciated that the emission filters340and340A are configured as filter wheels, which may be moved into and out of the optical detection path of the respective imaging detection modules320and320A. As such, during operation of the detection configuration1350A, the image detection modules320and320A capture topography information of the target object324when the light emitter310is used to emit laser beams322to illuminate the target object324, and the emission filters340and340A are moved out of the detection path of the respective image detection modules320and320A to detect the reflected light326from the target object324. In another aspect, the 3D scanning process may be optionally performed without having the emission filter340and340A moved out of the detection path of the image detection modules320and320A. For example, the light emitter module310may be configured to emit light that is able to pass through the emission filters340and340A to enable 3D scanning. One example of this process is that the light emitter module310may emit light at a wavelength of about 830 nm, which is able to pass through the band-pass emission filters340and340A that are centered at 830 nm, so that 3D scanning is able to be performed. Alternatively, to capture fluorescence information of the target object324, the light source450is used to emit light322, which is processed by the fluorescence excitation filter410that illuminates the target object324, and the emission filters340and340A are moved into the detection path of the respective image detection modules320and320A to detect the emitted light326from the target object324. It should also be appreciated that 3D scanning and fluorescence imaging may also be performed concurrently, or sequentially.

In yet another embodiment, the emission filters340and340A may comprise tunable filters. The use of the tunable filters allows hyperspectral imaging to be performed by the configuration1350A to capture multiple light wavelengths. In yet another embodiment, the emission filters340and340A may comprise a filter wheel that includes a plurality of narrow-band filters. As such, the configuration1350A is able to capture multiple light wavelengths of reflectance images or absorption images.

In yet another embodiment, the filter wheel embodying the emission filters340and340A may include filters that are suitable for imaging oxygen saturation. For example, images of tissue oxygen saturation (STO2) or venous oxygen saturation (SVO2) may be measured. For example, 660 nm and 950 nm filters may be used to capture an oxygen saturation image. The oxygen saturation may be calculated using the equation: StO2=value of oxygen-saturated hemoglobin/total hemoglobin value (unsaturated+saturated). It should also be appreciated that Cerenkov imaging may also be enabled by using the appropriate filters340and340A.

It should be appreciated that the different imaging modalities previously discussed may be obtained along with 3D scanning either sequentially or concurrently using the interleaved methods, previously described. It should be appreciated that a plurality of imaging modalities may be enabled by the present invention. For example, oxygen saturation imaging, color reflectance imaging, auto-fluorescence imaging and near infrared (NIR) imaging based on extrinsic contrast may be performed simultaneously at the same time. It should also be appreciated that filter340A and340may have different configurations to enable different imaging modalities. For example, the filter340may be configured to capture fluorescence image and the filter340A may be configured to capture multispectral reflectance images.

Furthermore, to perform the triangulation process, the laser module310emits a laser spot on the target object324and the detector320is used to search for the location of the laser dot on the target object324. Depending on the depth in which the laser strikes the target object324, the laser dot appears at a different place in the depth of the field of view of the camera or imaging sensor320. Therefore, the depth information of the target object324can be inferred based on the corresponding pixel location on the sensor320. Raster scanning the laser dot will generate a point cloud with depth information on the target object324. Optionally, the light emitter module310can output a laser stripe instead of a single laser dot, whereupon the laser stripe is swept across the target object324to speed up the topography scanning process.

In a further embodiment, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration1350B, as shown inFIG. 15B. Specifically, the imaging configuration1350B includes the image detection module320, the image detection module320A, the fluorescence imaging module510, the emission filter340, the light source450and the excitation filter410, as previously discussed with regard toFIG. 5F, with the addition of the light emitter module310being configured as a triangulation laser. Specifically, the triangulation laser310is configured to emit a plurality of laser beams322suitable for carrying out a triangulation process of the target object324. As such, during operation of the detection configuration1350B, the light emitter module310emits laser beams322to illuminate the target object324, whereupon the image detection modules320and320A detect the reflected laser light326from the target object324as topography information. Alternatively, the fluorescence detection module510captures fluorescence information of the target object324when the excitation light source450that have been processed by the excitation filter410to illuminate the target object324, whereupon the light326emitted by the target object324in response to being illuminated is processed by the emission filter340before being detected by the fluorescence detection module510as fluorescence information. It should be appreciated that the 3D scanning and fluorescence imaging processes may be performed concurrently, or sequentially.

Furthermore, to perform the triangulation process, the laser module310emits a laser spot on the target object324, and the detector320is used to search for the location of the laser dot on the target object324. Depending on the depth in which the laser strikes a given target object324, the laser dot appears at a different place in the depth of the field of view of the camera or imaging sensor320. Therefore, the depth information of the target object324can be inferred based on the corresponding pixel location on the sensor320. Raster scanning the laser dot will generate a point cloud with depth information on the target object324. Optionally, the light emitter module310may output a laser stripe instead of a single laser dot, whereupon the laser stripe is swept across the target object324to speed up the topography scanning process.

In further embodiments, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration1350C, as shown inFIG. 15C. Specifically, the imaging configuration1350C includes the image detection module320, the fluorescence imaging module510, the emission filter340, and the beam splitter610, the excitation light source450and the excitation filter410, as previously discussed with regard toFIG. 6C, with the addition of the light emitter module310being configured as a triangulation laser. Specifically, the triangulation laser310is configured to emit a plurality of laser beams322suitable for carrying out a triangulation process of the target object324. In particular, the fluorescence excitation filter410is configured for use in conjunction with the excitation filter410, and is operatively positioned so that the light322emitted from the light emitter module310passes therethrough before illuminating the target object324being imaged. As such, during operation of the detection configuration1350C, the light emitter module310emits laser beams322to illuminate the target object324, whereupon the light326reflected by the target object324in response to being illuminated is received by the beam splitter610, where a portion620of the reflected light326is permitted to pass through the beam splitter610for receipt by the image detection module320to capture topography data of the target object324. Alternatively, to detect fluorescence information from the target object324, the excitation light source450is operated to generate light322that is processed by the excitation filter410to illuminate the target object324. As such, the light326emitted from the target object324in response to being illuminated is received by the beam splitter610. Next, the portion622of the light emitted light326is reflected by the beam splitter610, whereupon it is directed to pass through the emission filter340for receipt by the fluorescence imaging module510to capture fluorescence information of the target object324.

For triangulation, the laser module310emits a laser spot on the target object324and the detector320is used to search for the location of the laser dot on the target object324. Depending on the depth in which the laser strikes a given target object324, the laser dot appears at a different place in the depth of the field of view of the camera or imaging sensor320. Therefore, the depth information of the target object324may be inferred based on the corresponding pixel location on the sensor. Raster scanning the laser dot will generate a point cloud with depth information on the target object324. Optionally, the light emitter module310may output a laser stripe instead of a single laser dot, whereupon the laser stripe is swept across the target object324to speed up the topography scanning process.

In further embodiments, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration1350D, as shown inFIG. 15D. Specifically, the imaging configuration1350D includes the image detection module320, the fluorescence imaging module510, the emission filter340, the beam splitter610A, the image detection module320A, the fluorescence imaging module510A and the emission filter340A, as previously discussed with regard toFIG. 7A, with the addition of the light emitter module310being configured as a triangulation laser. Specifically, the triangulation laser310is configured to emit a plurality of laser beams322suitable for carrying out a triangulation process of the target object324. As such, the image detection module320and the fluorescence imaging module510are positioned at a substantially right angle to each other, while the image detection module320A and the fluorescence imaging module510A are also positioned at a substantially right angle to each other. In addition, the beam splitter620is positioned at an oblique angle, such as about 45 degrees, relative to the image detection module320and the fluorescence imaging module510. Similarly, the beam splitter620A is positioned at an oblique angle, such as about 45 degrees, relative to the image detection module320A and the fluorescence imaging module510A. As such, during operation of the detection configuration1350D, the light emitter module310emits laser beams322to illuminate the target object324, whereupon the light326reflected and emitted by the target object324in response to being illuminated is received by the beam splitters610and610A, where a portion620of the light326, as light reflected by the target object324, is permitted to pass through the beam splitters610and610A for receipt by the image detection modules320and320A to capture topography data of the target object324. In addition, another portion622of the light326, as light emitted by the target object324, is received by the beam splitters610and610A is reflected by the beam splitters610and610A, whereupon it is directed to pass through the emission filters340and340A for receipt by the fluorescence imaging modules510and510A to capture fluorescence information of the target object324.

Fluorescence Imaging with Time-of-Flight Scanner/Range Finder for Topography Scanning

In other embodiments, a time-of-flight laser range finder, or Lidar, or laser scanner may be integrated or utilized in conjunction with the optical imaging capabilities of the present invention previously discussed. Various optical imaging modalities including but not limited to fluorescence, color reflectance, polarization, multispectral, hyperspectral, absorption and oxygen saturation can also be implemented. For example, in some embodiments, the time-of-flight laser range finder or scanner identifies the distance of the target object324by identifying the round-trip time that it takes for a pulse of light emitted by the range finder to reach the target object324and return back to the ranger finder. That is, a pulsed laser range finder emits a pulse of light and the amount of time before the light reflected by the target object324is captured by the detector is measured as the time-of-flight time.

In a further embodiment, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration1400A, as shown inFIG. 16A. Specifically, the imaging configuration1400A includes the image detection module320, the emission filter wheel340, the excitation light source450and the fluorescence excitation filter410as previously discussed with regard toFIG. 4C, with the addition of the light emitter module310being configured as a ranger finder. Specifically, the light emitter module310may be configured as a time-of-flight laser range finder, or any other suitable range finder, such as a laser range finger. In one embodiment, the laser310is configured to emit a plurality of laser beams322suitable for carrying out a range-finder process to identify the distance to the target object324. Additionally, the excitation light source450may be configured to generate any suitable light, which then passes through the excitation filter410. In particular, the fluorescence excitation filter410is positioned so that the light322emitted from the excitation light source450passes therethrough before striking the target object324being imaged. As such, during operation of the detection configuration1400A, the image detection module320captures topography information of the target object324when the emission filter340is moved out of the light326detection path of the image detection module320and the light emitter module310is used to generate laser beams322to illuminate the target object324. In another aspect, the 3D scanning process may be optionally performed without having the emission filter340moved out of the detection path of the image detection module320. For example, the light emitter module310may be configured to emit light that is able to pass through the emission filter340to enable 3D scanning. One example of this process is that the light emitter module310may emit light at a wavelength of about 830 nm, which is able to pass through the band-pass emission filters340that are centered at 830 nm, so that 3D scanning is enabled. Alternatively, the image detection module320captures fluorescence information from the target object324when the emission filter340is moved into the detection path of the image detection module320, and the excitation filter410is moved to process the light322emitted from the light source450to illuminate the target object324. As such, the light326emitted by the target object324in response to being illumined by the processed light322from the light source450is then processed by the emission filter340before being detected by the image detection module320as fluorescence information of the target object324.

In a further embodiment, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration1400B, as shown inFIG. 16B. Specifically, the imaging configuration1400B includes the image detection module320, the emission filter wheel340, the excitation light source450and the fluorescence excitation filter410as previously discussed with regard toFIG. 4C, with the addition of the light emitter module310being configured as a topography scanning module. Specifically, the topography scanning module forming the light emitter module310may be configured to utilize various topography scanning methods, including but not limited to conoscopic holography, modulated light, stereo camera, Fourier 3D scanning, low coherence interferometry, common-path interference 3D scanning, contact profilometers. The light emitter310may comprise the commercial depth sensing camera, such as the Microsoft Kinect™. In addition, such techniques can be integrated with advanced optical imaging methods, such as fluorescence imaging to produce both topography scanning and optical imaging data. It should also be appreciated that the excitation light source450may be configured to generate any suitable light, which then passes through the excitation filter410. In particular, the fluorescence excitation filter410is positioned so that the light322emitted from the excitation light source450passes therethrough before striking the target object324being imaged. As such, during operation of the detection configuration1400B, the image detection module320captures topography information of the target object324when the emission filter340is moved out of the light326detection path of the image detection module320and the light emitter module310is used to generate light beams322to illuminate the target object324. In another aspect, the 3D scanning process may be optionally performed without having the emission filter340moved out of the detection path of the image detection module320. For example, the light emitter module310may be configured to emit light that is able to pass through the emission filter340to enable 3D scanning. One example of this process is that the light emitter module310may emit light at a wavelength of about 830 nm, which is able to pass through the band-pass emission filters340that are centered at 830 nm, so that 3D scanning is enabled. Alternatively, the image detection module320captures fluorescence information from the target object324when the emission filter340is moved into the detection path of the image detection module320, and the excitation filter410is moved to process the light322emitted from the light source450to illuminate the target object324. As such, the light326emitted by the target object324in response to being illuminated by the processed light322from the light source450is then processed by the emission filter340before being detected by the image detection module320as fluorescence information of the target object324.

In yet another embodiment, the emission filter340may comprise a tunable filter. The use of the tunable filter allows hyperspectral imaging to be performed by the configuration1400B to capture multiple light wavelengths. In yet another embodiment, the emission filter340may comprise a filter wheel that includes a plurality of narrow-band filters. As such, the configuration1400B is able to capture multiple light wavelengths of reflectance images or absorption images.

In yet another embodiment, the filter wheel embodying the emission filter340may comprise filters that are suitable for imaging oxygen saturation. For example, images of tissue oxygen saturation (STO2) or venous oxygen saturation (SVO2) may be measured. For example, 660 nm and 950 nm filters may be used to capture the oxygen saturation image. The oxygen saturation can be calculated using the equation: StO2=value of oxygen-saturated hemoglobin/total hemoglobin value (unsaturated+saturated). It should also be appreciated that Cerenkov imaging may also be enabled by using the appropriate filter340.

It should also be appreciated that polarizers may be used instead of spectral filters, and also another polarizer may be placed in front of the light emitter module310to enable polarization imaging, and polarization difference/ratio imaging. It should be appreciated that the different imaging modalities previously discussed may be obtained, along with 3D scanning, either sequentially or concurrently using the interleaved methods, previously described. It should be appreciated that a plurality of imaging modalities may be enabled by the present invention. For example, oxygen saturation imaging, color reflectance imaging, auto-fluorescence imaging and near infrared (NIR) imaging based on extrinsic contrast may be enabled or performed simultaneously, at the same time, by the present invention.

Color-Encoded Stripe Indexing and Wavelength-Splitting Techniques

In further embodiments, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration1500, as shown inFIG. 17. Specifically, the imaging configuration1500includes the light emitter module310, which is configured as a high-speed color projector. In operative arrangement with the light emitter module310is a near-infrared (NIR) rejection filter1510and the polarizer830. It should be appreciated that in some embodiments the filter1510may comprise a short-pass filter. As such, during operation of the light emitter module310, the light322emitted by the light emitter module310is processed by the NIR rejection filter1510and the polarizer830, whereupon the processed light322illuminates the target object324. The imaging configuration1500also includes the light source450, which comprises a near-infrared LED (light emitting diode) array450. The light source450is in operative arrangement with the excitation filter410, and a light diffuser1520. Furthermore, the image detector module320, the light emitting module310, the light source450, and the fluorescence imaging module510are coupled to the controller130so that the components may communicate with each other using suitable communication techniques. As such, during operation of the light source450, the light322emitted by the light source450is processed by the excitation filter410and the light diffuser1520, whereupon the processed light322illuminates the target object324. In addition, the imaging configuration1500includes the beam splitter610, which may comprise a dichroic filter or any suitable device capable of reflecting light of one range of wavelengths, and passing light of another range of wavelengths. In addition, the imaging configuration1500includes the image detection module320, the fluorescence imaging module510, and the emission filter340, such that the image detection module320is positioned at an angle, such as a substantially right angle, to the fluorescence imaging module510and emission filter340. In addition, the beam splitter610is positioned at an oblique angle, such as about a 45 degree angle, relative to the image detection module320and the fluorescence imaging module510/emission filter340. It should be appreciated that the image detection module320is configured as a color sensor for capturing topography information of the target object324, while the fluorescence imaging module510is configured as a near-infrared (NIR) sensor for capturing fluorescence information of the target object324. In addition, an optical lens1550and the analyzer840are operatively arranged relative to the beam splitter610, such that the optical lens1550is placed proximate to the beam splitter610and the analyzer840is placed distal to the beam splitter610. In addition, the optical lens1550and the analyzer840are positioned parallel to the fluorescence imaging module510and at a substantially right angle to the image detection module320, while the beam splitter610is positioned at an oblique angle, such as about a 45-degree angle, relative to the optical lens1550and the analyzer840. Thus, light326that is reflected or emitted from the target object324in response to being illuminated by the light322emitted from the emitting module310or the light322emitted from the excitation light source450is processed by the analyzer840and the optical lens1550before being received by the beam splitter610, whereupon the processed light326is directed to the image detection module320and the emission filter340and the fluorescence imaging module510in the manner to be discussed.

As such, during operation of the detection configuration1500to detect topography information of the target object324, the light emitter module310emits light322, which is processed by the NIR rejection filter1510and the polarizer830to illuminate the target object324, whereupon the light326reflected by the target object324in response to being illuminated is received by the beam splitter610. Upon receipt of the light326at the beam splitter610, the beam splitter610reflects the light326as light rays622for receipt by the image detection module320to capture color topography data of the target object324. In addition, to detect fluorescence information of the target object324, the excitation light source450emits light322, which is processed by the excitation filter410and the light diffuser1520to illuminate the target object324, whereupon the light326that is emitted by the target object324in response to being illuminated is received by the beam splitter610for receipt by the beam splitter610. Upon receipt of the light326at the beam splitter610, the beam splitter610allows the emitted light326to pass through the emission filter340as light rays620for receipt by the fluorescence imaging module510as fluorescence information of the target object324.

It should be appreciated that the configuration1500of the present invention overcomes the limitations of past generation imaging systems that have slow scanning speeds and reduced near-infrared (NIR) detection sensitivity. In particular, the configuration1500of the present invention, as shown inFIG. 17uses a color-encoded stripe indexing single shot technique and a wavelength-splitting 2-sensor configuration that is based on high-speed CMOS imaging sensors320and510. In particular, the image detector module320, the light emitting module310, the light source450, and the fluorescence imaging module510and the controller130are coupled together so that the modules may communicate with each other using suitable communication techniques.

With regard to the image detection module320it may comprise a high-speed color CMOS sensor (LUPA 1300-2, ON Semiconductor) for real-time topography scanning. In some embodiments, the image detection module320may have a frame rate of about 500 frames per second.

In addition, the fluorescence imaging module510may comprise a NIR-sensitive CMOS sensor (MT9V032, Aptina), which may have a quantum efficiency of about 40% at about 800 nm. In addition, the emission filter340may comprise an 832 nm bandpass emission filter (#84-123, Edmund Optics) for fluorescence imaging with high sensitivity. In addition, the beam splitter610may comprise a short-pass dichroic beam splitter (FF756-SDi01-25x36, SEMROCK), which is used to separate the visible and NIR light components, as shown in theFIG. 17. The wavelength-splitting, 2-sensor configuration1500of the present invention enables the concurrent or simultaneous acquisition of topography and fluorescence information of the target object324. The image detection module320comprising the color CMOS sensor may also be configured to capture reflectance images as needed. A F/4 imaging system may be implemented by the image detection module320and fluorescence imaging module510to achieve a large depth of field and high fluorescence collection efficiency.

The light source450may comprise a filtered NIR LED (light emitting diode) array and the light emitting module310may comprise a filtered high-speed color projector. The color projector (Barco F50) comprising the light emitting module310provides fast, single-shot, color-encoded, structured illumination at 120 Hz for real-time topography scanning of the target object324. The color-encoded stripe indexing projection technique that is utilized by the configuration1500of the present invention is implemented using the techniques previously discussed. In addition, the NIR-rejection filter1510may comprise a 700 nm short-pass filter (#84-727, Edmund Optics) in order to remove any NIR component that hinders fluorescence detection. A polarizer/analyzer pair (#89-602, Edmund Optics)830and840are provided to remove photons diffused into the tissues or other material forming the target object324that are depolarized, and as such, operates to capture photons that are reflected off of the surface to ensure that an accurate topography acquisition of the target object324is performed. In addition, the light source450may comprise a 780 nm LED array (Roithner Lasertechnik LED 780-66-60), which has 60 LEDs and a 4 watt optical output power to provide uniform illumination over an area of about 30×24 cm at a 0.5 m working distance. In addition, in some embodiments only a defined field of interest, such as a surgical field, of the target object324will be illuminated to reduce the background. Additionally, the excitation filter410may comprise a 775 nm bandpass excitation filter (#840106, Edmund Optics) to block the emitted excitation light that is over 800 nm.

The controller130communicates with the image detector module320, the light emitting module310, the light source450, and the fluorescence imaging module510to ensure their synchronized operation. In one aspect, the controller130may comprise a FPGA (field programmable gate array) with any suitable memory, such as a 1 GB external memory that is implemented on the board (Opakelly), together with any suitable computer system, such as a Dell precision Tower 7810 Workstation for control and image processing. The FPGA may be programmed using any suitable programming language, such as Verilog, and in some embodiments may utilize the parallel processing capabilities of the FPGA to implement real-time imaging. The FPGA is configured to receive data from the CMOS image sensors320and510, which are transmitted via Camera-Link data buses. As such, the workstation forming the controller130receives the image from the FPGA board for storage, processing and display, and directly controls the structured light illumination from the digital projector.

Image Co-Registration Process

In addition, the present invention also provides a surface-based image co-registration algorithm for PET/CT (positron emission tomography) and/or CT (x-ray computerized tomography) with fluorescence imaging of the target object324, as shown inFIG. 18. As such, the algorithm allows image data from individual modalities, as well as co-registered data to be displayed upon command based on user preferences. For example, preoperative PET/CT images, as shown inFIGS. 18D,E, and fluorescence imaging, as shown inFIGS. 18B,C of lymph nodes in mice have been captured. As such, the present invention uses the topography information, shown inFIG. 18A, of the target object324to allow accurate co-registration between intraoperative fluorescence imaging and color reflectance imaging, with preoperative PET/CT data, as shown inFIGS. 18F-I. As such, the present invention enables multimodal 3D (three-dimensional) image guidance to be provided that is based on preoperative PET/CT surgical navigation and intraoperative fluorescence imaging, as shown inFIGS. 18F-I. That is image data from individual modalities shown inFIGS. 18A-Eand co-registered images ofFIGS. 18C and 18F-I may be presented based on a user's preferences. The co-registered images ofFIGS. 18G-Imay be rotated to facilitate surgical planning and intraoperative decision making.

Furthermore, the present invention overcomes the limitations of prior art systems that utilize offline manual co-registration by implementing a fast image co-registration algorithm or process1600at the control system130to offer multimodal image guidance intraoperatively. The steps performed by the algorithm or process1600are shown inFIG. 19. Initially, at step1610, prior to surgery, the surgical target object324will undergo a PET/CT scan to acquire preoperative image data. Next, at step1620, a point cloud {Pi=(xi, yi, zi, fi, ri, gi, bi), i=1, 2, . . . , N} is then obtained, where the vector (fi, ri, gi, bi) represents the fluorescence, red, green and blue data associated with individual surface points. At step1630, the imaging configuration1500of the present invention acquires intraoperative imaging data, including topography, fluorescence and color reflectance data, as previously discussed. After re-sampling of the optical and PET/CT data sets at step1640, the process1600continues to step1650where the surface-based registration is carried out using the iterative closest point (ICP) algorithm with k-d dimensional trees for minimizing processing time as previously described. It should be appreciated that in some embodiments, the co-registration algorithm may be implemented using any suitable programming language, such as MATLAB or Python. After co-registration, the preoperative PET/CT volumetric data will be incorporated, and the updated point cloud will become {Pi=(xi, yi, zi, fi, ri, gi, bi, pi, ci,), i=1, 2, . . . , N}, where the vector (fi, ri, gi, bi, pi, ci,) represents the fluorescence, red, green, blue, PET and CT data associated with individual points in the 3D volume. Finally, at step1660the co-registered preoperative image data and intraoperative image data is displayed, and in some embodiments displayed in real-time. During the display step1660, the fluorescence, PET and CT signals will be assigned different pseudocolor schemes in the composite image. It should also be appreciated that the registration process may also be optionally performed based on a polygon mesh representation instead of a point cloud representation. The graphic user interface (GUI) of the present invention is optimized to be suitable for both traditional keyboard/mouse input devices, as well as a touchscreen interface, gesture recognition, voice recognition or any other suitable interface device, thereby providing physicians extended flexibility in how to use the imaging system or configuration1500in the clinical settings.

Imaging Goggles with 3D Scanning and Medical Imaging Capabilities

In particular, the functions provided by the 3D scanning module110and the imaging module120may be embodied in an imaging configuration1700A, as shown inFIG. 20A. In particular, the configuration1700A includes the light emitter module310, the image detection module320, and the emission filter340, previously discussed with regard toFIG. 4A, with the addition of a wearable display1710. In particular, the wearable display1710is coupled to an output of the controller130, which is coupled to the light emitter module310and the image detection module320. As such, the topography information and the fluorescence information captured by the imaging configuration1700A is presented on the wearable display1710, in the manner to be discussed.

Furthermore, in another embodiment, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration1700B, as shown inFIG. 20B. Specifically, the configuration1700B includes the light emitter module310, the image detection module320, the emission filter340, and the fluorescence excitation filter410, as previously discussed with regard toFIG. 4B, in addition to the wearable display1710. In particular, the wearable display1710is coupled to an output of the controller130, which is coupled to the light emitter module310and the image detection module320. As such, the topography information and the fluorescence information captured by the imaging configuration1700A is presented on the wearable display1710, in the manner to be discussed. In one aspect, the emission filter340and the fluorescence excitation filter410are provided as movable filter wheels, as previously discussed.

In a further embodiment, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration1700C, as shown inFIG. 20C. Specifically, the imaging configuration1700C includes the light emitter module310, the image detection module320, the emission filter wheel340, the excitation light source450and the fluorescence excitation filter410, as previously discussed with regard toFIG. 4C, in addition to the wearable display1710. In particular, the wearable display1710is coupled to an output of the controller130, which is coupled to the image detection module320. As such, the topography information and the fluorescence information captured by the imaging configuration1700A is presented on the wearable display1710, in the manner to be discussed.

In a further embodiment, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration1700D, as shown inFIG. 20D. In particular, the configuration1700D includes the light emitter module310the image detection module320, the fluorescence detection module510, and the fluorescence emission filter340, as previously discussed with regard toFIG. 5A, in addition to the wearable display1710. In particular, the wearable display1710is coupled to an output of the controller130, which is coupled to the light emitter module310, the image detection module320and the fluorescence module510. As such, the topography information and the fluorescence information captured by the imaging configuration1700A is presented on the wearable display1710, in the manner to be discussed.

In a further embodiment, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration1700E, as shown inFIG. 20E. In particular, the configuration1700E includes the light emitter module310, an image detection module320, the fluorescence detection module510, the fluorescence emission filter340, and the fluorescence excitation filter410, as previously discussed with regard toFIG. 5B, in addition to the wearable display1710. In particular, the wearable display1710is coupled to an output of the controller130, which is coupled to the light emitter module310, the image detection module320, and the fluorescence imaging module510. As such, the topography information and the fluorescence information captured by the imaging configuration1700A is presented on the wearable display1710, in the manner to be discussed.

In a further embodiment, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration1700E, as shown inFIG. 20F. Specifically, the imaging configuration1700E includes the light emitter module310, the image detection module320, the fluorescence detection module510, the emission filter340, the excitation light source450and the fluorescence excitation filter410, as previously discussed with regard toFIG. 5C, in addition to the wearable display1710. In particular, the wearable display1710is coupled to an output of the controller130, which is coupled to the light emitter module310and the image detection module320. As such, the topography information and the fluorescence information captured by the imaging configuration1700A is presented on the wearable display1710, in the manner to be discussed.

As such, the imaging configurations1700A-F of the present invention include the wearable display1710, which may comprise any suitable display, such as a wearable display that is configured for being attached to and worn by a user. For example, the wearable display1710may be included as part of a goggle-type wearable device, which includes a wearable goggle or eye-piece frame that carries the display1710.

In one aspect, the display1710may comprise a single display element suitable for providing a single, continuous display that provides a single display surface that encompasses the totality of the user's field of view, or portion thereof. Alternatively, the display1710may include multiple separate display elements, such as a dedicated right display and a dedicated left display, such as in the case of a stereoscopic display, which provides independent displays, to provide a field of view for each user's eye.

Furthermore, the display1710may comprise a Liquid crystal on silicon (LCoS) display, an LCD (liquid crystal display) display, an OLED (organic light emitting diode) display, a projection display, a head-mounted display (HMD), a head-mounted projection display (HMPD), an optical-see through display, a switchable optical see-through display, a selective occlusion see-through head-mounted display, and a video see-through display. Furthermore, the display may comprise an augmented reality window, augmented monitors, a projection on the patient/projective head-mounted display, selective occlusion see-through head-mounted display, a retinal scanning display, or any other suitable display. In another aspect, the display1710may be configured to display any static or moving image. The display1710may also comprise a picture-in-picture (PIP) display that can display images from multiple independent image sources simultaneously. In one example, the ultrasound image and intraoperative merged fluorescence & topography image may be displayed in a picture-in-picture fashion. In another example, preoperative tomographic images, and intraoperative color images may be merged with topography and displayed in a picture-in-picture fashion.

In some embodiments, the preoperative tomographic images (e.g. MRI, CT, SPECT, and PET) may be co-registered with intraoperative color image, fluorescence images and topographic data, and shown in the display1710.

In other embodiments, the display1710may comprise a stereoscopic display that is capable of displaying stereoscopic images with depth perception. In further embodiments, the display1710may be capable of displaying 3-dimensional (3D) images with depth perception. In still other embodiments, the display1710may be configured to provide overlaid or superimposed images of various opacity/transparency to allow simultaneous viewing of multiple images on the display1710at one time. In yet another embodiment, the display1710may be at least partially light transparent to allow a user to view the image being displayed by the display1710, while simultaneously allowing the user to directly see with their eyes or natural vision through the display1710to also view the user's surrounding environment.

In other embodiments, the fluorescence imaging module510comprises a camera that is configured to capture red, green, and blue color information, as well as depth information or data and fluorescence imaging information.

In another aspect, the system100may capture 3D topography and fluorescence information of the target object324sequentially. To achieve this, the system100performs 3D scanning first, and then fluorescence imaging second.

In another aspect, the system may capture 3D topography and fluorescence information of the target object324concurrently or simultaneously. As such, the image detection module320is able to detect both a 3D scanning signal and fluorescence signals during a similar timeframe. Thus, the image frames captured by the image detection module320may be designated for the purpose of 3D scanning and fluorescence imaging, respectively. For example, if the frame rate of the image detection module320is 30 frames-per-second (FPS), during a one second period 15 frames (e.g. odd frames: 1, 3, 5, 7, 9 . . . ) can be used for capturing 3D topography, while the remaining 15 frames (e.g. 2, 4, 6, 8, 10 . . . ) can be used for fluorescence detection. It should be appreciated that any other combination of image frame designation may be used for concurrent/simultaneous scanning, for example two-thirds of the total image frames may be used for 3D scanning, while one-third of the total image frames are used for fluorescence imaging.

It should also be appreciated that the operation of the light emitter310and the image detection module320are coordinated by the operation of frame synchronization by the controller130. For example, if the image detection module320is operating at 30 frames-per-second (FPS), the light emitter310is able to emit a fringe pattern for capturing 3D topography for 15 frames (e.g. odd frames 1, 3, 5, 7, 9 . . . ) in synchronization with the image detection module320. The light emitter module310is also configured to emit a fluorescence excitation light for fluorescence imaging the remaining 15 frames (e.g. odd frames 1, 3, 5, 7, 9 . . . ) in synchronization with the detector320.

The imaging configurations1700A-F of the present invention can enable various optical imaging techniques. In yet another embodiment, the emission filter340may comprise a tunable filter. The use of the tunable filter allows hyperspectral imaging to be performed by the configurations1700A-F to capture multiple light wavelengths. In yet another embodiment, the emission filter340may comprise a filter wheel that includes a plurality of narrow-band filters. As such, the configurations1700A-F are able to capture multiple light wavelengths of reflectance images or absorption images. In yet another embodiment, the filter wheel embodying the emission filter340may comprise filters that are suitable for imaging oxygen saturation. For example, images of tissue oxygen saturation (STO2) or venous oxygen saturation (SVO2) may be measured. For example, 660 nm and 950 nm filters may be used to capture the oxygen saturation image. The oxygen saturation can be calculated using the equation: StO2=value of oxygen-saturated hemoglobin/total hemoglobin value (unsaturated+saturated). It should also be appreciated that Cerenkov imaging may also be enabled by using the appropriate filter340. It should also be appreciated that polarizers may be used instead of spectral filters, and also another polarizer may be placed in from of the light emitter module310to enable polarization imaging, and polarization difference/ratio imaging. It should be appreciated that the different imaging modalities previously discussed may be obtained along with 3D scanning, either sequentially or concurrently using the interleaved methods previously described. It should be appreciated that a plurality of imaging modalities may be enabled by the present invention. For example, oxygen saturation imaging, color reflectance imaging, auto-fluorescence imaging and near infrared (NIR) imaging based on extrinsic contrast may be enabled simultaneously, at the same time.

In the system, the connection between the wearable display1710and the controller130that is coupled to the imaging module320, the light emitter module310, the light source, and the fluorescence imaging module510may comprise a wired or wireless connection. In some embodiments, the wearable display1710, imaging module320and the light emitter module310are enclosed in a uni-body design to be worn by the user. In other embodiments, the imaging module320, fluorescence imaging module510, the light emitter310and the light source450may be mounted on a tripod, while the wearable display1710is worn by the user. In some embodiments, the excitation light source450for fluorescence excitation may be worn by the user or mounted on a stationary non-moving support.

It should also be appreciated that the controller130may comprise a built-in computing device or may be connected to an external computer/tablet computer/smartphone for system control and data processing.

In another example, gyroscopic tracking may be performed using a tracking module that is provided by the system100. An inertial measurement unit (IMU) may be used for tracking purposes. It should also be appreciated that in addition to the tracking techniques described above, other tracking techniques may be used, such as radio frequency tracking, optical tracking, electromagnetic tracking, video tracking (pattern recognition), acoustic tracking, mechanical tracking, and/or a combination thereof. In addition, the tracking method employed may utilize rigid body, flexible body or digitizer methods.

It should be appreciated that during the tracking and registration processes the controller130performs computations and executes the necessary steps to enable the accurate tracking and registration. In one aspect, the complete registration process may be performed by the following steps. Initially, the process obtains the position of the patient or target object324, the wearable imaging and display system100, and a handheld probe as a peripheral coupled to the interface. Next, the system acquires pre-operative imaging data. Next, a 3D model is created based on the pre-operative imaging data. In the next step, the position of the patient is tracked intra-operatively using any suitable technique, such as fiducial markers for example. Next, a transformation matrix is calculated between the pre-operative image space and the intra-operative object space (i.e. patient space). Continuing, the pre-operative image data is registered to the intra-operative object space (i.e. patient space). Next, the intra-operative imaging data is acquired from the imaging system100, such as fluorescence or color imaging for example. Continuing, the position of the wearable imaging and display system100is obtained, using any suitable technique, such as optical tracking or magnetic tracking). Next, the transformation matrix between the intra-operative imaging space (i.e. wearable imaging and display system) and the intraoperative object space (patient space) is calculated. The intra-operative imaging space (such as fluorescence image data) is then registered to the intra-operative object space (i.e. patient space). In addition, the process acquires handheld device imaging or sensing data, such as ultra-sound fiber microscope, and Raman spectroscopy for example. In addition, the position of the hand-held probe, such as an ultrasound fiber, a microscope, and Raman spectroscopy probe is tracked. Next, a transformation matrix is calculated between the hand-held imaging/sensing probe image space and the intra-operative object space (i.e. patient space). Continuing, the hand-held device image space (i.e. ultrasound or microscope) is registered to the intra-operative object space (i.e. patient space). Finally, the co-registered image data is presented on the display1710of wearable imaging system100.

In another aspect, the process may be configured, such that the tracking and registration process is performed without the image data acquired from the hand-held probe. As a result, the process only uses the intra-operative image data acquired by the imaging and display system100(i.e. goggle system) and the pre-operative surgical navigation image data.

In yet another aspect, the process may also be configured, such that the tracking and registration process is performed without the pre-operative surgical navigation image data. As a result, the process only uses the intra-operative image data acquired by the imaging and display system100(i.e. goggle system) and the image data acquired by the hand-held probe.

It should also be appreciated that in addition to the registration techniques discussed above, other registration techniques may be used, such as point-based registration, surface-based registration, and/or a combination thereof. The registration may comprise either intensity-based, landmark-based registration or feature-based registration. The transformation models used may comprise linear transformation, or non-rigid/elastic transformation. Spatial or frequency domain methods may be used, as well as automatic or interactive methods. The registration process can be performed based on the point cloud representation, the mesh representation, or a combination thereof.

Polarization-Based and Wavelength-Based Optical Scanning at Different Depths

Light of different wavelengths penetrate into the target object324, such as biological tissues, with different wavelengths. For example, near infrared (NIR) light at 800 nm can penetrate into biological tissues deeper than blue light at 450 nm. Also, when linearly polarized light is used, the outer surface of tissues or target object324will reflect the light with polarization state being preserved. In contrast, light that penetrates deeper into tissues of the target object will lose its linear polarization due to the scattering events as photons travel into tissues. These properties can be used to configure the system100as a 3D scanning system that can generate topography at different depths (e.g. most outer surface, 0.5 mm under surface, 1 mm under surface).

Thus, in some embodiments the 3D scanning module110and the imaging module120may be embodied in an imaging configuration1800A, as shown inFIG. 21A. Specifically, the imaging configuration1800A includes the light emitter module310and the image detection module320, as previously discussed with regard toFIG. 4A, with the addition of the polarizer830and the analyzer840. In particular, the polarizer830is operatively arranged for use in conjunction with the light emitter module310, while the analyzer840is operatively arranged for use with the image detection module320. Accordingly, during operation of the imaging configuration1800A, the light emitter module310generates light322that is processed by the linear polarizer830to illuminate the target object324. In addition, the light reflected by the target object324in response to being illuminated is then processed by the analyzer840before being detected as topography information of the target object324. As such, the use of the polarizer830and the analyzer840by the imaging configuration1800A allows depth-resolved topography of the target object324to be captured. Thus, when the polarization state of the linear polarizer830and the analyzer840are aligned (i.e. co-polarization) with each other, the image detection module320is able to detect the photons that reflected off the most outer surface of the target object324to obtain an outer surface topography. Additionally, when the polarization state of the linear polarizer830and the analyzer840are orthogonal (i.e. cross-polarization) to each other, the image detection module320is able to detect the photons that travel into the inner portion of the target object324in order to obtain surface topography at a slightly deeper depth within the target object324.

Furthermore, in other embodiments, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration1800B, as shown inFIG. 21B. Specifically, the imaging configuration1800B includes the light emitter module310and the image detection module320, the linear polarizer830and the analyzer840, as previously discussed with regard toFIG. 21A, with the addition of a pair of spectral filters1810and1810A. In particular, the spectral filter1810A and the polarizer830are operatively arranged for use in conjunction with the light emitter module310, while the spectral filter1810and the analyzer840are operatively arranged for use with the image detection module320. Accordingly, during operation of the imaging configuration1800A, the light emitter module310generates light322that is processed by spectral filter1810A and the linear polarizer830to illuminate the target object324. In addition, the light reflected by the target object324in response to being illuminated is then processed by the analyzer840and spectral filter1810before being detected as topography information of the target object324. Thus, when the spectral filters1810and1810A are used in combination with polarizer830and analyzer840, depth resolved topography of the target object324in a layered fashion can be achieved. For example, band-pass filters centered at about 400 nm, 500 nm, 600 nm, 700 nm, 800 nm can be placed in 2 filter wheels, whereby each filter wheel has a set of filters of 400 nm, 500 nm, 600 nm, 700 nm, 800 nm, such that one filter wheel is in operative arrangement with the image detection module320and the other filter wheel is in operative arrangement with the light emitter module310. When the analyzer840and the polarizer830are in a cross-polarization configuration, which permits orthogonal polarization states, the topography of the target object324can be scanned when both image detection module320and the and light emitter module310are both using the spectral filters1810at the certain wavelength (e.g. both are using 400 nm filters, then both use 500 nm filters; then both use 600 nm filters, etc.). As a result, depth resolved topography of the target object324can be obtained. The longer the wavelength, the deeper the depth where the topography of the target object324will be scanned. As such, the details of the target object324at different wavelengths can be obtained. It should be appreciated that tunable filters can be used instead of filter wheels of narrow band filters.

In addition to topography image data, reflectance images at different wavelengths (e.g. 400 nm, 500 nm, 600 nm, 700 nm, etc.) can be obtained using the configuration1800A. As the longer wavelengths penetrate deeper into the biological tissues or other media, longer wavelength reflectance image data (e.g. 800 nm image) carries more information about the target object324in its deeper layers than the short wavelength reflectance image data (e.g. 400 nm image). As such, subtraction of the two images obtained for different illumination wavelengths following normalization cancels out most of the image information arising from photons that were reflected before reaching the deeper depth where the embedded object is located. It should be appreciated that different illumination wavelengths may be created by having light emitter module310emit light322that is processed by different spectral filters1810A. For example, if the 400 nm reflectance image is expressed as I400(x,y) and the 800 nm reflectance image is expressed as I800(x,y), the normalized difference image (NDI) can be mathematically represented as INDI(x,y)=I800(x,y)−A·I400(x,y), where INDI(x,y) is the normalized difference image (NDI) and A is the normalization coefficient to cancel out most of the common information due to the objects at shallower penetration depth. In one aspect, the A will be calculated iteratively based on the similarities between I800(x,y) and I400(x,y). It should be appreciated that A can be a matrix of different values at different pixel locations. As such, the equation for the NDI image can be expressed as INDI(x,y)=I800(x,y)−A(x,y)·I400(x,y).

Projection of Fluorescence Information onto Tissue Surface of Target Object

It should be appreciated that prior generation image projection systems suffered from problems associated with the accurate projection of image information onto a non-flat surface, as such prior generation systems were designed to work with only with a flat surface. As such, when prior generation image projection systems are required to project images onto a non-flat, or curved surface, the projected image becomes distorted. That is, because typical projectors produce a larger image at a further distance than at a smaller distance, the non-flat projection surface, such as the target object324, introduces different projector-to-surface distances throughout different parts of the surface, which must be taken into account to ensure that the image projected onto a curved projection surface is sharp, and without distortion.

To overcome this problem of current projection systems, the system100of the present invention takes into account the surface of the object, such as the target object324, onto which an image is to be projected using a projection mapping process. Specifically, projection mapping process provided by the system100processes the image information to be projected based on the 3D shape of the surface upon which the image information is to be projected upon, such as the target object324. In addition, the projection mapping process analyzes and calculates the relative distances between the image projector and various portions/points/regions of the non-flat surface to be projected with images, such as the target object324. Based on the distance calculation, the system100generates a corrected image that takes the surface geometry of the surface onto which the image is to be projected (target object324) into account, so that the distortion of the projected image that is observed by the user is reduced or minimized.

Thus, in one embodiment of the system100, the 3D scanning module110and the imaging module120may be embodied in an imaging and projection configuration1900A, as shown inFIG. 22A. In particular, the configuration1900A includes the light emitter module310and the image detection module320, as previously discussed, however, the light emitter module310is configured to perform both surface scanning of the target object and to perform image projection for user visualization. The present invention is different from previous projection mapping technologies that processes the projection patterns based on 3D models created by a separate 3D scanner before the projection, as the present invention allows concurrent or simultaneous 3D scanning and image projection. Specifically, the frames of the light emitter module310when operating as an image projector can be interleaved. For example, if the total frame rate of the projector is 60 frames-per-second (FPS), ½ of the total frames (odd frames) may be used for 3D scanning, while the remaining ½ of total frames (even frames) are used for projection of images or videos onto a non-flat surface. To minimize confusion to the eyes of a viewer of the target object324, the 3D scanning frames of the target object324collected by the image detection module320may use near-infrared (NIR) wavelengths for illumination by the light emitter module310that are invisible to the eye of a viewer of the target object324, while the projection of the image/video back onto the target object324by the projection function of the light emitter module310may be performed using visible wavelengths.

In another embodiment of the system100, the 3D scanning module110and the imaging module120may be embodied in another imaging and projection configuration1900B, as shown inFIG. 22B. In particular, the configuration1900B includes the light emitter module310and the image detection module320, as discussed with regard toFIG. 22A, however the image detection module320is configured as a depth camera or 3D scanner. As such, the depth camera or 3D scanner provided by the image detection module320is able to obtain surface topography and a 3D model of the target object324, which is then used for processing of the images to be projected by the light emitter module310that is configured as the projector. Thus, when imaging a dynamic scene, which includes fast moving objects, sport scenes, and images as seen from a car's point of view, the projection mapping process is performed by the configuration1900B in real-time based on the data obtained from the depth camera of the image detection module320. It should be appreciated that any depth camera or 3D scanner can be used by the image detection module320. Specifically, the depth camera or 3D scanner320may utilize various topography scanning methods, including but not limited to conoscopic holography, modulated light, stereo camera, Fourier 3D scanning, low coherence interferometry, common-path interference 3D scanning, and contact profilometers.

In another embodiment of the system100, the 3D scanning module110and the imaging module120may be embodied in another imaging and projection configuration1900C, as shown inFIG. 22C. In particular, the configuration1900C includes the image detection module320and the light emitter module310, as previously discussed with regard toFIG. 22B, with the addition of another light emitter module310A. As such the light emitter module310is configured to perform 3D scanning functions, while the light emitter module310A functions as an image projector of images back onto the target object324using image projection mapping previously discussed.

In another embodiment of the system100, the 3D scanning module110and the imaging module120may be embodied in another imaging and projection configuration1900D, as shown inFIG. 22D. In particular, the configuration1900D includes the image detection module320and the light emitter module310, as configured in configuration1900A discussed with regard toFIG. 22A, with the addition of the fluorescence emission filter340and the fluorescence excitation filter410. In particular, the emission filter340is operatively arranged for use in conjunction with the image detection module320and the excitation filter410is operatively arranged for use in conjunction with the light emitting module310. It should also be appreciated that both the emission filter340and the excitation filter410may be configured as filter wheels, such that the emission filter340may be selectively moved into or out of the light detection path of the image detection module320and the excitation filter410may be selectively moved into or out of the light emission path of the light emitting module310. As such, configuration1900D allows fluorescence imaging by the imaging detection module320to be enhanced when the excitation filter410and the emission filter340are used.

In another embodiment of the system100, the 3D scanning module110and the imaging module120may be embodied in another imaging and projection configuration1900E, as shown inFIG. 22E. In particular, the configuration1900E includes the image detection module320, the light emitter module310, the fluorescence emission filter340, and the fluorescence excitation filter410, as previously discussed with regard toFIG. 22D, with the addition of the excitation light source450. In particular, the excitation light source450is configured to be used in conjunction with the excitation filter410during the capturing of fluorescence information of the target object324by the emission filter340and image detection module320. It should also be appreciated that both the emission filter340may be configured as filter wheel, which can be selectively moved into and out of the light detection path of the image detection module320. As such, configuration1900E allows fluorescence imaging by the imaging detection module320to be enhanced when the excitation filter410and the emission filter340are used.

In another embodiment of the system100, the 3D scanning module110and the imaging module120may be embodied in another imaging and projection configuration1900F, as shown inFIG. 22F. In particular, the configuration1900F includes the image detection module320and the light emitter module310, the fluorescence emission filter340and the fluorescence excitation filter410, as discussed with regard toFIG. 22D, with the addition of the additional light emitter module310A. In particular, the light emitter module310is configured as a light projector for 3D scanning, while the light emitter module310A is configured as an image projector, as discussed with regard toFIG. 22C. In particular, the emission filter340is operatively arranged for use in conjunction with the image detection module320and the excitation filter410is operatively arranged for use in conjunction with the light emitting module310. It should also be appreciated that both the emission filter340and the excitation filter410may be configured as filter wheels, such that the emission filter340may be selectively moved into or out of the light detection path of the image detection module320and the excitation filter410may be selectively moved into or out of the light emission path of the light emitting module310. As such, configuration1900F allows fluorescence imaging by the imaging detection module320to be enhanced when the excitation filter410and the emission filter340are used.

Vein Imaging

The system100may also be configured to facilitate imaging of a location where a human or animal vein target object324is located to facilitate intravenous placement of a needle or injection. To enable this imaging function, another embodiment of the system100, including the 3D scanning module110and the imaging module120may be embodied in an imaging and projection configuration2000A, as shown inFIG. 23A. In particular, the configuration2000A includes the light emitter module310and the image detection module320, whereby, the light emitter module310is configured to perform both 3D surface scanning of the target object and to perform image projection for user visualization, as previously discussed with regard toFIG. 22A, with the addition of an infrared light source450. However, it should be appreciated that the infrared light source450may comprise any suitable light source with other wavelengths. In particular, the imaging configuration2000A includes transmission imaging geometry, whereby the infrared light source450and the image detection module320are positioned on different or opposite sides of the tissues or target object324, whereby the portion2020of the emitted IR light2030that light passes through the tissue is imaged by the image detection module320. Red, near-infrared or infrared wavelength is preferred for this application as they penetrate deeper into biological tissues of the target object320. The system captures the 3D shape of the target object324and the projector310projects the images back to the biological tissues, or target object324, such as hands. As such, the vein and vasculatures of the target object hand324may be displayed back onto the target object tissues324to guide IV injection. The projector310also splits imaging frames between topography scanning of the target object324and image projection back onto the target object324in an interleaved manner, as previously discussed. The camera also splits imaging frames between transmission mode imaging and topography scanning. For example, the image detection module320and projector provided by the light emitter module310can synchronize their frame rate at 60 fps and share the same clock signal. For odd imaging frames (1, 3, 5, 7 . . . ), the projector310and the image detection module320work together to produce topography scanning of the target object324. For even imaging frames (2, 4, 6, 8, . . . ), the image detection module320works with the light source2010to generate a transmission mode imaging (e.g. vein imaging in hand324), and the projector310projects the processed image back onto the target object tissues324for visualization by the eyes of a viewer. It should be appreciated that the projected images are processed based on the 3D shape the target tissue surface324to minimize distortion. It should be also appreciated the configuration2000A may be used to image attenuation within any objects such as other organs, tissues or plastic parts, instead of imaging vasculatures.

In another embodiment, the system100, including the 3D scanning module110and the imaging module120may be embodied in an imaging and projection configuration2000B, as shown inFIG. 23B. In particular, the configuration2000B includes the light emitter module310, the image detection module320and the light source450whereby, the light emitter module310is configured to perform both 3D surface scanning of the target object and to perform image projection for user visualization, as previously discussed with regard toFIG. 23A. In addition, the light source450and image detection module320are both placed in operative arrangement with respective spectral filters1810and1810A of various wavelengths. For example, both of the spectral filters1810and1810A may be coupled with filters wheels, each of which also include sets of bandpass filters centered at 400 nm, 500 nm, 600 nm, 700 nm, 800 nm. In operation, both the light source450and image detection module320may both use 400 nm filters to capture an image of the target object324, and capture the 500 nm, 600 nm, 700 nm, 800 nm images in a similar fashion. If desirable, the normalized difference image (NDI) may be mathematically represented as INDI(x,y)=I800(x,y)−A·I400(x,y), where INDI(x,y) is the normalized difference image (NDI) and A is the normalization coefficient to cancel out most of the common information due to the objects at shallower penetration depth. It should be appreciated that any wavelengths can be chosen for multispectral imaging to facilitate imaging of specific biological structures or diseases. Similar to the examples previous discussed, the image detection module320, the projector310and the light source450will be synchronized and frames will be interleaved to allow concurrent multispectral imaging and topography scanning.

Projection Mapping and Distortion Compensation

The 3D shape and depth for the non-flat surface to be projected upon, such as the target object324, enables processing of images to achieve optimized projection and to minimize image distortion of the projected image. Thus, the 3D model of the target object324captured and generated by the present invention is linked to the physical non-flat target object324by a spatial transform, which maps individual points in real-time from the 3D model to the projection surface of the target object324so that the image of the 3D model is projected accurately onto the non-flat physical target object324surface with a one-to-one correspondence. The spatial transformation is calculated and processed by any suitable computing system, including the controller130, or any suitable computer system connected thereto by a wireless or wired connection. The spatial transformation calculation uses a transformation T between a 3D model space Xm, an object space Xo, and an image space Xion the projection surface of the target object324using Cartesian coordinates of a point (x,y,z) that are represented by (xw, yw, zw, w), wherein w is a scalar factor. It should be understood that this calculation can be done on a central processing unit (CPU) or graphics processing unit (GPU), or a combination thereof, or any suitable computing system, such as that previously discussed. The transformation T is a one-to-one mapping process, which includes the transformation T1from the 3D model space Xmto the object space Xoand the transformation T2from the object space Xoto the image on the object surface Xi. The transformation T rotates, scales, reflects, and shears the30model to the target object324in the object space using the coordinates formulated for T1and T2, consistent with standard image manipulation techniques. The T1and T2are governed by the following equations 1-3:
T1*Xm=Xo,  Eq. (1)
T2*Xo=Xi,  Eq. (2),
wherein Xm, Xo, and Xiare column vectors of 4 dimensions (xw, yw, zw, w) and T1and T2are 4×4 matrices.
Combining equation (1) and (2) provides:
T2*T1*Xm=Xi,  Eq. (3).

Thus, identifying the topography and having the 3D model from the system100of the present invention facilitates the calculation of the transformation matrix T1 and T2 for accurate projection mapping.

Projection Quality Monitoring, Projection Optimization and Least-Squares Errors

The quality of the image projection onto the target object324may be monitored in real-time by our system. For example, if a biological tissue lesion or wound is emitting near-infrared (NIR) fluorescence at 830 nm it is imaged by the present invention and projected back to the tissue surface of the target object324psedocolored in a color, such as green2050, as shown inFIG. 24, we can compare the projected image2050(in green color) with the true signal2052(near infrared signal) for quality monitoring and optimization. Briefly, the points from the true signal Pt(xt,yt) will be compared to corresponding points from the projected image Pp(xp,yp). The NIR and green color can be imaged by the system100. The correspondences are established based on feature matching and pattern recognition. A vector Vtp may be calculated pointing from Pt(xt, yt) to Pp(xp,yp) (eg. Vtp=Pp(xp,yp)−Pt(xt, yt)). The vector Vtp guides the optimization of the projection mapping by reducing the norm of the vector. Pluralities of mapping point pairs can be used for optimization and least-square error method can be used. For example, if n mapping point pairs (Pp(xp,yp), Pt(xt, yt)) are chosen, the combined square error can be expressed as:
Σi=1n[Ppi(xp,yp)−Pti(xt,yt)]2,
wherein [Ppi(xp,yp)−Pti(xt, yt)]2represents the sum of squared residuals. Optimization is achieved when the system100minimizes the sum of the squared residuals, whereby a residual is the difference between the projected image value and the imaging value. In one aspect, the optimization algorithm can be performed on an iterative basis. As such, the iterative algorithm may stop iterations when the calculated squares error sum is smaller than a value that has been predetermined/preset. It should be appreciated that the preset value can be either set manually or by machine learning algorithms that are trained by similar data sets.

It should be appreciated that this optimization can be applied to other optical imaging mechanisms other than fluorescence. For example, the vein imaging systems previously discussed with regard toFIGS. 23A-Bmay use this optimization algorithm. Similarly, multi-spectral imaging, oxygen saturation imaging with projection mapping on a non-flat surface of the target object324can also utilize this algorithm. It should also be appreciated that the real-time monitoring of the projected image on the non-flat surface of the target object324provides a technique to monitor projection quality dynamically.

System for Capturing Both 3D Shape Imaging, Fluorescence Imaging, Absorption Coefficient and Scattering Coefficient

In another embodiment of the system100, including the 3D scanning module110and the imaging module120may be configured, such that the absorption coefficient (μa) and the reduced scattering coefficient (μs′) may be obtained in addition to fluorescence image data/information and 3D topography data/information. Mapping of the optical properties of biological tissue of the target object324using spatial frequency domain imaging (SFDI) may be integrated into the system100, and carried out by the controller130, for characterization of the biochemical composition and structure of the target object tissue324. Modulated imaging is applied to analyze the diffuse reflectance from a spatially-modulated sinusoidal fringe illumination to provide the absorption coefficient (μa) and reduced scattering coefficient (μs′) maps. An SFDI system comprises an image projector embodied as the light emitter module310and the image detection module320, which are geometrically calibrated.

A procedure or process2100steps for acquisition of the optical property maps using spatially-modulated imaging is shown inFIG. 25. In particular, at step2110, the projector310illuminates two grayscale sinusoidal fringe patterns of the target object324at different spatial frequencies (fx) with 3 phase shifted by 120 degrees. Sequential illumination of the spatial frequency patterns are controlled by a program running on the controller/computer130and filters are used to generate spatial frequency illumination with different wavelengths. The reflected light will be acquired by the image detection module320and saved for processing. Low spatial frequency patterns (fx=0) and high spatial frequency patterns are employed to extract the DC and AC components of the diffusely reflected intensity (I), at step2120. Then, the demodulated AC (MAC) and DC (MDC) amplitudes will be calculated for each spatial location (xi) as

MAC⁡(xi,fx)=23⁢[I1⁡(xi)-I2⁡(xi)]2+[I2⁡(xi)-I3⁡(xi)]2+[I3⁡(xi)-I1⁡(xi)]2Eq.⁢(4)⁢MDC⁢⁡(xi)=13⁡[I1⁡(xi)+I2⁡(xi)+I3⁡(xi)],Eq.⁢(5)
where, I1, I2and I3are the image intensities with shifted spatial phases. Techniques employing transform functions, such as Fourier and Hilbert transforms, allow for the determination of demodulated intensities using a single frame of high spatial frequency patterns, thereby increasing imaging speed significantly and for use in real-time SFDI. The diffuse reflectance of an object (Rd) can be measured from the diffuse reflectance of a turbid phantom (Rd, ref) with known optical properties by:

Rd⁡(xi,fx)=MAC⁡(xi,fx)MAC,ref⁡(xi,fx)⁢Rd,ref⁡(fx).Eq.⁢(6)
Tissue-simulating phantoms are prepared by diluting and blending intralipid as a scattering agent and India ink as an absorber with distilled water, as indicated at step2130. Reflectance measurements of the homogeneous liquid phantoms with a wide range of absorption and scattering values are performed at step2140to acquire demodulated AC (MAC, ref) and DC (MDC, ref) images. The diffuse reflectance of the phantoms is predicted by applying forward models based on diffusion-based and transport-based approaches, such as diffusion approximation and white Monte Carlo simulations, for given sets of absorption and reduced scattering coefficients. The inversion methods, such as least-squares minimization and two-frequency lookup table approaches are utilized to extract optical properties of the object, as indicated at step2150. Analysis of the diffuse reflectance for all pixels produces absorption coefficient and reduced scattering coefficient maps which are then processed to merge with 3D models captured by the system.

It should be appreciated that the absorption coefficient (μa) and reduced scattering coefficient (μs′) images can be registered with fluorescence image, color reflectance image and preoperative images (MRI, CT, SPECT, PET, etc.) using the registration algorithm described previously above.

Deformation Compensation Using Biomechanical Modeling and FEM Modeling

Biological tissues, especially soft tissues, of the target object324are likely to deform. If the soft tissues have different deformation between preoperative imaging and intraoperative imaging, registration errors between the images will occur, if deformation is not properly addressed. Accordingly, the present invention allows 3D topography that is captured by the system100to be used to update the preoperative images and account for the tissue deformation.

Briefly, a surface mesh or volumetric mesh can be generated based on preoperative tomographic imaging data (e.g. MRI, CT, SPECT, PET, 3D ultrasound, OCT, etc.). Image segmentation can be performed to isolate the organ of interest. Features and landmarks can be extracted from both the preoperative image model (based on tomographic imaging data) and the intraoperative image model (3D topography captured by our invention). A first image registration is performed based on the features and landmarks identified. It should be appreciated that the features and landmarks identified tend to have less deformation. After image registration, the closest point distances between mesh nodes of interest and the intraoperatively deformed surface are calculated. It should be appreciated that closest point distances can be calculated based on the mesh data, the point cloud data or a combination thereof. Subsequently, boundary conditions will be generated based on the preoperative image data and the intraoperative image data previously registered. In one aspect, the boundary conditions comprise initial deformations of the tissue/organ of interest associated with mesh nodes or points. Subsequently, the model solutions can be calculated using finite element modeling (FEM) methods using the boundary conditions iteratively (step A). In one aspect, part of the closest point distances are used to determine a displacement boundary condition on the mesh nodes. After each iteration, the locations of new mesh nodes or point clouds of preoperative image data are updated to reflect the mechanical deformation, based on the FEM model solution calculated. Subsequently, a new boundary conditions can be generated and step A can be repeated. It should be appreciated that the partial differential equations are solved with boundary conditions set based on patient anatomy, body force, tissue material properties, etc. As the iterative algorithm is executed, the computer/controller130of the system100minimizes the least-squares errors for the closest point distances for all mesh nodes or corresponding points. The summation of square errors for all mesh nodes or points can be calculated dynamically and compared to a preset value. The iterative algorithm may stop iterations when the calculated squares error sum is smaller than the preset value. It should be appreciated that the preset value can be either set manually or by machine learning algorithms that were trained by similar data sets. It should be appreciated that the FEM can be performed either on the CPU or the GPU of the computer/controller.

Registration Algorithm Between 2 Image Detectors Based on Topography and Depth Profile

In a further embodiment, the 3D scanning module110and the imaging module120may be embodied in an imaging configuration2200, as shown inFIG. 26. Specifically, the imaging configuration2200includes the light emitter module310, the image detection module320, the emission filter340, the image detection module320A, and the excitation light source450which operates in conjunction with the fluorescence excitation filter410, as previously discussed with regard toFIG. 4F. Thus, the configuration2200, as well as other embodiments previously discussed, may use multiple image detection modules320, in conjunction with 3D scanning and depth sensing capacity provided by the present invention. Thus, the image registration between 2 different image detection modules,320and320A may be optimized based on the depth profile. For example, it is possible to register the images captured by image detector320(fluorescence images) to the images captured by image detector module320A (color images). The transformation matrix for image registration depends on the distance of the target object324to the image detector modules320and320A. In the present invention, the distance of the target object324is already known. Therefore, the image detector module320and the image detector module320A can be registered using the correct transformation matrix calculated for a fixed distance. For example, transformation matrices may be obtained for various distances and working distances (20 cm, 21 cm, 22 cm, etc.). With the distance/depth of the target object324captured by the present invention, the correct transformation matrix may be used for optimized registration results.

In one aspect, the registration matrices at various working distances are calibrated using calibration targets, such as a chessboard pattern. In addition, the transformation matrices at various distances can be obtained during the calibration process. For example, cameras320and320A may image the same calibration target, such as a chessboard pattern, and intersections and/or edges are identified. As such, the common features (e.g. intersections and/or edges) can be used for the calculation of transformation from the image captured by camera320to the image that is captured by camera320A at a given working distance. As the present invention offers depth information and topography data of the objects324imaged, the transformation matrices at the correct working distances are identified and used for image registration. For example, if the objects324are about 22 cm away from the system100, the system100will detect the depth and use the transformation matrix for 22 cm working distance for image registration between camera320and camera320A.

In another aspect, the registration may be performed between two 2D (two-dimensional) images. The transformation matrix is calculated and processed by the computer/controller130, which uses a transformation matrix Tdbetween the image space of the first detector/camera X1and the image space of the first detector/camera X2(equation 7), whereby
Td*X1=X2Eq. (7).

It should be appreciated that this calculation may be performed on central processing unit (CPU) or graphics processing unit (GPU), or a combination thereof. The transformation T rotates, scales, reflects, and shears the image from detector320to register with image from detector320A. It should also be appreciated that the registration algorithm can be either a rigid registration or a non-rigid registration. Using a calibration pattern, such as chessboard patterns, the transformation matrix Tdat each working distances can be obtained accurately. A set of values such as {T20, T21, T22, T23, T24} can be obtained for various working distances (20 cm, 21 cm, 22 cm . . . ) and depths. It should be appreciated that a tolerance range may be assigned to each matrix, whereas T20 can be assigned for a range between 19.1-21.0 cm, and T22 can be assigned to 21.1-23.0 cm. It should be further appreciated that the two image detection modules320and320A may be of different types. For example, image detection module320can be a silicon based CCD/CMOS camera for color imaging and image detection module320A can be a thermographic camera comprises of elements including, but not limited to indium antimonide, indium arsenide, mercury cadmium telluride (MGT), lead sulfide, lead selenide, amorphous silicon (a-Si), vanadium oxide (VOx), lanthanum barium manganite (LBMO), lead zirconate titanate (PZT), lanthanum doped lead zirconate titanate (PLZT), lead scandium tantalate (PST), lead lanthanum titanate (PLT), lead titanate (PT), lead zinc niobate (PZN), lead strontium titanate (PSrT), barium strontium titanate (BST), barium titanate (BT), antimony sulfoiodide (SbSI), and polyvinylidene difluoride (PVDF). In another example, image detector module320may be a near infrared (NIR) night vision camera and image detector module320A may be a thermographic camera, where the two types of images may be co-registered based on the distance of the object of interest324from system100.

In some circumstances, the target objects324being imaged are at different depths. For most accurate registration results, a depth-region-based-registration process may be used. Specifically, the images may be segmented into different regions based on the depth profile captured by the system100. For example, the images can be segmented into 2 regions, where region 1 has an object 20 cm away from the imaging system and region 2 has an object 30 cm away from the imaging system. For each region, the correct transformation matrix will be used, so T20(for 20 cm working distance) will used for registration of region 1 (X1r1) and T30(for 30 cm working distance) will used for registration of region 2 (X1r2). It should be appreciated that the image can be segmented into a plurality of regions, which is more than 2. It should be further appreciated that each region that is segmented may be represented by a new stack of images, where the pixels within the region still preserve the original intensity values, while the pixels outside of the regions may be assigned the intensity value of 0. As a result, the coordinates and dimensions of the images representing each of the regions remain the same, so each region-based image may be readily used with the original registration transformation matrices. In one aspect,

One mathematic representation is shown below;
Td1*X1r1=X2r1,Td2*X1r2=X2r2, . . . ,Tdi*X1ri=X2ri,
wherein Tdi is the transformation matrix for the ith region, X1riis the image space from detector/camera320for the ith region, X2riis the image space from detector/camera320for the ith region. As such, the previously generated stack of images may be transformed using the transformation matrices of the corresponding working distances. Subsequently, the final transformed image may be digitally synthesized by creating a composite image from the transformed stack of images of various working distances.

In one embodiment, the depth-region-based-registration algorithm or process may be carried out using the following process. Initially, the process calibrates the transformation matrices between 2 detectors, such as image detection modules320and320A, at different working distances. Next, the process obtains the depth profile of objects with the imaging field of view. In the following step, the images that are captured by the 2 detectors are segmented based on the depth profile. The process then generates a stack of images of various working distance, with each stack image containing objects324and a given or predetermined depth. Thus, for each stack image of a particular working distance, the process uses the transformation matrix of the corresponding working distance for registration of the images. Finally, the process adds/overlays the transformed stack images together to synthesize the registered final image.

It should also be appreciated that the depth-region-based-registration algorithm or process may be used to register a plurality of images, from more than 2 detectors or imaging devices. It should also be appreciated that the depth-region-based-registration algorithm or process may use depth profile or 3D topography information that is captured by instruments other than that provided by the system100, such as a 3D scanner or depth camera. The algorithm or process may also operate independently of the use of the image system100. In one aspect, the depth-region-based-registration algorithm or process may also be used to register a plurality of images using a known depth map or topography profile of the target objects324.

Endoscope Configuration

In some embodiments, the system100may be embodied in an imaging configuration2400A, as shown inFIG. 27A. Specifically, the configuration2400A includes the light emitting module310, the image detection module320, the emission filter340, with the addition of the beam splitter610, and an imaging probe2410. As such, the light emitter module310is configured to perform both surface scanning of the target object and to perform optical imaging such as fluorescence imaging and color reflectance imaging. Thus, the present invention allows concurrent or simultaneous 3D scanning and optical imaging in a cavity or other places where access is difficult by free space optics.

It should be appreciated that the imaging probe2410may be embodied in the form of an endoscope, a laparoscope or any scope that can enter or be placed into cavities within a human, animal, buildings, cars, or any other physical object. In one embodiment, the endoscope configuration of the imaging probe2410is similar to embodiments previously discussed, with the additional requirement that the system is miniaturized so that it can fit into a small cavity. Electrical cords with insulation can be used to deliver the power to the imaging probe2410via the duct, intestines, surgical ports or other access routes. In particular, light emitter module310, and the image detection module320and emission filter340, are arranged at an angle to each other, such as a substantially right angle, while the beam splitter610is positioned at an oblique angle, such as about a 45-degree angle, relative to the image detection module320and the image detection module320and the emission filter340. In addition, the configuration2400A includes lenses2420and2430, whereby lens2420is disposed at one end of the imaging probe2410proximate to the beam splitter610, and the other lens2430is positioned at the other end of the imaging probe2410, distal to the beam splitter610. As such, lens2430is suitably configured to enter the various cavities in which imaging is desired.

As such, during operation of the detection configuration2400A, the light emitter module310emits light322to illuminate the target object324via the imaging probe2410, whereupon the light326reflected and emitted by the target object324in response to being illuminated is passed through the beam splitter610for receipt by the image detection module320to capture topography and fluorescence data of the target object324. In some embodiments, the image data may be represented by a display device, such as LCD monitor or a wearable display. In addition, the projector embodied by the light emitter module310may also project the fluorescence image detected by the image detection module320through the imaging probe2410onto the target object324.

In other embodiments, the system100may be embodied in an imaging configuration2400B, as shown inFIG. 27B. Specifically, the configuration2400B includes the light emitting module310, the image detection module320, the emission filter340, and the imaging probe2410, as discussed with regard toFIG. 27A, but without the use of the beam splitter610. In addition, the configuration2400B includes an addition imaging probe2410A that include corresponding lenses2420A and2430A with the addition of the beam splitter610, and an imaging probe2410. As such, the light emitter module310is configured to perform both surface scanning of the target object and to perform optical imaging. Thus, the present invention allows concurrent or simultaneous 3D scanning and optical imaging. Particularly, during operation the imaging probe2410is used to capture topography information and fluorescence image information of the target object324, while the imaging probe2410A is used for projecting the pattern for 3D scanning onto the target object324, and provide fluorescence excitation light to the target object324.

In other embodiments, the system100may be embodied in an imaging configuration2400C, as shown inFIG. 27C. Specifically, the configuration2400C includes the light emitting module310, the image detection module320, the emission filter340, the beam spotter610, and the imaging probe2410, as discussed with regard toFIG. 27A, with the addition of the imaging probe2410A, a lens2460, the spectral filter1810and the fight source450. As such, the light emitter module310is configured to perform both surface scanning of the target object and optical imaging, such as fluorescence imaging and color reflectance imaging. As such, during operation of the configuration2400C, the spectral filter1810, and the lens2460process the light emitted from the light source450, whereupon the processed light passes through the imaging probe2410A to illuminate the target object324. In addition, the light emitter module310projects, via the beam splitter610and through the imaging probe2410, the pattern for 3D scanning of the target object324. In addition, the image detection module320receives through the beam splitter610the reflected and emitted light from the target object324in response to being illuminated, which passes through the internal detection device2410imaging probe2410, as topography and fluorescence information of the target object324. Thus, the present invention allows concurrent or simultaneous 3D scanning and optical imaging.

It should also be appreciated that the imaging probe2410of the configurations2400A-C may comprise a liquid light guide, relay lens system in lieu of fiber bundles. It should be further appreciated that color endoscopy, fluorescence endoscopy, oxygen saturation endoscopic imaging, hyperspectral endoscopic imaging can be enabled by the present invention, along with 3D surface topography scanning. It should be further appreciated that the endoscopic images can be registered with CT, MRI, PET, SPECT, and ultrasound images, or any other diagnostic image, using the methods previously discussed. It should be further appreciated that the various 3D scanning techniques described previously, such as laser triangulation or structured light illumination, may be implemented in the endoscope for 3D scanning. Specifically, the topography scanning module forming the light emitter module310in the endoscope configuration may be configured to utilize various topography scanning methods, including but not limited to conoscopic holography, modulated light, stereo camera, Fourier 3D scanning, low coherence interferometry, common-path interference 3D scanning, and contact profilometers.

It should also be appreciated that the components of the system100, including the light emitter module310, the image detection module320, the fluorescence detection module510, the projector910, the light source450, as well as the various filters340and410when implemented as moveable devices, or any other component of the system100requiring control, are coupled to, or are otherwise in a wired or wireless communication with, the controller130. Accordingly, the controller130, which may be embodied in any suitable computing device, such as a standalone computing device or portable computing device, which has the necessary hardware, software or combination thereof to coordinate and synchronize the operation of the components coupled thereto. Furthermore, in some embodiments, the controller130may be configured to communicate through a wired or wireless communication network with a remote computing device (e.g. a portable or standalone computing device) that may be configured to coordinate and synchronize the operation of the components of the system100, such that the controller130serves as an interface between the components of the system100and the remote computing device.

Therefore, one advantage of the present invention is that an imaging system is configured to simultaneously obtain topography and optical imaging data such as fluorescence information. Still another advantage of the present invention is that the imaging system enables topography and optical imaging data such as fluorescence information that are collected to be co-registered. Another advantage of the present invention is that a 3D point cloud/polygon mesh of a shape and fluorescence value is generated, which can be readily registered to preoperative imaging data, such as a PET/CT or MRI image data for example. Still another advantage of the present invention is that the imaging system utilizes a surface 3D profile to track surface deformation, and to provide surface-based registration. Still another advantage of the present invention is that the imaging system allows capturing and projecting of information onto a non-even (i.e. uneven, curved, contoured) surface in real-time or near real-time with low spatial latencies. Still another advantage of the present invention is that the imaging system enables accurate registration between multiple images based on topography and depth profile of a target object. Still another advantage of the present invention is that the imaging system enables depth-resolved scanning by using polarization gating and/or multispectral scanning. Still another advantage of the present invention is the imaging system enables a 3D scanning goggle, which can perform optical imaging. Still another advantage of the present invention is that the imaging system enables a 3D scanning endoscope to perform optical imaging.

Thus, it can be seen that the objects of the present invention have been satisfied by the structure and its method for use presented above. While in accordance with the Patent Statutes, only the best mode and preferred embodiments have been presented and described in detail, with it being understood that the present invention is not limited thereto or thereby. Accordingly, for an appreciation of the true scope and breadth of the invention, reference should be made to the following claims.