Calibrating RGB-D sensors to medical image scanners

A computer-implemented method for automatically calibrating an RGB-D sensor and an imaging device using a transformation matrix includes using a medical image scanner to acquire a first dataset representative of an apparatus attached to a downward facing surface of a patient table, wherein corners of the apparatus are located at a plurality of corner locations. The plurality of corner locations are identified based on the first dataset and the RGB-D sensor is used to acquire a second dataset representative of a plurality of calibration markers displayed on an upward facing surface of the patient table at the corner locations. A plurality of calibration marker locations are identified based on the second dataset and the transformation matrix is generated by aligning the first dataset and the second dataset using the plurality of corner locations and the plurality of calibration marker locations. Then, a translation is added to the transformation matrix corresponding to a maximum height associated with the apparatus.

TECHNOLOGY FIELD

The present invention relates generally to methods, systems, and apparatuses for calibrating RGB-D sensors with medical scanners into a single coordinate system. The disclosed techniques may be applied to, for example, Computed Tomography (CT) scanners, as well as scanners for other imaging modalities.

BACKGROUND

Over the past few years, several devices for non-radiographic and non-invasive patient monitoring have been introduced such as, for example, Microsoft Kinect and Asus Xtion Pro. These devices, referred to herein as “RGB-D sensors.” offer a high degree of precision of 3D dense surface data in real time. The data captured from an RGB-D sensor can be utilized for assist medial image acquisition workflow in several ways. For example, using an RGB-D sensor in operating rooms, surgeons can navigate through medical images during procedures. In order to use the information obtained from the RGB-D sensor for medical scanner acquisition related tasks, the two imaging systems should be calibrated into a single coordinate system.

Any calibration approach must not impose any additional requirements on the environment where the medical scanner is placed. For instance, algorithms must not require specific size of the room or height of ceiling. This ensures that the techniques are applicable to both RGB-D sensors mounted on the medical scanners (possibly during assembly at industry) as well as sensors mounted on ceiling or external fixtures. In other words, the calibration should be carried out by technicians for each individual medical scanner during the scanner setup. Therefore, a simple calibration solution that does not require technicians to have knowledge of camera calibration is preferred.

SUMMARY

Embodiments of the present invention address and overcome one or more of the above shortcomings and drawbacks, by providing methods, systems, and apparatuses related to technologies enabling an interactive, inside view of the patient using various image scanner modalities. An RGB-D camera acquires the observed person's surface in 3D space. A detector recognizes salient corresponding landmarks between an RGB-D cue and a scanner modality. The detected landmarks support the automated registration with scanner modalities.

According to some embodiments of the present invention, a computer-implemented method for automatically calibrating an RGB-D sensor and an imaging device using a transformation matrix includes using a medical image scanner (e.g., a CT scanner, a MRI scanner, or a PET scanner) to acquire a first dataset representative of an apparatus attached to a downward facing surface of a patient table, wherein corners of the apparatus are located at a plurality of corner locations. The corner locations are identified based on the first dataset and the RGB-D sensor is used to acquire a second dataset representative of a plurality of calibration markers displayed on an upward facing surface of the patient table at the corner locations. In some embodiments, each calibration marker is a distinct color and may comprise a dot pattern. Calibration marker locations are identified based on the second dataset and the transformation matrix is generated by aligning the first dataset and the second dataset using the corner locations and the calibration marker locations. Then, a translation is added to the transformation matrix corresponding to a maximum height associated with the apparatus.

Various refinements, additions, or other modifications may be made to the aforementioned method in different embodiments of the present invention. In some embodiments, the method further includes acquiring an image using the medical image scanner and using the transformation matrix to place the image in a coordinate system associated with the RGB-D sensor. In other embodiments, the method further includes acquiring RGB-D data using the RGB-D sensor and using the transformation matrix to place the RGB-D data in a coordinate system associated with the medical image scanner. In some embodiments, prior to using the RGB-D sensor to acquire the second dataset, the patient table is moved to a lowest and out-most position.

According to other embodiments, a second computer-implemented method for automatically calibrating an RGB-D sensor and an imaging device includes receiving a user request for auto-calibration and, in response to the user request, projecting a predefined RGB pattern on a patient table using a projector associated with a medical image scanner (e.g., a CT scanner, a MRI scanner, or a PET scanner). The RGB-D sensor is used to capture an image of the patient table with the predefined RGB pattern. The predefined RGB pattern is identified in the image and a coordinate transformation is estimated between the projector and the RGB-D sensor using the predefined RGB pattern.

Various refinements, additions, or other modifications may be made to the aforementioned second method in different embodiments of the present invention. For example, in some embodiments, the method further comprises acquiring a new image using the medical image scanner and using the coordinate transformation to place the new image in a coordinate system associated with the RGB-D sensor. In other embodiments, the second method further includes acquiring RGB-D data using the RGB-D sensor and using the coordinate transformation to place the RGB-D data in a coordinate system associated with the medical image scanner.

According to other embodiments, a system for image coordinate system calibration includes a patient table, an apparatus, a plurality of calibration markers, a medical image scanner (e.g., a CT scanner, a MRI scanner, or a PET scanner), an RGB-D sensor, and a computer comprising at least one processor. The patient table comprises an upward facing surface and a downward facing surface. The apparatus is attached to the downward facing surface of the patient table and the corners of the apparatus are located at a plurality of corner locations. Calibration markers are displayed at plurality of calibration marker locations on the upward facing surface of the patient table above the corner locations. In some embodiments, each of the calibration markers is a distinct color and comprises a dot pattern. The medical image scanner included in the system is configured to acquire a first dataset representative of the apparatus and the RGB-D sensor is configured to acquire a second dataset representative of the calibration markers. The computer's processor is configured to identify the corner locations based on the first dataset; identify the calibration marker locations based on the second dataset; generate a transformation matrix by aligning the first dataset and the second dataset using the corner locations and the calibration marker locations; and add a translation to the transformation matrix corresponding to a maximum height associated with the apparatus.

In some embodiments of the aforementioned system, the processor is further configured to use the transformation matrix to place new image data acquired with the medical image scanner into a coordinate system associated with the RGB-D sensor. In other embodiments, the processor is further configured to use the transformation matrix to place new RGB-D data acquired with the RGB-D sensor into a coordinate system associated with the medical image scanner.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

The following disclosure describes the present invention according to several embodiments directed at methods, systems, and apparatuses related to the automatic and semi-automatic calibration processes that can accurately align an RGB-D sensor to a medical image scanner (e.g., a CT scanner). Once the two imaging systems are calibrated, the information and data can be directly transferred from one system to the other. Moreover, the image scanner can offer high quality volume data as the ground truth data to the RGB-D sensor. For clarity, some embodiments of the subject matter disclosed herein are described with respect to a CT image scanner. However, in light of the present disclosure, those skilled in the art will recognize that the suggested calibration techniques described herein may be used with any image scanner system generally known in the art, including scanners associated with Magnetic Resonance Imaging (MRI) and Positron Emission Tomography (PET) modalities.

FIG. 1provides an overview of the visualization system100that may be applied in a clinical setting, according to some embodiments. The visualization system includes an RGB-D Sensor105which is configured to combine RGB color information with depth information. Various types of RGB-sensors generally known in the art may be utilized in implementing the visualization techniques described. In the example ofFIG. 1, the RGB-D Sensor105uses structured light technology. Using such technology, the RGB-D Sensor105projects an infrared pattern125. The reflection of the infrared pattern125is captured by an infrared camera in the RGB-D Sensor105. The captured data is then compared to reference patterns stored in the RGB-D Sensor105to estimate pixel depth. This depth data is correlated to an RGB camera in the RGB-D Sensor105to yield an RGB image with depth information (i.e., “RGB-D” data). Examples of a commercial RGB-D sensors using structured light technology include the first generation of the Microsoft Kinect and Asus Xtion PRO. It should be noted that structured light is only one example of sensing technology that can be used for implementing the RGB-D Sensor105. In other embodiments, other RGB-D technologies may be used such as, for example, Time of Flight (ToF) cameras implemented by devices such as the second generation of the Microsoft Kinect.

The RGB-D Sensor105is mounted to the ceiling of a room in which an imaging device is installed. In the example ofFIG. 1, this imaging device is a Computed Tomography (CT) Imaging Device comprising a patient table110and the gantry120. CT is an imaging technology that uses computer-processed X-ray beams to produce tomographic images of specific areas of a scanned patient. The patient lies on the patient table110and is moved during the examination along a system axis130, also referred to as the z-axis, into the measurement field generated in the gantry120. Inside the gantry120, the patient is showered with X-ray beams Each X-ray beam comprises bundles of energy (or “photons”) which, depending on the structure of the imaged material, may pass through, be absorbed by, or be redirected (i.e., scattered) by the structure. The degree to which an X-ray beam is reduced by an object during imaging is referred to as attenuation. Based on this attenuation, a 3D image of the patient's internal anatomy is generated.

In order to use the information obtained from the RGB-D sensor105for CT acquisition related tasks, the two imaging systems are calibrated into a single coordinate system. InFIG. 1, this calibration process is performed by an operator computer115. The operator computer115receives RGB-D and CT image data from the RGB-D Sensor105and the CT imaging device, respectively, over a computer network (not shown inFIG. 1). Once the data has been received, various calibration procedures may be applied.FIGS. 2-9below describe these calibration procedures in detail, as well as different ways the system shown inFIG. 1may be configured to perform these procedures.

FIG. 2provides an illustration of a visualization system200where auto-calibration is performed using a calibration apparatus225B. Based on a CT scan, the surface of the calibration apparatus225B is retrieved. If this surface is also visible to the depth sensor, the calibration can be preceded by aligning several common key points in the data of the two systems. Based on this assumption, the two systems may be calibrated. The calibration apparatus225B is placed at a fixed location on the table and visible to the depth sensor. Circle marks may be placed on the patient table210where the calibration apparatus225B can be installed. For example, as shown inFIG. 2, the calibration apparatus225B may be used to replace a factory designed headrest225A installed on the patient table210. It should be noted that the composition of the calibration apparatus may vary according to the imaging modality. For example, if an MRI image scanner is used instead of a CT image scanner, the calibration apparatus would be comprised of magnetic resonant materials.

In the example ofFIG. 2, the calibration apparatus225B has sharp corners and flat surfaces. The corners can be automatically detected by the RGB-D sensor205and used to initialize the calibration. The surfaces may have more dense points that can be used to build a more accurate correspondence of the two data sources, and then refine the initial calibration result. It should be noted that various characteristics of the calibration apparatus225B may be changed in different embodiments of the present invention. For example, the calibration apparatus225B shown inFIG. 2is step-shaped. In other embodiments, the dimensions of the shapes may be changed or, alternatively, different shapes may be used.

FIG. 3provides a flow chart illustrating a calibration method300where the system shown inFIG. 2is applied. At step305, the calibration apparatus is installed to a pre-defined place on the table. Next at step310, the table is moved to the lowest and out-most position and depth data is acquired using the RGB-D sensor. At step315, the CT imaging device is used to perform a CT scan of the calibration apparatus and acquire corresponding CT image data. Next, at step320, the corner information is extracted from the depth data and the CT image data. This corner information is used to calculate the initial transformation. At step325, the depth data is aligned to the CT image data with the initial transformation and additional correspondences are determined with the iterative closest point (ICP)-based surface registration method. Then, at step330, the transformation is refined with the new correspondences. The accuracy of this calibration method300can be affected by several aspects, such as the corner detection result and the depth data noise level. To address the first aspect, in some embodiments, the system200illustrated inFIG. 2has a mode that allows a technician to manually annotate the set of corners. For the second aspect, the accuracy can be improved by iteratively running the last two steps of the calibration procedure.

FIG. 4provides an illustration of an alternative visualization system400where auto-calibration is performed with CT and RGB data, according to some embodiments. RGB-D sensors generally offer higher resolution RGB images than depth images. Compared to the systems and methods discussed above with respect toFIGS. 2 and 3, an auto calibration method with RGB data can accurately estimate the transformation by using only a few landmarks due to the high precision of both CT and RGB data. However, the landmarks in RGB images (usually special patterns) cannot be directly used for calibration because those patterns are not visible in the CT data. Therefore, in various embodiments, the RGB data is calibrated to the CT with apparatus integrated to the CT table, which is shown inFIG. 4.

More specifically,FIG. 4shows two views400and405illustrating how several calibration markers (e.g., dot patterns) with different colors may be attached to the CT table surface. The positions of these markers in the image coordinate system can be automatically detected from the RGB images with standard computer vision technologies generally known in the art. One or more apparatuses may be installed beneath the CT table and the corners of the apparatus(es) are aligned with the markers. Various types of apparatuses may be used within the context of the present invention, including those made of wood, plastic, and composite materials. The defining feature of the apparatus(es) is that it has well-defined corners such that the corners are detectable and identifiable within the CT data. It should be noted that the composition of the apparatus(es) may vary according to the imaging modality. For example, if an MRI image scanner is used instead of a CT image scanner, the apparatus(es) would be comprised of magnetic resonant materials. Using the corner and marker information, the calibration transformation can be calculated by aligning the markets to the corners of the apparatus(es). Additionally, a translation is added equal to the maximum height of the apparatus(es).

FIG. 5provides a flow chart illustrating a calibration method500where the system illustrated inFIG. 4is applied. At step505, the apparatus is installed in a pre-defined place beneath the CT table as shown in view400ofFIG. 4. Next, at step510, the table is moved to the lowest and out-most position and an RGB snapshot of the markers is acquired using the RGB-D sensor. At step515, the locations of the markers in the RGB image are detected. Next, at step520, a CT scan is performed of the apparatus and, at step525, the corners are extracted from the acquired CT data. At step530, the RGB markers are aligned to the corners to determine the transformation matrix. Then, at step535, a translation is added to the transformation matrix that equals to the apparatus height.

FIG. 6provides an illustration of another alternative visualization system600where auto-calibration is performed with a laser beam, according to some embodiments. The gantry620has a laser beam indicator used for aligning the patient prior to performing a CT scan. In this example, an apparatus625comprised of a cube630on top of a 3-axis tripod platform is used. The cube has guidelines inscribed or otherwise marked on its outer surface (or, alternatively, the cube's inner surface if the cube is transparent or translucent). The apparatus625is attached to a flat bottom surface aligned with the surface of the patient table610. The CT laser alignment lines are then projected onto the cube of the apparatus625as shownFIG. 6. These lines are assumed to be aligned with the CT coordinates. The cube is adjusted such that its guidelines align with the laser alignment lines. The RGB-D Sensor605is then used to acquire an image of the cube with the projected lines.

FIG. 7provides a flow chart illustrating a calibration method700where the system shown inFIG. 6is applied, according to some embodiments. Starting at step705, the apparatus is mounted on the table at a predefined fixed position. Next, at step710, guidelines are projected on the cube using the laser beam indicator provided by the CT machine (seeFIG. 5). Also at step710, the cube is manually adjusted by an operator such that the guidelines are aligned with laser alignment lines. Then, at step715, the table is moved so the cube is within the field of view (FOV) of the RGB-D sensor. At step720, the cube top surface normal and the bottom surface normal are estimated and, at step725, the intersection of guidelines on the cube top surface are detected by the RGB-D sensor. Steps715-725may be repeated at several different table positions to provide robust transformation estimation between the RGB-D sensor, patient table, and CT machine.

FIG. 8provides an illustration of another alternative visualization system800where auto-calibration is performed with a laser projector, according to some embodiments. No external apparatus is required here. A laser projector835attached to the gantry820is used to project a pattern on the patient table810. The exact pattern projected by the laser projector835can vary and a combination of different patterns may be used in some embodiments. The RGB-D sensor805detects the pattern and performs the calibration procedure.

FIG. 9illustrates a calibration method900where the system shown inFIG. 8is applied, according to some embodiments. The calibration process is started at step905when an operator pushes an “auto-calibrate” button (e.g., on the CT device or via an operator interface). In response to the activation of the button, at step910, the laser projector projects a pre-defined RGB pattern on the table. Next, at step915, the RGB-D device is triggered and a screenshot of the table is captured. Then, at step920, the projected pattern is detected in the RGB-D screenshot image and, at step925, the transformation between the laser projector (CT) and the RGB-D camera is estimated using the projected pattern.

FIG. 10provides a table1000showing a comparison of the techniques described above with reference toFIGS. 2-9. For comparison, the first column provides the corresponding features for the conventional calibration methods of expert annotation tools and a phantom CT scan. In this table1000, the number of circles in a cell indicates how beneficial a particular technique is compared to the other techniques. For example, the gantry integrated projected is completely automated and is hence marked with3circles.

The techniques described herein may be applied in clinical settings to provide enhanced support to surgical operations. For example, in some embodiments, a device equipped with positioning capabilities is used to interactively track the live patient surface using data gathered from the RGB-D sensor and the image scanner. The pose is used to update a display, either on the positioning device, or separately, with the rendering of the scanner modalities to be fused with the live view to achieve a see-through effect. The see-through techniques enable the joint continuous visualization of body hull and internal body structures at various levels of detail.

FIG. 11illustrates an exemplary computing environment1100within which embodiments of the invention may be implemented. For example, this computing environment1100may be used to implement the methods300,500,700,900illustrated inFIGS. 3, 5, 7, and 9, respectively. The computing environment1100may include computer system1110, which is one example of a computing system upon which embodiments of the invention may be implemented. Computers and computing environments, such as computer system1110and computing environment1100, are known to those of skill in the art and thus are described briefly here.

As shown inFIG. 11, the computer system1110may include a communication mechanism such as a bus1121or other communication mechanism for communicating information within the computer system1110. The computer system1110further includes one or more processors1120coupled with the bus1121for processing the information. The processors1120may include one or more central processing units (CPUs), graphical processing units (GPUs), or any other processor known in the art.

The computer system1110also includes a system memory1130coupled to the bus1121for storing information and instructions to be executed by processors1120. The system memory1130may include computer readable storage media in the form of volatile and/or nonvolatile memory, such as read only memory (ROM)1131and/or random access memory (RAM)1132. The system memory RAM1132may include other dynamic storage device(s) (e.g., dynamic RAM, static RAM, and synchronous DRAM). The system memory ROM1131may include other static storage device(s) (e.g., programmable ROM, erasable PROM, and electrically erasable PROM). In addition, the system memory1130may be used for storing temporary variables or other intermediate information during the execution of instructions by the processors1120. A basic input/output system (BIOS)1133containing the basic routines that help to transfer information between elements within computer system1110, such as during start-up, may be stored in ROM1131. RAM1132may contain data and/or program modules that are immediately accessible to and/or presently being operated on by the processors1120. System memory1130may additionally include, for example, operating system1134, application programs1135, other program modules1136and program data1137.

The computer system1110also includes a disk controller1140coupled to the bus1121to control one or more storage devices for storing information and instructions, such as a hard disk1141and a removable media drive1142(e.g., floppy disk drive, compact disc drive, tape drive, and/or solid state drive). The storage devices may be added to the computer system1110using an appropriate device interface (e.g., a small computer system interface (SCSI), integrated device electronics (IDE), Universal Serial Bus (USB), or FireWire).

The computer system1110may also include a display controller1165coupled to the bus1121to control a display1166, such as a cathode ray tube (CRT) or liquid crystal display (LCD), for displaying information to a computer user. The computer system includes an input interface1160and one or more input devices, such as a keyboard1162and a pointing device1161, for interacting with a computer user and providing information to the processor1120. The pointing device1161, for example, may be a mouse, a trackball, or a pointing stick for communicating direction information and command selections to the processor1120and for controlling cursor movement on the display1166. The display1166may provide a touch screen interface which allows input to supplement or replace the communication of direction information and command selections by the pointing device1161.

The computer system1110may perform a portion or all of the processing steps of embodiments of the invention in response to the processors1120executing one or more sequences of one or more instructions contained in a memory, such as the system memory1130. Such instructions may be read into the system memory1130from another computer readable medium, such as a hard disk1141or a removable media drive1142. The hard disk1141may contain one or more datastores and data files used by embodiments of the present invention. Datastore contents and data files may be encrypted to improve security. The processors1120may also be employed in a multi-processing arrangement to execute the one or more sequences of instructions contained in system memory1130. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions. Thus, embodiments are not limited to any specific combination of hardware circuitry and software.

As stated above, the computer system1110may include at least one computer readable medium or memory for holding instructions programmed according to embodiments of the invention and for containing data structures, tables, records, or other data described herein. The term “computer readable medium” as used herein refers to any medium that participates in providing instructions to the processor1120for execution. A computer readable medium may take many forms including, but not limited to, non-volatile media, volatile media, and transmission media. Non-limiting examples of non-volatile media include optical disks, solid state drives, magnetic disks, and magneto-optical disks, such as hard disk1141or removable media drive1142. Non-limiting examples of volatile media include dynamic memory, such as system memory1130. Non-limiting examples of transmission media include coaxial cables, copper wire, and fiber optics, including the wires that make up the bus1121. Transmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

The computing environment1100may further include the computer system1110operating in a networked environment using logical connections to one or more remote computers, such as remote computer1180. Remote computer1180may be a personal computer (laptop or desktop), a mobile device, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to computer system1110. When used in a networking environment, computer system1110may include modem1172for establishing communications over a network1171, such as the Internet. Modem1172may be connected to bus1121via user network interface1170, or via another appropriate mechanism.

The embodiments of the present disclosure may be implemented with any combination of hardware and software. In addition, the embodiments of the present disclosure may be included in an article of manufacture (e.g., one or more computer program products) having, for example, computer-readable, non-transitory media. The media has embodied therein, for instance, computer readable program code for providing and facilitating the mechanisms of the embodiments of the present disclosure. The article of manufacture can be included as part of a computer system or sold separately.

A graphical user interface (GUI), as used herein, comprises one or more display images, generated by a display processor and enabling user interaction with a processor or other device and associated data acquisition and processing functions. The GUI also includes an executable procedure or executable application. The executable procedure or executable application conditions the display processor to generate signals representing the GUI display images. These signals are supplied to a display device which displays the image for viewing by the user. The processor, under control of an executable procedure or executable application, manipulates the GUI display images in response to signals received from the input devices. In this way, the user may interact with the display image using the input devices, enabling user interaction with the processor or other device.

The functions and process steps herein may be performed automatically or wholly or partially in response to user command. An activity (including a step) performed automatically is performed in response to one or more executable instructions or device operation without user direct initiation of the activity.