Patent Publication Number: US-2021177370-A1

Title: Patient viewing system

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application claims priority from U.S. Provisional Patent Application No. 62/721,516, filed on Aug. 22, 2018 and U.S. Provisional Patent Application No. 62/771,534, filed on Nov. 26, 2018, each of which is incorporated herein by reference in their entirety 
    
    
     BACKGROUND OF THE INVENTION 
     1). Field of the Invention 
     This invention relates to a patient viewing system. 
     2). Discussion of Related Art 
     Endoscopic catheters are used by surgeons, doctors and other viewers to capture images of body parts within patients. A bronchoscope, for example, is an endoscopic catheter that is used to inspect segmental bronchi. Catheters also serve other functions, such as other endoscopic viewing functions, to treat diseases or to perform surgical procedures. 
     Because a tip of a catheter is inserted into the body of a patient with minimal or no invasive surgery, the viewer cannot see the location of the tip of the catheter with the naked eye. To assist the viewer, a screen is provided with a combination of images. These images commonly include computed tomography (CT) images that are generated preoperatively and intraoperatively, along with live video data from a catheter camera in the tip of the catheter. These images are usually provided in different quadrants of the display. 
     In the case of bronchi, all bronchi tunnels look the same. As a result, a viewer is frequently guessing where the bronchus is that they are currently navigating. Such a viewer will frequently take multiple CT scans as they push the bronchoscope through the bronchi in order to determine where they are in the lungs. 
     SUMMARY OF THE INVENTION 
     The invention provides a patient viewing system including a catheter having a lumen and a tip, a tip tracking device that detects movement of the tip, left and right projectors, left and right light wave guides connected to the left and right projectors, a processor, a computer-readable medium connected to the processor, a data store on the computer-readable medium and a set of instructions stored on the computer-readable medium and executable by the processor. The set of instructions may include 1) a catheter tracking system that is connected to the tip tracking device and receives a measurement based on movement detected by the tip tracking device and determines a position of the tip based on the measurement and store the position of the tip in the data store, and 2) a stereoscopic analyzer connected to the data store to receive the image data, the stereoscopic analyzer determining left and right image data sets, the left and right projectors projecting the left and right image data sets respectively, the left and right image data sets differing from one another to give the viewer a perception of a three-dimensional rendering. 
     The invention further provides a method of viewing a patient including inserting a tip of a catheter into a body of a patient, detecting movement of the tip with a tip tracking device, receiving a measurement based on movement detected by the tip tracking device, determining a position of the tip based on the measurement, storing the position of the tip, determining left and right image data sets based on the position of the tip, generating light in a pattern representative of the position of the tip using left and right projectors projecting the left and right image data sets respectively as light, and guiding the light to retinas of left and right eyes of a viewer so that the viewer sees position of the tip, the left and right image data sets differing from one another to give the viewer a perception of a three-dimensional rendering. 
     The invention also provides a patient viewing system including a catheter having a lumen and a tip, a tip tracking device that detects movement of the tip, a projector, a light wave guides connected to the projector, a processor, a computer-readable medium connected to the processor, a data store on the computer-readable medium and a set of instructions stored on the computer-readable medium and executable by the processor. The set of instructions may include 1) a catheter tracking system that is connected to the tip tracking device and receives a measurement based on movement detected by the tip tracking device and determines a position of the tip based on the measurement and store the position of the tip in the data store, 2) a past path calculator that stores a past path of the tip in the data store, and 3) a catheter display integrator, the catheter display integrator displaying the past path of the tip together with position of the tip. 
     The invention further provides a method of viewing a patient including inserting a tip of a catheter into a body of a patient, detecting movement of the tip with a tip tracking device, receiving a measurement based on movement detected by the tip tracking device, determining a position of the tip based on the measurement, storing the position of the tip, determining left and right image data sets based on the position of the tip, generating light in a pattern representative of the position of the tip using left and right projectors projecting the left and right image data sets respectively as light, guiding the light to retinas of left and right eyes of a viewer so that the viewer sees position of the tip, the left and right image data sets differing from one another to give the viewer a perception of a three-dimensional rendering, storing a past path of the tip in the data store, and displaying the past path of the tip together with position of the tip. 
     The invention also provides a patient viewing system including a catheter having a lumen and a tip, a tip tracking device that detects movement of the tip, a projector, a light wave guides connected to the projector, a processor, a computer-readable medium connected to the processor, a data store on the computer-readable medium and a set of instructions stored on the computer-readable medium and executable by the processor. The set of instructions may include 1) a catheter tracking system that is connected to the tip tracking device and receives a measurement based on movement detected by the tip tracking device and determines a position of the tip based on the measurement and store the position of the tip in the data store, 2) a prospective path calculator that calculates a future path of the tip based the position of the tip, the catheter display integrator displaying the future path, and 3) a catheter display integrator, the catheter display integrator displaying the future path of the tip together with position of the tip. 
     The invention further provides a method of viewing a patient including inserting a tip of a catheter into a body of a patient, detecting movement of the tip with a tip tracking device, receiving a measurement based on movement detected by the tip tracking device, determining a position of the tip based on the measurement, storing the position of the tip, determining left and right image data sets based on the position of the tip, generating light in a pattern representative of the position of the tip using left and right projectors projecting the left and right image data sets respectively as light, guiding the light to retinas of left and right eyes of a viewer so that the viewer sees position of the tip, the left and right image data sets differing from one another to give the viewer a perception of a three-dimensional rendering, calculating a future path of the tip based the position of the tip, and displaying the future path of the tip together with position of the tip. 
     The invention also provides a patient viewing system including a transmitter, an energy source connected to the transmitter to activate the transmitter, a body of a patient being positionable relative to the transmitter for the transmitter to generate a forward wave at a body part within the body, a receiver positionable relative to the body to detect a return wave from the body part, the return wave from the body part being in response to the forward wave created by the transmitter, a processor, a computer-readable medium connected to the processor, a data store on the computer-readable medium and a set of instructions stored on the computer-readable medium and executable by the processor. The set of instructions may include 1) a raw data reception unit that receives raw data of the return wave detected by the receiver and stores the raw data in the data store, 2) an image generation unit connected to the data store to process the raw data of the return wave to create image data representing an image and store the image data in the data store, 3) an image data reception unit that receives the image data from the data store, 4) a projector connected to the image data reception unit to receive the image data, the projector generating light in a pattern representative of the image data, and 5) a light wave guide connected to the projector to guide the light to a retina of an eye of a viewer while light from an external surface of the body transmits to the retina of the eye so that the viewer sees the external surface of the body augmented with a rendering of the body part. 
     The invention further provides a method of viewing a patient including activating a transmitter to generate a forward wave at a body part within the body, detecting, with a receiver, a return wave from the body part, the return wave from the body part being in response to the forward wave created by the transmitter, receiving raw data of the return wave detected by the receiver, storing the raw data in a data store, processing the raw data of the return wave to create image data representing an image, storing the image data in the data store, receiving the image data from the data store, generating light in a pattern representative of the image data, and guiding the light to a retina of an eye of a viewer while light from an external surface of the body transmits to the retina of the eye so that the viewer sees the external surface of the body augmented with a rendering of the body part. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The invention is further described by way of example with reference to the accompanying drawings, wherein: 
         FIG. 1  is a block diagram of a patient viewing system according to an embodiment of the invention; 
         FIG. 2  is a partial perspective view and partial block diagram of a CT scanner, a data reception unit, an image generation unit and a data store forming part of the patient viewing system, according to some embodiments; 
         FIG. 3  is a partial perspective view and partial block diagram of a display system, a catheter and a data store forming part of the patient viewing system, according to some embodiments; 
         FIG. 4  is a block diagram illustrating a catheter integration system forming part of the display system in  FIG. 3  and further illustrates the catheter, according to some embodiments; 
         FIG. 5  is a perspective view illustrating a viewer in the form of a surgeon, the viewer seeing a body of a patient and a rendering of a body part inside the patient and further seeing a rendering of a tip of a catheter and a past path of the tip, according to some embodiments; 
         FIG. 6  is top plan view of  FIG. 5 , according to some embodiments; 
         FIG. 7  is a view as seen by the viewer, according to some embodiments; 
         FIG. 8  is a view similar to  FIG. 6  after the viewer has moved counterclockwise around the body of the patient and has moved their head counterclockwise to keep sight of the body of the patient, according to some embodiments; 
         FIG. 9  is a view similar to  FIG. 7  showing how the body of the patient and a rendering is modified within the view, according to some embodiments; 
         FIG. 10  illustrates renderings that are shown to the viewer in  FIGS. 7 and 9  in enlarged detail, according to some embodiments; 
         FIG. 11  is an enlarged view of a portion of a view as seen by the viewer in  FIGS. 7 and 9 , according to some embodiments; 
         FIG. 12  is an enlarged view of live video data and a mesh as seen by the viewer in the views of  FIGS. 7 and 9 , according to some embodiments; 
         FIG. 13  is a graph illustrating three degrees of freedom of movement of the tip of the catheter and movement from a first position to a second position by a first amount, according to some embodiments; 
         FIG. 14  is a view of the tip of the catheter in the first position, according to some embodiments; 
         FIG. 15  is view of the catheter in the second position, according to some embodiments; 
         FIG. 16  is a view similar to  FIG. 16  illustrating a path extending through a lumen and a path that does not extend through the lumen to avoid injury, according to some embodiments; 
         FIG. 17  is a view similar to  FIG. 16  illustrating a calculated future path of the tip that is displayed as a rendering to the viewer in three-dimension, according to some embodiments; 
         FIG. 18  is a block diagram of a machine in the form of a computer that can find application in the present invention system, in accordance with one embodiment of the invention; and 
         FIG. 19  illustrates a timing relationship for adjusting medical instrument inputs as a function of patient observed position changes, in accordance with one embodiment of the invention. 
     
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       FIG. 1  of the accompanying drawings illustrates a patient viewing system  20 , according to an embodiment of the invention, that includes a CT scanner  22 , a data store  24 , a catheter  26 , and a display system  28 . 
     The data store  24  is connected to the CT scanner  22 . Raw data from the CT scanner  22  may be stored in the data store  24 . The data store  24  also stores image data that is based on the raw data. 
     The display system  28  is connected to the data store  24  to be able to retrieve the image data from the data store  24 . The catheter  26  is connected to the display system  28  so that the display system  28  can retrieve measurement and video data from the catheter  26  for further processing or for display to a viewer. 
     In use, a patient is located at a station  32  at the CT scanner  22 . A body  30  of the patient is scanned with the CT scanner  22  to obtain raw data that the CT scanner  22  stores in the data store  24 . The raw data is then processed to obtain 3D image data. 
     The patient is transferred from the station  32  at the CT scanner  22  to a station  34  at the display system  28 . A viewer uses the display system  28  to view the body  30  of the patient. The display system  28  also retrieves the image data from the data store  24 . The viewer uses the display system  28  to view an image in the form of a 3D rendering of the body  30  of the patient. The viewer inserts the catheter  26  into the body  30 . The display system  28  retrieves data from a tip of the catheter  26  for further processing or for display to the viewer. 
       FIG. 2  illustrates components of the patient viewing system  20 , including the CT scanner  22 , the data store  24 , an energy source  36 , a data reception unit  38 , and an image generation unit  40 , according to some embodiments. 
     The CT scanner  22  includes a base  42 , a platform  44 , a rotor  46 , an x-ray transmitter  48 , and a plurality of x-ray detectors  50 . 
     The platform  44  is secured to the base  42  through a mechanism (not shown) that permits translational movement of the platform  44  relative to the base  42 . An actuator such as a stepper motor (not shown) is operable to cause translational movement of the platform  44  relative to the base  42 . 
     The rotor  46  has an opening  52 . The x-ray transmitter  48  is secured to the rotor  46  on one side of the opening  52  and the x-ray detectors  50  are secured to the rotor  46  on an opposing side of the opening  52 . The rotor  46  is mounted to the base  42  around the platform  44 . The platform  44  moves relative to the opening  52  during its translational movement. A motor (not shown) is connected between the base  42  and the rotor  46  and is operable to rotate the rotor  46  around the platform  44 . 
     The energy source  36  may be connected to the x-ray transmitter  48  through a switch  54 . The x-ray detectors  50  are connected to the data reception unit  38 . The data reception unit  38  may be a software unit that resides on a computer-readable medium of a computer. The data store  24  resides on the computer-readable medium. The computer-readable medium may be a single computer-readable medium or may be separated within one personal computer or a number of personal computers that are connected to one another on a network. The data reception unit  38  is connected to the data store  24 , either directly or over a network. 
     The image generation unit  40  may be a computer program that resides on the computer-readable medium. The image generation unit  40  is connected to the data store  24 , either directly or over a network. 
     In use, an operator of the CT scanner  22  places the patient with their body  30  laying on the platform  44 . The motor connected between the base  42  and the rotor  46  is then switched on so that the rotor  46  rotates in a direction  58  about the platform  44  and the body  30  of the patient. The operator also switches the motor on that moves the platform  44  in a translation direction relative to the base  42  so that the platform  44  moves in a direction  60  relative to the rotor  46 . The operator then connects the switch  54  between the energy source  36  and the x-ray transmitter  48  to activate the x-ray transmitter  48 . The x-ray transmitter then generates a forward x-ray wave  62 . 
     The body  30  of the patient is positioned relative to the x-ray transmitter  48  so that the forward x-ray wave  62  penetrates the body  30  to a body part (not shown) within the body  30 . For purposes of this example, the body parts that are scanned are the lungs of a patent. A lung has many bronchi through which a catheter can travel. It may also be possible for a catheter to travel though hollow passages in the heart, arteries and veins of the blood circulation system etc. The system described herein may also find application for viewing internal body parts without the use of a catheter for vision, surgery or intervention, for example for viewing a growth within the abdomen, for analyzing the internal functioning of a knee, etc. The body part reduces the energy of the forward x-ray wave  62 . Different materials within the body part reduce the energy by different amounts. One of the x-ray detectors  50  is positioned relative to the body  30  to detect a return x-ray wave  64  from the body part. The return x-ray wave  64  from the body part is being detected in response to the forward x-ray wave  62  and is essentially the forward x-ray wave  62  that has reduced power because of the reduction in the power by the body part. Further forward x-ray wave  66  is also illustrated. The further x-ray waves are generated between the forward x-ray waves  62  and  66  and are detected by respective ones of the x-ray detectors  50 . In this manner, return x-ray waves are received from different parts of the body part. 
     The x-ray transmitter  48  and x-ray detectors  50  rotate together with the rotor  46  around the body part within the body  30  of the patient. In this manner, the body part may be scanned from different angles to create a two-dimensional “slice” of the anatomy. CT scans are capable of showing bone, organs, soft tissue. Subsequent slices are taken by moving the platform  44  in the direction  60 . Each slice thus represents two-dimensional data and the slices together represent data in three-dimensions of the body part. 
     The data reception unit  38  receives raw data of the return x-ray wave  64  from the x-ray detectors  50 . The raw data includes a time sequenced correlation between an angle of the x-ray transmitter  48  relative to the body part within the body  30  of the patient, an energy detected by each one of the x-ray detectors  50 , the location of each one of the x-ray detectors  50 , and a position of the platform  44 . The data reception unit  38  stores the raw data as raw data  68  of the return x-ray wave detected by the x-ray detectors  50 . 
     When enough raw data  68  of the body part is collected, the operator disconnects the switch  54  and stops the platform  44 . The operator then stops the rotor  46  and removes the patient from the platform  44 . 
     The image generation unit  40  retrieves the raw data  68  from the data store  24 . The image generation unit  40  generates image data based on the raw data  68 . The image data includes a three-dimensional rendering of the body part. The image generation unit  40  then stores the image data as image data  70  in the data store  24 . The data store  24  may be a single data store or may be distributed between platforms, and as such, the raw data  68  and the image data  70  can be located within a single data store within a personal computer or within several data stores within several personal computers. 
       FIG. 3  illustrates components of the patient viewing system  20  in more detail, and shows the data store  24  (holding the image data  70 ), the catheter  26 , and the display system  28 , according to some embodiments. 
     The catheter  26  includes a lumen  76  and a tip  78  attached to an end of the lumen  76 . The lumen is an elongate member (e.g. the cavity of a tubular part) that forms most of the length of the catheter  26 . The lumen  76  includes a mechanism (not shown) that is operable to move the tip  78  in at least four orthogonal directions and all directions in between the orthogonal directions. The tip  78  is thus steerable with the mechanism in the lumen  76 . The lumen has a hollow bore that is sufficiently large to hold the mechanism that is used to steer the tip together with any electrical cables and/or an optic fiber that may be required for relaying signals from the tip through the lumen  76  to the display system  28 . 
     The catheter  26  further includes a catheter inertial motion unit (IMU)  80  and a catheter camera  82  secured to the tip  78 . The catheter IMU  80  may for example be a semiconductor chip that has a number of measurement devices formed therein. The measurement devices include one or more gyroscopes and one or more accelerometers. Measurements from the gyroscopes and accelerometers, individually or in combination, provide data that indicates movement of the tip  78 . Such movement can be tracked in six degrees of freedom, for example translation in x-, y- and z-directions and rotation about x-, y-, and z-axes. 
     The catheter camera  82  has a lens (not shown) on the side of the tip  78  opposing the lumen  76 . The catheter camera  82  is positioned to capture images in the form of live video data in an area in front of the tip  78 , i.e. on a side opposing the lumen  76 . There may be multiple light sources and multiple cameras on different sides of the tip of the camera, although for ease of discussion it will be assumed that there is only a single camera, for example a built-in camera and light source on a distal end of the catheter. 
     The display system  28  includes a head-mountable frame  86 , left and right projectors  88 A and  88 B, left and right wave guides  90 A and  90 B, detection devices  92 , and vision algorithms  94 . The left and right projectors  88 A and  88 B, left and right wave guides  90 A and  90 B and the detection devices  92  are secured to the head-mountable frame  86 . The head-mountable frame  86  is shaped to be mounted to a head of a viewer. Components of the head-mountable frame  86  may, for example, include a strap (not shown) that wraps around the back of a head of a viewer. 
     The left and right projectors  88 A and  88 B are connected to power supplies. Each projector  88 A or  88 B has a respective input for image data to be provided to the respective projector  88 A or  88 B. The respective projector  88 A or  88 B, when powered, generates light in a two-dimensional pattern and emanates the light therefrom. The left and right wave guides  90 A and  90 B are positioned to receive the light from the left and right projectors  88 A and  88 B, respectively. The left and right wave guides  90 A and  90 B are transparent wave guides. 
     The detection devices  92  include a head unit IMU  100  and one or more head unit cameras  102 . The head unit IMU  100  includes one or more gyroscopes and one or more accelerometers. The gyroscopes and accelerometers are typically formed in a semiconductor chip and are capable of detecting movement of the head unit IMU  100  and the head-mountable frame  86 , including movement along three orthogonal axes and rotation about three orthogonal axes. 
     The head unit cameras  102  continually capture images from an environment around the head-mountable frame  86 . The images can be compared to one another to detect movement of the head-mountable frame  86  and the head of the viewer. 
     The vision algorithms  94  include an image data reception unit  106 , a display positioning algorithm  108 , a catheter integration system  110 , a display adjustment algorithm  112 , an image processing system  114 , and a stereoscopic analyzer  116 . The image data reception unit  106  is connected to the data store  24  through a direct connection or over a network. The components of the vision algorithm  94  are linked to one another through subroutines or calls. Through such subroutines and calls, the image data reception unit  106  is linked via the display positioning algorithm  108  to the stereoscopic analyzer  116 . 
     The catheter integration system  110  may be connected to the catheter IMU  80  and the catheter camera  82  through conductors in the lumen  76 . One of ordinary skill in the art will appreciate that the vision algorithms  94  reside on a computing system and that the catheter integration system  110  receives signals from the catheter camera  82  and the catheter IMU  80  and that such signals may convert from analog or digital data to computer software data. The catheter integration system  110  may be connected through subroutines or calls to the stereoscopic analyzer  116 . 
     The display adjustment algorithm  112  and the image processing system  114  are connected to the head unit IMU  100  and the head unit cameras  102  respectively. Such connections are through conductors and, where applicable, through inverters that convert analog or digital data to computer software data. The display adjustment algorithm  112  may be connected through subroutines and calls to the display positioning algorithm  108 . The image processing system  114  may be connected though calls and subroutines to the display adjustment algorithm  112 . 
     In use, a viewer mounts the head-mountable frame  86  to their head. The left and right wave guides  90 A and  90 B are then located in front of left and right eyes  120 A and  120 B of the viewer. 
     The image data reception unit  106  retrieves the image data  70  from the data store  24  and provides the image data  70  to the display positioning algorithm  108 . The display positioning algorithm  108  enters the image data  70  into the stereoscopic analyzer  116 . The image data  70  is three-dimensional image data of the body part as described above. The stereoscopic analyzer  116  analyzes the image data  70  to determine left and right image data sets based on the image data  70 . The left and right image data sets are data sets that represent two-dimensional images that differ slightly from one another for purposes of giving the viewer a perception of a three-dimensional rendering. The image data  70  is a static data set which does not change over time. 
     The stereoscopic analyzer  116  enters the left and right image data sets in to the left and right projectors  88 A and  88 B. The left and right projectors  88 A and  88 B then create left and right light patterns  122 A and  122 B. The components of the display system  28  are shown in plan view and the left and right light patterns  122 A and  122 B are shown in front elevation views. Each light pattern  122 A and  122 B includes a plurality of pixels. For purposes of illustration, light rays  124 A and  126 A from two of the pixels are shown leaving the left projector  88 A and entering the left wave guide  90 A. The light rays  124 A and  126 A reflect from sides of the left wave guide  90 A. It is shown that the light rays  124 A and  126 A propagate through internal reflection from left to right within the left wave guide  90 A, although it should be understood that the light rays  124 A and  126 A also propagate in a direction into the paper using refractory and reflective systems. The light rays  124 A and  126 A exit the left light wave guide  90 A through a pupil  128 A and then enter the left eye  120 A through a pupil  130 A of the left eye. The light rays  124 A and  126 A then fall on a retina  132 A of the left eye  120 A. In this manner, the left light pattern  122 A falls on the retina  132 A of the left eye  120 A. The viewer is given the perception that the pixels that are formed on the retina  132 A are pixels  134 A and  136 A that the viewer perceives to be at some distance on a side of the left wave guide  90 A opposing the left eye  120 A. 
     In a similar manner, the stereoscopic analyzer  116  enters the right image data set into the right projector  88 B. The right projector  88 B transmits the right light pattern  122 B, which is represented by pixels in the form of light rays  124 B and  126 B. The light rays  124 B and  126 B reflect within the right wave guide  90 B and exit through a pupil  128 B. The light rays  124 B and  126 B then enter through a pupil  130 B of the right eye  120 B and fall on a retina  132 B of the right eye  120 B. The pixels of the light rays  124 B and  126 B are perceived as pixels  134 B and  136 B behind the right light wave guide  90 B. 
     The patterns that are created on the retinas  132 A and  132 B are individually perceived as left and right images  140 A and  140 B that are shown in front elevation view. The left and right images  140 A and  140 B differ slightly from one another due to the functioning of the stereoscopic analyzer  116 . The left and right images  140 A and  140 B are perceived in a mind of the viewer as a three-dimensional rendering. 
     As mentioned, the left and right wave guides  90 A and  90 B are transparent. Light from a real-life object on a side of the left and right wave guides  90 A and  90 B opposing the eyes  120 A and  120 B can project through the left and right wave guides  90 A and  90 B and fall on the retinas  132 A and  132 B. In particular, light from a surface of the body  30  of the patient falls on the retinas  132 A and  132 B so that the viewer can see the surface of the body  30  of the patient. An augmented reality is created wherein the surface of the body  30  of the patient that the viewer sees is augmented with a three-dimensional rendering that is perceived by the viewer due to the left and right images  140 A and  140 B that are, in combination, perceived by the viewer. 
     The head unit IMU  100  detects every movement of the head of the viewer. Should the viewer, for example, move counterclockwise around the body  30  of the patient and simultaneously rotate their head counterclockwise to continue to see the body  30  of the patient, such movement will be detected by the gyroscopes and accelerometers in the head unit IMU  100 . The head unit IMU  100  provides the measurement from the gyroscopes and accelerometers to the display adjustment algorithm  112 . The display adjustment algorithm  112  calculates a placement value and provides the placement value to the display positioning algorithm  108 . The display positioning algorithm  108  modifies the image data  70  to compensate for movement of the head of the viewer. The display positioning algorithm  108  provides the modified image data  70  to the stereoscopic analyzer  116  for display to the viewer. 
     The head unit cameras  102  continually capture images as the viewer moves their head. The image processing system  114  analyzes the images by identifying images of objects within the image. The image processing system  114  analyzes movement of the objects to determine a pose position of the head-mountable frame  86 . The image processing system  114  provides the pose position to the display adjustment algorithm  112 . The display adjustment algorithm  112  uses the pose position to further refine the placement value that the display adjustment algorithm  112  provides to the display positioning algorithm  108 . The display positioning algorithm  108  thus modifies the image data  70  based on a combination of motion sensors in the head unit IMU  100  and images taken by the head unit cameras  102 . 
     The catheter integration system  110  may detect a location of the tip  78  of the catheter  26  before the viewer inserts the tip  78  in to the body  30  of the patient. The viewer subsequently inserts the tip  78  in to the body  30  of the patient. The tip  78  is then not visible to the viewer. The catheter IMU  80  provides signals to the catheter integration system  110  that indicate every movement of the tip  78 . The catheter integrations system  110  can thus track the position of the tip  78  using the motion sensors in the catheter IMU  80 . Unlike the image data  70  that is static, the position of the tip  78  changes over time. The catheter integration system  110  provides the position of the tip  78  to the stereoscopic analyzer  116 . The position of the tip  78  may be dynamic in that it changes over time and moves in three-dimensions. The stereoscopic analyzer  116  positions the tip  78  within the left and right image data sets that are inserted into the left and right projectors  88 A and  88 B. The viewer can thus see the location in the tip  78  within the left and right images  140 A and  140 B. The location of the tip  78  varies slightly within the left and right images  140 A and  140 B so that the viewer perceives the location of the tip  78  in three-dimensions. The rendering of the location of the tip  78  as provided by the left and right images  140 A and  140 B changes over time as the tip  78  makes its way through the body  30  of the patient. Such movement of the location of tip  78  as a rendering changes in three-dimensions so that the viewer perceives the rendering of the tip  78  as moving in three-dimensions, i.e. left, right, up, down, forward, backward, etc. 
     The catheter camera  82  continues to capture video data and provides the video data to the catheter integration system  110 . The catheter integration system  110  provides the video data to the stereoscopic analyzer  116 . The stereoscopic analyzer  116  places the video data at a fixed location within the view of the viewer unless or until a user interaction event is detected indicating the location should change. The video data changes over time as different images are captured by the catheter camera  82 . 
     The vision algorithms  94  are a set of instructions that are stored together with the data store  24  on a computer-readable medium. The set of instructions are executable by a processor to carry out the method described above. The computer-readable medium that stores the vision algorithms  94  may be located on a belt pack worn by the viewer. 
       FIG. 4  illustrates components of the patient viewing system  20  in more detail, in particular, components of the catheter integration system  110  and their relationship with the catheter IMU  80  and the catheter camera  82  in the tip  78  and the stereoscopic analyzer  116 . 
     The catheter integration system  110  includes a catheter tracking system  150 , a past path calculator  152 , a mesh generator  154 , a prospective path calculator  156 , a video data reception unit  158 , and a catheter display integrator  160 . The catheter tracking system  150  is connected to the catheter IMU  80 . The catheter tracking system  150  calculates a position of the tip  78  based on movement detected by the catheter IMU  80 . The catheter IMU  80  includes a number of tip tracking devices, including a number of gyroscopes and accelerometer to track its movement in six degrees of freedom. The catheter tracking system  150  stores a current position of the tip  78  as a position  162  in the data store  24 . The catheter tracking system  150  continues to monitor the catheter IMU  80 , continues to calculate a current position of the tip  78 , and continues to store a current position of the tip  78  as a current position  162  in the data store  24 . 
     The catheter display integrator  160  receives the current position  162  from the data store  24  and provides the current position  162  to the stereoscopic analyzer  116 . The stereoscopic analyzer  116  displays the current position  162  of the tip  78  as a rendering to the viewer so that the viewer can see the position of the tip  78  as a rendering in three-dimensions. 
     Past path calculator  152  retrieves every position  162  at every moment in time from the data store  24 . The past path calculator  152  calculates a past path of the tip  78  in three-dimensions and stores the past path as a past path  164  in the data store  24 . The catheter display integrator  160  receives the past path  164  from the data store  24  and provides the past path  164  to the stereoscopic analyzer  116 . The stereoscopic analyzer  116  displays the past path  164  to the viewer as a three-dimensional rendering. 
     The mesh generator  154  retrieves the past path  164  from the data store and generates a three-dimensional mesh around the past path  164 . The mesh generator  154  then stores the mesh as a mesh  166  in the data store  24 . The catheter display integrator  160  retrieves the mesh  166  from the data store  24  and provides the mesh  166  to the stereoscopic analyzer  116 . The stereoscopic analyzer  116  displays the mesh  166  to the viewer. The stereoscopic analyzer  116  creates a three-dimensional rendering of the mesh  166  that, in some embodiments, overlays the past path  164 . 
     The perspective path calculator  156  retrieves every position  162  of the tip  78  from the data store  24  and calculates a future path of the tip  78  based on the position  162  and past positions retrieved from the data store  24 . The perspective path calculator  156  then stores the future path as a future path  168  in the data store  24 . The catheter display integrator  160  retrieves the future path  168  from the data store  24  and provides the future path  168  to the stereoscopic analyzer  116 . The stereoscopic analyzer  116  displays the future path  168  to the viewer as a three-dimensional rendering. 
     The video data reception unit  158  receives live video from the catheter camera  82 . The video data reception unit  158  provides the live video data to the catheter display integrator  160 . The catheter display integrator  160  provides the live video data to the stereoscopic analyzer  116 . The stereoscopic analyzer  116  displays the live video data to the viewer. The live video data is a two-dimensional display that is displayed to the viewer at a certain predetermined distance in three-dimensional space. The catheter display integrator also integrates the mesh  166  with the video data from the video data reception unit  158  so that the mesh  166  is displayed on the video data. As the video data changes, with a changing position of the catheter  26  within the body  30  of the patient, the mesh  166  also changes accordingly. 
       FIG. 5  illustrates the use of the patient viewing system  20  as hereinbefore described by a viewer  172  in the form of a surgeon who uses the catheter  26  as a bronchoscope for purposes of examining a body part  174  comprising segmental bronchi in lungs of a patient, according to some embodiments. 
     The viewer  172  can see the body  30  of the patient through the left and right wave guides  90 A and  90 B. The body part  174  is inside the body  30  of the patient, thus the viewer cannot see the real (i.e. physical) body part  174 . 
     The viewer  172  also sees a three-dimensional rendering  176  which is based on the image data  70  as hereinbefore described. In the particular embodiment, the rendering  176  is located next to the body  30  of the patient. The rendering  176  is included in the figure to show the location where the viewer  172  perceives the rendering  176  relative to the body  30  of the patient, although it will be understood that, from the viewpoint of the reader of this document, the rendering  176  does not exist in the real world. The insert  180  shows that the viewer  172  can see a three-dimensional rendering  182  of the body part  174  as part of the rendering  176 . 
     The viewer  172  inserts the tip  78  of the catheter  26  into a mouth of the patient. The viewer  172  then progresses the tip  78  into the body part  174 . The locations of the tip  78  are monitored at instances that are closely spaced in time as hereinbefore described, and its past path is stored in three-dimensions. Sampling times may vary depending on the use case, and optimizations are possible, such as only capturing data while the endoscope is inside the patient&#39;s body, or after the user activates a “start recording/sampling” feature. The insert  184  shows that the rendering  176  includes a rendering  186  of the location of the tip  78  in three-dimensions and a rendering  188  of the past path of the tip  78  in three-dimensions. The renderings  182 ,  186 , and  188  may be displayed to the viewer  172  simultaneously so that the viewer sees the renderings  186  and  188  within the rendering  182 . 
       FIG. 6  is a top plan view showing a location of the viewer  172  relative to the body  30  of the patient and further illustrates the location of the rendering  176  within the view of the viewer  172 , according to some embodiments. The rendering  176  may be placed in any position relative to the body  30  of the patient, based on user preference, pre-programmed default settings, or any other suitable means. The particular relative location of body  30  of the patient to the rendering  176  in  FIG. 6  is for illustration purposes only and should in no way be considered limiting. 
       FIG. 7  illustrates a view  192  as seen by the viewer  172  in  FIG. 6 , according to some embodiments. The viewer  172  can see the actual body  30  of the patient and the rendering  176 . The view  192  further includes a live video based on the video data that is captured by the catheter camera  82  in  FIG. 4 . The view  192  further shows a mesh  196  that overlays the video  194 . The mesh  196  is a display of the mesh  166  in  FIG. 4 . 
     In  FIG. 8 , the viewer  172  has moved counterclockwise around the body  30  of the patient and has also rotated their head counterclockwise to continue to see the body  30  of the patient, according to some embodiments. The display adjustment algorithm  112  detects the movement of the head of the viewer  172  and adjusts a positioning of the rendering  176  accordingly so that the rendering  176  appears to remain stationary relative to the body  30  of the patient within the view of the viewer  172 . 
     In  FIG. 9 , the body  30  of the patient has rotated clockwise relative to  FIG. 7 , according to some embodiments. The rendering  176  has also rotated clockwise so that it remains stationary relative to the body  30  of the patient. The location of the live video  194  has however not changed from the view  192  in  FIG. 7  to the view  192  in  FIG. 9 . The viewer  172  thus sees the live video  194  and the mesh  196  in the same location and these components do not move upon movement of the head of the viewer  172 . The viewer  172  can thus view the body  30  of the patient and the rendering  176  from different sides and angles without losing sight of the live video  194  and the mesh  196 . The purpose of the mesh  196  may be to assist the viewer in guiding the tip  78  of the catheter  26  when the viewer  172  inserts the tip  78  into a passage in the body part  174  a second time after the mesh has been created, or during removal of the catheter as the catheter moves through the same path in the opposite direction. Some embodiments may have different viewing configurations for the virtual content (e.g. mesh  196 , live video  194 , rendering  176 ), in which some or all of the virtual content is fixed relative to real world coordinates, or are fixed relative to the viewer. 
       FIG. 10  shows components of the rendering  176  that are displayed to the viewer that are too small to be seen in the views of  FIGS. 7 and 9 , according to some embodiments. The viewer  172  sees the renderings  182 ,  186  and  188  of the body part  174 , the tip  78  and the past path of the tip. The viewer also sees a three-dimensional rendering of the mesh  196 . The mesh  196  is shown separated from the renderings  182 ,  186  and  188  for purposes of illustration, although it should be understood that the mesh  196  may overlay the rendering  182  of the body part  174 . 
     As shown in  FIGS. 11 and 12 , in some embodiments, the viewer  172  sees the mesh  196  in two locations, namely as part of the rendering  176  ( FIG. 11 ) and overlaying the live video  194  ( FIG. 12 ). 
       FIG. 13  illustrates the functioning of the perspective path calculator  156  in  FIG. 4 , according to some embodiments. The graph illustrates rotation of the tip  78  of the catheter  26  about x- y- and x-axes over time. Rotation about each axis may be analyzed over a short amount of time to determine a first amount  200  of movement from a first position  202  to a second position  204 . The first amount  200  of movement may be used to calculate a prediction of future movement of the tip  78 . 
       FIG. 14  illustrates the tip  78  in the first position  202 , according to some embodiments. In  FIG. 15 , the tip  78  has moved from the first position  202  to the second position  204  by the first amount  200 , according to some embodiments. The first amount  200  shown in  FIG. 15  is the sumtotal of all vectors of all movements about all axes and in all translation directions. 
     In  FIG. 14 , an assumption can be made that the direction that the tip  78  will follow, if the tip  78  is further inserted into the body part  174  is along an extension  206  of the lumen  76 .  FIG. 16  shows an extension  208  of the lumen  76  after the lumen  76  has moved by the first amount  200 , according to some embodiments. A viewer will typically not progress the tip  78  along the path of the extension  208  because it will make contact with the body part  174  and cause injury. Instead, the viewer  172  will prefer to follow a path  210  in order to avoid injury. The path  210  is displaced by a second amount  212  from the second position  204  to a third position  214 . The first amount  200  and the second amount  212  are measured in the same direction for ease of reference. 
       FIG. 17  illustrates an actual path  218  that the tip  78  will likely follow, according to some embodiments. The path  218  leaves the path  208  and approaches the path  214  as the viewer inserts the tip  78  further into the body part  174 . The stereoscopic analyzer  116  displays the path  218  to the viewer  172  in three-dimensions. 
       FIG. 18  shows a diagrammatic representation of a machine in the exemplary form of a computer system  900  within which a set of instructions, for causing the machine to perform any one or more of the methodologies discussed herein, may be executed, according to some embodiments. In alternative embodiments, the machine operates as a standalone device or may be connected (e.g., networked) to other machines. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein. 
     The exemplary computer system  900  includes a processor  902  (e.g., a central processing unit (CPU), a graphics processing unit (GPU) or both), a main memory  904  (e.g., read only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), etc.), and a static memory  906  (e.g., flash memory, static random access memory (SRAM), etc.), which communicate with each other via a bus  908 . 
     The computer system  900  may further include a disk drive unit  916 , and a network interface device  920 . 
     The disk drive unit  916  includes a machine-readable medium  922  on which is stored one or more sets of instructions  924  (e.g., software) embodying any one or more of the methodologies or functions described herein. The software may also reside, completely or at least partially, within the main memory  904  and/or within the processor  902  during execution thereof by the computer system  900 , the main memory  904  and the processor  902  also constituting machine-readable media. 
     The software may further be transmitted or received over a network  928  via the network interface device  920 . 
     The computer system  900  includes a laser driver chip  950  that is used to drive projectors to generate laser light. The laser driver chip  950  includes its own data store and its own processor  962 . 
     While the machine-readable medium  922  is shown in an exemplary embodiment to be a single medium, the term “machine-readable medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The term “machine-readable medium” shall also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present invention. The term “machine-readable medium” shall accordingly be taken to include, but not be limited to, solid-state memories, optical and magnetic media, and carrier wave signals. 
     The implementation described above uses a CT scanner  22  to scan the body part  174 . A CT scanner has transmitter in the form of an x-ray transmitter, a receiver in the form of an x-ray detector, and transmits and receives waves in the form of x-ray waves. It may be possible to use other scanning devices that use other transmitters and receivers and transmit and detect different waves. For example, a sonar system uses a sound transmitter to transmit a sound wave and a sound receiver to receive a sound wave. A visual system may include a light source that is inserted into the body part that transmits a light wave and have a camera that is located within the body part that captures a light wave reflected from the body part. 
     CT scanners are, however, preferred over other scanning devices because CT scanners provide very highly detailed raw data of the body part in three-dimensions and such data can readily be converted with an image generation unit to create three-dimensional image data. CT data also as the advantage that it can include data with respect to particular substances, materials, and densities of materials. The implementation described shows a rendering  176  placed next to the body  30  of a patient in the view  192  of the viewer  172 . It may also be possible to match the rendering with the body  30  of the patient so that the rendering of the body part be where the actual body part is and the rendering of the tip of the catheter is where the actual position of the tip of the catheter is. 
     Aspects of the invention can also be implemented without a catheter. For example, it may be possible to scan a body of a patient to determine a growth and for a viewer to use a display system to overlay a rendering of the growth in three-dimensions on the actual body of the patient. In this manner, the viewer can “see” the growth “within′ the actual body of the patient. 
     In certain aspects and embodiments, invasive surgical tools in general are utilized in lieu of or in addition to the described catheter. For example, with reference to  FIGS. 14-17  the tip  78  may indicate the leading point of a thermal imaging camera coupled to a catheter to collect temperature data through the anatomical channel it is passing through. Thermal data may then be converted from black and white thermal images and displayed through the image reception unit  106  as a variable-color overlay to an operator. It will be appreciated that not every display need common imagery to every user. A first operator or observer (local or remote) may desire certain information display, such as temperature data indicative of arterial blood vs. venous blood or frostbite tissue, whereas a second operator or observer may desire certain information such as that indicative of position of the surgical tool. In other words, display of information need not be limited to images of a position of a tool within a patient, but may be images collected from the tool as well. In some thermal imaging embodiments, a temperature gradient can be selected per user; a first operator may want to discern organic tissue from inorganic material and set a base temperature to 98.6 degrees Fahrenheit whereas a second operator may intend to track surgical tools specifically and set a base temperature for imaging to a preset temperature rather than an absolute. 
     In some embodiments, operational envelopes are implemented. Returning to  FIGS. 14-17 , while navigating anatomical channels, a patient&#39;s particular disposition may alter the operational loads of the instrument. For example, if patient imaging indicates sensitive tissue proximate the instrument, the load may be proportionally adjusted. To further illustrate, if imaging of a patient with atrial fibrillation indicates a thin left atrial wall, an ablation catheter with a standard axial load of 0.1 N may be tented to a range of 0.01 or 0.05 N while operating in proximity to the left atrial wall. In some embodiments, the loading paradigm as a function of position, both absolute and relative to an anatomical marker such as sensitive tissue, may be displayed to operators in conjunction with instrument imagery such as position or collected images. Visual feedback may therefore be provided, indicating what the load on the instrument is capped at as function of position, to inform operators of device capabilities or limitations at that instant. Alternatively, additional operators or observers can immediately identify when an instrument is not in the correct position or exceeding patient parameters for a given procedure. Though such feedback has been described as visual, feedback may take other forms such as audio or haptic (increased resistance or friction at the instrument controls). 
     In some embodiments, observable motion artifacts provide positional adjustments to the instrument. Head unit cameras  102  may image patient positional data, and provide real time positional adjustment to the instrument. In some embodiments, patient breathing rhythm is observed and instrument controls are modified to accommodate for the breathing state. Head unit cameras  102  may register positional data, such as by fiducial markers within the operating environment or fixed machines of known dimensions to compare a chest position during exhale and inhale and associate changes in dimensions. In some embodiments, as the lungs expand during an inhale and surrounding anatomy compresses in reaction, instrument motion may correspondingly slow and then resume normal operational parameters as the lungs contract during exhale, or x-y-z adjustments made to match the rise and fall of the chest cavity, such that an instrument moves in absolute terms in an anatomical channel but is stable relative to that anatomical reference. Other observed triggers may provide operational adjustments as well, and heart rate and expected contraction/dilation of blood vessels may provide positional updates to the instrument. 
     In some embodiments, the feedback control of patient activity (e.g., breathing rhythm, heartbeat) is collected at three separate times by head unit cameras  102  at times T 1  to T 2  to T 3 , and the images are analyzed at time T 4  to determine a change in patient position (due to movement of the patient, their breathing rhythm, heartbeat, etc.). An instrument operatively coupled to head unit camera  102  is given a control input at time T 0  (i.e. before or concurrent with T 1 ), and an adjustment to the control input based on the change in patient position observed over the times T 1  to T 3  is made at time T 5 . In some embodiments, the change in control input is a fraction of the measured change in patient position. 
       FIG. 19  illustrates an exemplary relationship between measured patient position data and control feedback adjustments. As depicted, a theoretical y direction change in patient position by chest expansion incident to breathing is represented by curve  1901 . Observed patient position data is collected at times T 1 , T 2 , and T 3 , corresponding to chest position y 1 , y 2 , and y 3  respectively. Preferably, three collection times are made to provide trends analysis for the target measurement. For example, the average human breathing rate is 15 breaths per minute, allocating approximately 2 seconds for any given inhale or exhale. To avoid corrections to instrument positions that are based on measured inhales but are applied during exhales, at least three measurements are taken to provide trend analysis. Given average human breathing rate, a time interval between measurements of at least 0.667 second or 1.5 Hz is preferred. Such frequency need not be the time between each of T 0  and T 5 , and it is preferable to have the associated actions at T 4  and T 5  applied as quickly as possible. 
     Returning to  FIG. 19 , the change in position y 1  to y 2  to y 3  indicates to the system during analysis at T 4  that an inhale is concluding proximate to T 3 . As such, the feedback control at T 5  may provide no adjustment, to avoid applying an “inhale correction” during an exhale, or may provide a negative y position adjustment based on the time relation of T 5  to T 3  and extrapolated y position change. For example, if the time between T 3  and T 5  is similar to the time between T 2  and T 3 , and the system recognizes from the change as between y 1  and y 2  as compared to y 2  and y 3  that the patient is experiencing a change in y direction, then the adjustment may be less than or equal to the change as measured from T 2  to T 3 . This may be depicted as logic relationship below: 
       IF( T   5   −T   3   =T   3   −T   2 )∩( y   3   −y   2 )&lt;( y   2   −y   1 ), then correction at  T   5 &lt;( y   3   −y   2 ).
 
     While certain exemplary embodiments have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative and not restrictive of the current invention, and that this invention is not restricted to the specific constructions and arrangements shown and described since modifications may occur to those ordinarily skilled in the art.