Patent Description:
<CIT> discloses a system for inserting intracranial catheters. <CIT> discloses lung biopsy devices for locating and biopsying an object. <CIT> discloses an imaging probe with a US transducer array and an integrated optical imaging sub-system. <CIT> discloses an apparatus for proper transesophageal echocardiography probe positioning by using camera for ultrasound imaging. <CIT> discloses an endoscopic apparatus.

Ultrasound imaging employing an ultrasound transducer array mounted on the end of an insertion tube, and in particular transesophageal echocardiography (TEE), is an existing imaging methodology with various uses, most commonly for diagnostic purposes for cardiac patients and for providing image guidance during catheter-based cardiac interventional procedures. TEE involves an approach for cardiac ultrasound imaging in which the ultrasound probe includes a cable or tube with the ultrasound transducer located at its tip. The TEE probe is inserted into the esophagus to place the ultrasound transducers at its distal tip close to the heart.

In catheter-based structural heart interventions TEE has been widely adapted as a reliable approach to imaging the interventional catheter instrument used in treating structural heart disease. Three-dimensional (3D) trans-esophageal ultrasound (US) is used for interventional guidance in catheter-lab procedures since it offers real-time volumetric imaging that enhances visualization of cardiac anatomy, compared to two-dimensional (2D) slice visualization with B-mode ultrasound, and provides exceptional soft tissue visualization, which is missing in x-ray. For many structural heart disease (SHD) interventions (e.g., mitral valve replacement), TEE is commonly used.

Typically, a TEE probe is inserted into the esophagus by a trained sonographer (or cardiologist) and is adjusted manually towards a number of standard viewing positions such that a particular anatomy and perspective of the heart is within the field of view of the US device. Different measurements or inspections might require different field of views / perspectives of the same anatomy, in which case the probe needs to be repositioned. During interventions, the probe is often moved between view positions in order to accommodate X-Ray imaging as well as tracking the progress of the intervention as the device is maneuvered. Sometimes, the probe is moved incidentally due to physiological motion, or inadvertently due to other reasons, and must be restored to a desired view position.

TEE probes typically include cable-driven mechanical joints at the distal end of the probe that can be manually operated by knobs on a handle of the TEE probe. The distal dexterity provided by these joints, along with manually controlled rotation and insertion distance of the TEE probe, and electronic beam steering of the ultrasound imaging plane(s), provides substantial flexibility in positioning the ultrasound transducer and the imaging plane so as to acquire a desired view of the heart. However, concerns include a risk of perforating the esophagus, and difficulty in manipulating the many degrees of control to achieve a desired clinical view, and TEE operator's exposure to harmful radiation from the x-ray source during interventions.

In addition to TEE, other types of ultrasound imaging that employ a probe having a tube sized for insertion into a patient (i.e. a catheter) with an ultrasound transducer disposed at the distal end of the tube include: Intracardiac Echo (ICE) probes which are usually thinner than TEE probes and are inserted into blood vessels to move the ultrasound transducer array inside the heart; and Intravascular Ultrasound (IVUS) probes which are also thin and are inserted into blood vessels to image various anatomy from interior vantage points.

The following discloses certain improvements to overcome these problems and others.

In one aspect, an ultrasound device includes a probe including a tube sized for insertion into a patient and an ultrasound transducer disposed at a distal end of the tube. A camera is mounted at the distal end of the tube in a fixed spatial relationship to the ultrasound transducer. At least one electronic processor is programmed to: control the ultrasound transducer and the camera to acquire ultrasound images and camera images respectively while the ultrasound transducer is disposed in vivo inside the patient; and construct a keyframe representative of an in vivo position of the ultrasound transducer including at least ultrasound image features extracted from at least one of the ultrasound images acquired at the in vivo position of the ultrasound transducer and camera image features extracted from one of the camera images acquired at the in vivo position of the ultrasound transducer. The keyframe further includes one or more settings of the ultrasound transducer at the acquisition time of the ultrasound image acquired at the in vivo position of the ultrasound transducer.

One advantage resides in providing proper positioning of an ultrasound probe to acquire cardiac images at specific views.

Another advantage resides in providing an ultrasound probe with multiple image devices to acquire cardiac images.

Another advantage resides in providing an ultrasound probe that provides feedback to a user for maneuvering the ultrasound probe.

Another advantage resides in providing an ultrasound probe with less operational complexity, reducing errors and costs.

Another advantage resides in providing an ultrasound probe with servomotors that automatically maneuver the ultrasound probe through an esophagus, blood vessel, or other anatomy having a traversable lumen.

Another advantage resides in reducing x-ray exposure to an operator of an ultrasound device by actuating the ultrasound probe to be controlled remotely and/or automatically.

The disclosure may take form in various components and arrangements of components, and in various steps and arrangements of steps. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the disclosure.

The systems disclosed herein provide for keyframes. As used herein, a keyframe (and variants thereof) refer to an image signature representing a particular position of a TEE probe (or another catheter-based ultrasound probe). It is recognized herein that ultrasound images alone can be insufficient for generating reliable keyframes, because the ultrasound imaging can be intermittent and provides a relatively low-resolution image. To provide more robust keyframes, a video camera is integrated into the probe tip, attached with the ultrasound transducer or positioned closely thereto on the probe so as to move together.

In a typical workflow, the TEE probe acquires keyframes at points along the traversal of the esophagus. For example, a new keyframe may be acquired each time the image loses (due to movement and/or electronic beam steering) more than a threshold fraction of image features (e.g. in an ultrasound image or a camera image). Optionally, when the physician reaches a desired view a manual acquisition of a keyframe may be acquired and labeled with the view. Alternatively, the view may be recognized automatically based on image analysis automatically identifying defining image features, and the corresponding keyframe labeled with the view. It can also be defined by a dwell time, where there is no major motion change of the probe. If the physician then wants to return to a previous view, the servo motor is reversed to move the probe tip backwards, and the acquired images are compared with key points along the way to automatically trace and adjust (if needed) the backtracking traversal process. In other embodiments, the servo motor is moved via a sequence of servo motions corresponding to keyframe transitions that link a current view to a desired view. In some examples, visual servo supplement open loop motion from keyframe to keyframe. The keyframe motion includes (but is not limited to) physical motion of the probe and electronic beam steering. The motion can also include imaging settings (e.g., ultrasound parameters, camera settings, and so forth).

In some embodiments disclosed herein, a manual mode is implemented. In this case, the TEE probe is a manually operated probe having knobs for controlling the joints of the TEE probe, and the system provides control prompts such as "advance insertion", "retract", 'rotate', 'flex', "at view", or so forth based on the feedback obtained by comparing real-time images with previously acquired keyframes. In other embodiments, the TEE probe is partly or completely robotic, with servomotors replacing the mechanical manipulation to operate the TEE probe, such as distal joint control, insertion, rotation, etc. In this case, the system can directly control the servomotors to execute the desired TEE probe manipulations. The system can also control ultrasound imaging parameters (e.g., imaging depth, beam steering angle, gains, etc.) to recall previous settings for a given view.

In some embodiments disclosed herein, the ultrasound transducer is side-emitting while the video camera is forward looking, which is a convenient arrangement as a side-emitting ultrasound transducer is well-placed to image the heart, while the forward-looking video camera provides a vantage that is not provided by the side-emitting transducer. Of particular value, a forward-looking camera can detect an obstruction that would prevent further insertion of the TEE probe, and can visualize the appropriate action (e.g. turning of a probe joint) to avoid collision with the obstruction.

<FIG> and <FIG> illustrate one exemplary embodiment of an ultrasound device <NUM> for a medical procedure, in particular a cardiac imaging procedure. Although referred to herein as a TEE ultrasound device, the ultrasound device <NUM> may be any suitable catheter-based ultrasound device (e.g., an ultrasound device for an intracardiac echo (ICE) procedures, intravascular US procedures, among others). As shown in <FIG>, the ultrasound device <NUM> includes a probe <NUM> configured as, for example, a flexible cable or tube that serves as a catheter for insertion into a lumen of the patient (e.g., the lumen may be an esophageal lumen, or a blood vessel lumen, or so forth). The probe <NUM> can be any suitable, commercially available probe (e.g., a Philips x7-<NUM> TEE probe). As described herein, the probe <NUM> is described as being used in a TEE procedure including inserting the probe into an esophagus of a patient to acquire images of the patient's heart, it will be appreciated that the probe can be inserted into any portion of the patient to acquire images of any target tissue.

The probe <NUM> includes a tube <NUM> that is sized for insertion into a portion of a patient (e.g., an esophagus). The tube <NUM> can be flexible or rigid. In some examples, the tube <NUM> has a handle <NUM> that is disposed outside of the patient and is manipulated by the user. The tube <NUM> includes a distal end <NUM> with an ultrasound transducer <NUM> disposed thereat. The ultrasound transducer <NUM> is configured to acquire ultrasound images <NUM> of a target tissue (e.g., a heart or surround vasculature). A camera <NUM> (e.g., a video camera such as an RGB or other color camera, a monochrome camera, an infrared (IR) camera, a stereo camera, a depth camera, a spectral camera, an optical coherence tomography (OCT) camera, and so forth) is also disposed at the distal end <NUM> of the tube <NUM>. The camera <NUM> is configured to acquire camera (e.g., still and/or video) images <NUM> of the target tissue. The camera <NUM> can be any suitable, commercially available camera (such as a camera described in <NPL>).

The camera <NUM> is mounted in a fixed spatial relationship to the ultrasound transducer <NUM>. In one example embodiment, the ultrasound transducer <NUM> and the camera <NUM> are attached to each other, or, as shown in <FIG> and <FIG>, housed or otherwise secured to a common housing <NUM> located at the distal end <NUM> of the tube <NUM>. In particular, as shown in <FIG>, the ultrasound transducer <NUM> is arranged to be side-emitting, and the camera <NUM> is arranged to be forward-facing. Advantageously, this arrangement as shown in <FIG> provides side-emitting ultrasound transducer <NUM> is well-placed to image the heart, while the forward-looking video camera <NUM> provides a vantage (e.g., of the heart) that is not provided by the side-emitting transducer. In other example embodiments, the camera <NUM> can be steerable using robotic actuation, or manually using cables or other means, such as mirror lenses (not shown). In another example, the ultrasound transducer <NUM> and the camera <NUM> can be connected by some other mechanism (e.g., a flexible cable) and tracked relative to each other.

The ultrasound device <NUM> also includes an electronic controller <NUM>, which can comprise a workstation, such as an electronic processing device, a workstation computer, a smart tablet, or more generally a computer. In the non-limiting illustrative example, the electronic controller <NUM> is a Philips EPIQ class ultrasound workstation. (Note that the ultrasound workstation <NUM> and the TEE probe <NUM> are shown at different scales). The electronic controller <NUM> can control operation of the ultrasound device <NUM>, including, for example, controlling the ultrasound transducer <NUM> and/or the camera <NUM> to acquire images, along with controlling movement of the probe <NUM> through the esophagus by controlling one or more servomotors <NUM> of the ultrasound device <NUM> which are connected to drive its articulating joints (not shown) and/or to rotate and/or extend and retract the tube <NUM>. Alternatively, one or more knobs <NUM> may be provided by which the user manually operates the drive joints to maneuver the probe through the esophagus.

While <FIG> shows both knob and servomotor components <NUM>, <NUM> for illustrative purposes, typically the ultrasound probe <NUM> will be either manual (having only knobs) or robotic (having only servomotors), although hybrid manual/robotic designs are contemplated, such as a design in which the user manually extends/retracts the tube <NUM> while servomotors are provided to robotically operate the probe position and its joints.

The workstation <NUM> includes typical components, such as at least one electronic processor <NUM> (e.g., a microprocessor) including connectors <NUM> for plugging in ultrasound probes (a dashed cable is shown in <FIG> diagrammatically indicating the TEE probe <NUM> is connected with the ultrasound workstation <NUM>), at least one user input device (e.g., a mouse, a joystick, a keyboard, a trackball, and/or the like) <NUM>, and at least one display device <NUM> (e.g. an LCD display, plasma display, heads-up display, augmented reality display, cathode ray tube display, and/or so forth). The illustrative ultrasound workstation <NUM> includes two display devices <NUM>: a larger upper display device on which ultrasound images are displayed, and a smaller lower display device on which a graphical user interface (GUI) <NUM> for controlling the workstation <NUM> is displayed. In some embodiments, the display device <NUM> can be a separate component from the workstation <NUM>. The display device <NUM> may also comprise two or more display devices. In some embodiments, the user input device <NUM> can be a separate component from the workstation <NUM>, and in some cases can be virtual, provided inside an augmented reality system.

The electronic processor <NUM> is operatively connected with a one or more non-transitory storage media <NUM>. The non-transitory storage media <NUM> may, by way of non-limiting illustrative example, include one or more of a magnetic disk, RAID, or other magnetic storage medium; a solid-state drive, flash drive, electronically erasable read-only memory (EEROM) or other electronic memory; an optical disk or other optical storage; various combinations thereof; or so forth; and may be for example a network storage, an internal hard drive of the workstation <NUM>, various combinations thereof, or so forth. While shown separately from the controller <NUM>, in some embodiments, a portion or all of the one or more non-transitory storage media <NUM> may be integral with the ultrasound workstation <NUM>, for example comprising an internal hard disk drive or solid-state drive. It is to be further understood that any reference to a non-transitory medium or media <NUM> herein is to be broadly construed as encompassing a single medium or multiple media of the same or different types. Likewise, the electronic processor <NUM> may be embodied as a single electronic processor or as two or more electronic processors. The non-transitory storage media <NUM> stores instructions executable by the at least one electronic processor <NUM>.

The ultrasound device <NUM> is configured as described above to perform a control process <NUM> for controlling movement of the probe <NUM>. The non-transitory storage medium <NUM> stores instructions which are readable and executable by the at least one electronic processor <NUM> of the workstation <NUM> to perform disclosed operations including performing the control process <NUM>. In some examples, the control process <NUM> may be performed at least in part by cloud processing.

Referring now to <FIG>, and with continuing reference to <FIG> and <FIG>, an illustrative embodiment of the control process <NUM> is diagrammatically shown as a flowchart. At an operation <NUM>, the at least one electronic processor <NUM> is programmed to control the ultrasound transducer <NUM> and the camera <NUM> to acquire ultrasound images <NUM> and camera images <NUM> respectively while the ultrasound transducer (and also the camera <NUM> and the common rigid housing <NUM>) is disposed in vivo inside the esophagus of the patient.

At an operation <NUM>, the at least one electronic processor <NUM> is programmed to construct a keyframe <NUM> representative of an in vivo position of the ultrasound transducer <NUM> (e.g., within the esophagus). To construct the keyframe <NUM>, the at least one electronic processor <NUM> is programmed to extract ultrasound image features <NUM> from at least one of the ultrasound images <NUM>, and/or extract camera image features <NUM> from at least one of the camera images <NUM>. In another example, the keyframes <NUM> can include the ultrasound images <NUM> and/or the camera images <NUM> themselves. The ultrasound images <NUM> and the camera images <NUM> can be stored in the one or more non-transitory computer media <NUM>, and/or displayed on the display device <NUM>. The extraction process can include an algorithm to extract feature sets between the at least one ultrasound image feature <NUM> and the at least one camera image feature <NUM>. Such algorithms can include, for example, a scale-invariant feature transform (SIFT) algorithm, a multi-scale-oriented patches (MOPS), algorithm, a vessel tracking algorithm, or any other suitable matching algorithm known in the art. In a variant embodiment, the operation <NUM> acquires only ultrasound images using the ultrasound transducer <NUM> (in which case the camera <NUM> may optionally be omitted), and the operation <NUM> constructs the keyframe using features <NUM> extracted only from the ultrasound images. However, it is expected that constructing the keyframe <NUM> using features extracted from both the ultrasound image <NUM> and the camera image <NUM> will provide the keyframe <NUM> with a higher level of discriminativeness for uniquely identifying a given view, and moreover the camera image <NUM> can be useful in situations in which the ultrasound image has low contrast or otherwise has information-deficient features (and vice versa, if the camera image is information-deficient then this is compensated by the features extracted from the ultrasound image).

The keyframe <NUM> further includes features comprising one or more settings (e.g., beam steering, depth, resolution, width, and so forth) of the ultrasound transducer <NUM> at the acquisition time of the ultrasound image <NUM> from which the image feature <NUM> is extracted at the in vivo position of the transducer. In another example, the keyframe <NUM> can include rotation settings and/or insertion settings of the probe <NUM> and/or joint position settings of the probe at the acquisition time of one or more of the ultrasound images <NUM> and/or the camera images <NUM>. The joint position settings can include, for example, settings such as "insert," rotate," "flex" and so forth. These settings can be determined from positional feedback devices (not shown), such as encoders) and/or sensor feedback devices (not shown) such as force sensors or torque sensors, related to the shape and location of the probe <NUM>.

In some example embodiments, the operation <NUM> includes constructing a keyframe <NUM> responsive to satisfaction of one or more keyframe acquisition criteria <NUM> (which can be stored in the one or more non-transitory computer readable media <NUM>). In one example, the keyframe acquisition criterion <NUM> can include a comparison between a "last-acquired" keyframe <NUM> and currently acquired ultrasound images <NUM> and/or currently acquired camera images <NUM>. In another example, the keyframe acquisition criterion <NUM> can include a comparison between a keyframe <NUM> acquired at a preset previous amount of time (e.g., a keyframe acquired, for example, <NUM> seconds previously) and currently acquired ultrasound images <NUM> and/or currently acquired camera images <NUM>. The keyframes <NUM> can be stored in the one or more non-transitory computer media <NUM>, and/or displayed on the display device <NUM>. Once stored, the keyframes <NUM> can be access at any time by the user via the workstation <NUM>. The comparison can include a comparison of a change in a number of features between the last-acquired keyframe <NUM> and the ultrasound images <NUM>/camera images <NUM>, a spatial shift of one of the ultrasound images <NUM> or one of the camera images, with the last-acquired keyframe, and so forth. In another example, the keyframe acquisition criterion <NUM> can include a recognition of a defining image feature of a target tissue imaged in a current ultrasound image <NUM> (e.g., the left or right ventricle, the left or right aorta, a specific blood vessel of a heart of the patient, such as the aorta or vena cava, and so forth). The comparison process can include a matching algorithm to match the feature sets <NUM> and <NUM> of the at least one ultrasound image <NUM> and the at least one camera image <NUM>, respectively. Such algorithms can include, for example, using a sum of squared differences (SSD) algorithm. In some examples, a deformable registration algorithm known in the art that uses the feature sets <NUM> and <NUM> to improve matching between multiple keyframes <NUM>. To increase the robustness of the keyframe matching, a sequence of the most recently generated keyframes <NUM> are used in the matching process.

In an optional operation <NUM>, the at least one electronic processor <NUM> is programmed to label, with a label <NUM>, a keyframe <NUM> representative of the in vivo position of the ultrasound transducer <NUM> upon receiving a user input from a user via the at least one user input device <NUM> of the workstation <NUM>. In one approach, the GUI <NUM> may provide a drop-down list GUI dialog of standard anatomical views (a midesophageal (ME) four chamber view, a ME (long axis (LAX) view, a transgastric (TG) Midpapillary short axis (SAX) view, among others) and the user can select one of the listed items as the label <NUM>. Alternatively, a free-form text entry GUI dialog may be provided via which the user types in the label <NUM>, or further annotates a label selected from a drop-down list. In addition, keyframes <NUM> can also be labeled as being indicative or representative of intermediate positions of the ultrasound transducer <NUM> (e.g., a position of the ultrasound transducer in a position between positions shown in "adjacent" ultrasound images <NUM> and/or camera images <NUM>). In another example, a 2D or 3D visual representation of canonical views of the probe <NUM> can be shown, in which a current state of the probe is shown, while other, previously acquired views of the probe can be selected for display by the user. The labels <NUM> and the labeled keyframes <NUM> can be stored in the one or more non-transitory computer readable media <NUM>.

In some examples, rather than (or in addition to) employing manual labeling, the at least one electronic processor <NUM> can be programmed to label or otherwise classify the ultrasound images <NUM> and/or the camera images <NUM> according to particular anatomical views shown in the images (e.g., ME four chamber view, ME LAX view, TG Midpapillary SAX view, among others). The images <NUM> and <NUM> can be manually labelled by the user via the at least one user input device <NUM>, or automatically labelled using ultrasound image matching algorithms known in the art.

Referring briefly now to <FIG>, and with continuing reference to <FIG>, the probe <NUM> is manipulatable (manually using knobs <NUM> or other manual manipulation, and/or robotically using servomotors <NUM>, depending upon the embodiment) in a variety of manners. The probe <NUM> is able to advance along an "insertion" direction (i.e., into the esophagus) (labeled along an axis <NUM>(a) in <FIG>); and a "retraction" direction along an axis <NUM>(b); rotate/tum along a forward angle direction along an axis <NUM>(a), and rotate/turn along a back-angle direction along an axis <NUM>(b). The distal end <NUM> of the probe <NUM> is configured to move (via user operation of the knobs <NUM>) in an anterior/posterior flexion direction along an axis <NUM>(a) and <NUM>(b); a right/left direction along an axis <NUM>(a) <NUM>(b). The probe <NUM> can be moved along the axes <NUM>(a), <NUM>(b), <NUM>(a), <NUM>(b) by direct manipulation of the probe by a user, while movement along the axes <NUM>(a), <NUM>(b), <NUM>(a), <NUM>(b) using the knobs <NUM>. These are illustrative degrees of freedom; a specific ultrasound probe implementation may provide more, fewer, and/or different degrees of freedom for manipulating the probe position in vivo, while subset of these degrees of freedom can be actuated manual, while others could be passively implemented.

Returning to <FIG>, in another optional operation <NUM>, the at least one electronic processor <NUM> is programmed to guide (and, in the case of robotic embodiments, control) movement of the probe <NUM> through the esophagus via the construction of multiple keyframes <NUM>. To do so, the at least one electronic processor <NUM> is programmed to construct a keyframe <NUM> that is representative of a first view consisting of a first in vivo position of the ultrasound transducer <NUM>. During traversal of the ultrasound transducer <NUM> from the first view to a second view consisting of a second in vivo position of the ultrasound transducer, the at least one electronic processor <NUM> is programmed to construct keyframes <NUM> representative of "intermediate" positions of the ultrasound transducer (e.g., positions between the first and second views). In one example, these intermediate positions are implicitly included in the ultrasound and camera images <NUM>, <NUM>, in which case the probe <NUM> can be moved to match a target keyframe <NUM>. In another example, the intermediate position can be an estimate of motion from one keyframe <NUM> to the next relative to an ultrasound volume, a camera image space, or a kinematic transformation in Cartesian or joint space, or a combination of any of these. At the end of the traversal of the ultrasound transducer <NUM>, the at least one electronic processor <NUM> is programmed to construct a second keyframe <NUM> representative of the second view.

The operation <NUM> can include an operation in which the at least one electronic processor <NUM> is programmed to detect when a new keyframe <NUM> representative of the "intermediate positions" should be acquired and saved (i.e., during the transition from the first view to the second view). To do so, the most recently constructed keyframe <NUM> is compared to the most recently acquired ultrasound images <NUM> and the most recently acquired camera images <NUM>. In one example if the number of features (e.g., anatomical features, and so forth) in the images <NUM>, <NUM> changes in a way that exceeds a predetermined comparison threshold (<NUM>% of the features) as to the number of features in the keyframe <NUM>, a new keyframe is generated. In another example, the average pixel displacement in the acquired images <NUM>, <NUM> changes by a predetermined comparison threshold (e.g., x% of the image size) relative to the pixel displacement of the keyframe <NUM>, then a new keyframe is generated. Other examples can include deformable matching algorithms known in the art to improve the images <NUM>, <NUM> to image tracking. These thresholds can be empirically tuned.

In one example embodiment, the operation <NUM> is implemented in a manual mode. To do so, the at least one electronic processor <NUM> is programmed to provide human-perceptible guidance <NUM> during a manually executed (e.g. via knobs <NUM>) backtracking traversal (i.e., "reverse" movement) of the ultrasound transducer <NUM> back from the second view to the first view (or to an intermediate view therebetween). The guidance <NUM> is based on comparisons of the ultrasound images <NUM> and the camera images <NUM> (acquired during backtracking traversal) with the keyframes <NUM> representative of the intermediate positions and the keyframe representative of the first view. The guidance <NUM> can include commands including one or more of: advancement of the ultrasound device <NUM> through the esophagus (e.g., "go forward and variants thereof); retraction of the ultrasound device through the esophagus (e.g., "reverse" and variants thereof), "turn," "capture a keyframe", and so forth. The guidance <NUM> can be output visually on the display device <NUM>, audibly via a loudspeaker (not shown), and so forth. In addition, the guidance <NUM> can be displayed as overlaying the images <NUM> and <NUM> as displayed on the display device <NUM>.

In another example embodiment, the operation <NUM> is implemented in an automated mode, in which the probe <NUM> is automatically moved through the esophagus by action of servomotors <NUM>. To do so, the at least one electronic processor <NUM> is programmed to control the one or more servomotors <NUM> of the probe <NUM> to perform the traversal of the ultrasound transducer <NUM> from the first view to the second view. The at least one electronic processor <NUM> is then programmed to control the servomotors <NUM> of the probe <NUM> to perform a backtracking traversal of the ultrasound transducer <NUM> back from the second view to the first view based on comparisons of the ultrasound images <NUM> and the camera images <NUM> (acquired during the backtracking traversal) with the keyframes <NUM> representative of the intermediate positions, and the keyframe representative of the first view.

In both the manual mode and the automated mode, the at least one electronic processor <NUM> is programmed to guide the user in regard to the movement of the probe <NUM> through the esophagus by generating the GUI <NUM> for display on the display device <NUM>. The user can use the GUI <NUM> to select a desired view or keyframe <NUM> using the at least one user input device <NUM>. The desired view of keyframe <NUM> can include a keyframe that was previously acquired and stored in the non-transitory computer readable medium <NUM>, keyframes acquired during a current procedure, or predefined keyframes stored in the non-transitory computer readable medium. The matching algorithm for the image feature sets <NUM>, <NUM> can be used to find a set of keyframes <NUM> that is closest to a current acquired keyframe as shown on the display device <NUM>. For example, keyframes <NUM> from "view A" to "view N" are created by a user at the beginning of a procedure and saved in the non-transitory computer readable media <NUM>. The views between adjacent views (e.g., "view A" to view "B", "view B" to "view C", and so forth) are linked using the "intermediate" keyframes <NUM>. To do so, incremental motion between a current keyframe (e.g., "view B") and a next keyframe (e.g., "view C") using, for example, motion estimation such as a basic optical flow of features, to estimate which way the probe <NUM> should move. The incremental motion direction that is required to move the probe <NUM> to the next keyframe to a desired view is implemented on the GUI <NUM>. The incremental motion can be presented relative to, for example, a view of the camera <NUM>, a view of the ultrasound transducer <NUM>, a model of the probe <NUM>, a model of the heart, a model of the patient, and so forth. The incremental motion can be shown, for example as a three-dimensional area indicated the direction of movement.

In some embodiments, once the desired final view is close, a beam forming-based image steering process can be used to obtain the desired final view.

In other embodiments, the knobs <NUM> can be used to alter the flexion and extension settings of the probe to a correlated data set of image features <NUM>, <NUM> to improve the matching of keyframes <NUM> and the generated guidance <NUM>. The knobs <NUM> can be manipulated (i.e., turned or switched), and the flexion and extension settings can be included in the GUI <NUM>.

In further embodiments, the electronic processor <NUM> can robotically control the ultrasound probe <NUM> using the servomotors <NUM>. By adding the keyframe <NUM> tracking and guidance <NUM> as feedback to the robotic control, a Cartesian velocity control loop can be used to smoothly and reliably move the probe <NUM> to the desired views. This allows for efficient, precise, and safe automatic positioning of the probe <NUM> between different views. In some embodiments, a forward kinematics pose of the robot can be used as another feature in the generation of the keyframes <NUM>. In case of poor ultrasound image or camera image keyframe <NUM> matching, the prestored kinematic pose information can be used for transitioning to next keyframe.

In some embodiments, a data driven approach could be used for estimation of the current keyframe <NUM> relative to an atlas of keyframes (not shown) from many patients that is stored in the non-transitory computer readable media <NUM>. This would enable guidance in patients without first seeing the desired views and building of the keyframes <NUM>.

In other embodiments, a detection of adverse event can be shown on the GUI <NUM> such as, for example, when the sequence of images <NUM>, <NUM> shows an image that is close to an expected camera image in a known sequence but contains new information, such as blood, or discoloration, or new anatomical topographical features like tear.

In some embodiments, an estimated motion of the probe <NUM> can be used for special volume stitching of multiple ultrasound images <NUM>. In case of 3D data, it can provide excellent initialization information.

In further embodiments, a 3D model (not shown) can be created from poses of the probe <NUM> from the keyframe <NUM> locations with super-imposed ultrasound images <NUM>. The user can select a particular keyframe <NUM> on the 3D model as a desired view, or for more information to review stored images <NUM> associated with that area.

In some embodiments, the probe <NUM> can include an integrated force sensing mechanism (not shown) that can be used for a variety of operations, including: recalling a desired view that involves pressure on the esophagus; maintaining contact and drift, limiting deformation on the esophagus to manage perforation risk; elastography, creating a 3D heat map of force loading regions of the esophagus or stomach.

<FIG> shows an example flow chart of the movement operation <NUM>. At an operation <NUM>, a view labeled as "view A" of the probe <NUM> is acquired. At an operation <NUM>, the positions of the ultrasound transducer <NUM>, the camera <NUM>, and any joint positions (not shown) of the probe <NUM> at View A are saved. At an operation <NUM>, the probe <NUM> is moved from view A towards a new, desired view labeled as "View B. " At an operation <NUM>, the positions of the ultrasound transducer <NUM>, the camera <NUM>, and any joint positions (not shown) of the probe <NUM> as it moves from View A to View B. At an operation <NUM>, intervening keyframes <NUM> having a change of <NUM>% between View A and View B are created. At an operation <NUM>, the probe <NUM> is moved until it reaches View B. At an operation <NUM>, the probe <NUM> is moved from View B back towards View A using a visual servo on the GUI <NUM> in a reverse sequence of the intervening keyframes <NUM>. At an operation <NUM>, the probe <NUM> is move so as to traverse predefined intermediate views shown in the intervening keyframes <NUM>. At an operation <NUM>, when the probe <NUM> is near View A, the probe <NUM> is switched to an ultrasound-based servoing process. At an operation <NUM>, the probe <NUM> is moved until it reaches View A. In addition to traversing from view B to A (e.g., operations <NUM>, <NUM>, and <NUM>), keyframes <NUM> can be updated or added to the set describing the traversal from A to B and vice-versa. These can be used to enrich the resolution and richness of the keyframes <NUM>.

<FIG> shows an example use of the ultrasound device <NUM> inserted in vivo into a patient's esophagus. As shown in <FIG>, the probe <NUM> is inserted down the esophagus of the patient so that the ultrasound transducer <NUM> and the camera <NUM> can acquire the respective ultrasound images <NUM> and the camera images <NUM> of the patient's heart. It will be appreciated that this is merely one specific application of the disclosed approaches for guiding a catheter-based ultrasound probe. For example, an Intracardiac Echo (ICE) or Intravascular Ultrasound (IVUS) probe can be analogously guided through a major blood vessel(s) of the patient to reach desired anatomical views, and to backtrack to a previous anatomical view.

Claim 1:
An ultrasound device (<NUM>), comprising:
a probe (<NUM>) including a tube (<NUM>) sized for insertion into a patient and an ultrasound transducer (<NUM>) disposed at a distal end (<NUM>) of the tube;
a camera (<NUM>) mounted at the distal end of the tube in a fixed spatial relationship to the ultrasound transducer; and
at least one electronic processor (<NUM>) programmed to:
control the ultrasound transducer and the camera to acquire ultrasound images (<NUM>) and camera images (<NUM>) respectively while the ultrasound transducer is disposed in vivo inside the patient; and
construct a keyframe (<NUM>) representative of an in vivo position of the ultrasound transducer including at least ultrasound image features (<NUM>) extracted from at least one of the ultrasound images acquired at the in vivo position of the ultrasound transducer and camera image features (<NUM>) extracted from one of the camera images acquired at the in vivo position of the ultrasound transducer, characterized in that the keyframe (<NUM>) further includes one or more settings of the ultrasound transducer (<NUM>) at the acquisition time of the ultrasound image (<NUM>) acquired at the in vivo position of the ultrasound transducer.