Patent Description:
This is a common problem in endovascular interventions, for example, where the interventional device comprises a catheter and a guidewire that are operated using both hands and are manipulated at access site (e.g., groin) which is often far from the treatment site (e.g., lower leg). Again, additional skilled user is needed to manipulate and hold the ultrasound probe to image the anatomy and the catheter and guidewire positioned in the anatomy. Similarly, percutaneous interventions and minimally invasive interventions, such as such as ablations and atherectomies, in which multiple devices must be controlled. Accordingly, there is a need for enabling consolidated control of interventional and imaging devices, such that a single user is able to seamlessly operate both during the interventional procedure.

<CIT> discloses a controller for controlling a robotic device configured for insertion into an anatomical structure of a subject, the controller comprising:.

According to an aspect of the present disclosure, a controller according to claim <NUM> is provided.

According to another aspect of the present disclosure, a non-transitory computer readable medium according to claim <NUM> is provided.

The example embodiments are best understood from the following detailed description when read with the accompanying drawing figures. It is emphasized that the various features are not necessarily drawn to scale. In fact, the dimensions may be arbitrarily increased or decreased for clarity of discussion. Wherever applicable and practical, like reference numerals refer to like elements.

In the following detailed description, for the purposes of explanation and not limitation, representative embodiments disclosing specific details are set forth in order to provide a thorough understanding of an embodiment according to the present teachings. Descriptions of known systems, devices, materials, methods of operation and methods of manufacture may be omitted so as to avoid obscuring the description of the representative embodiments. Nonetheless, systems, devices, materials and methods that are within the purview of one of ordinary skill in the art are within the scope of the present teachings and may be used in accordance with the representative embodiments. It is to be understood that the terminology used herein is for purposes of describing particular embodiments only and is not intended to be limiting. The defined terms are in addition to the technical and scientific meanings of the defined terms as commonly understood and accepted in the technical field of the present teachings.

It will be understood that, although the terms first, second, third, etc. may be used herein to describe various elements or components, these elements or components should not be limited by these terms. These terms are only used to distinguish one element or component from another element or component. Thus, a first element or component discussed below could be termed a second element or component without departing from the teachings of the inventive concept.

The terminology used herein is for purposes of describing particular embodiments only and is not intended to be limiting. As used in the specification and appended claims, the singular forms of terms "a," "an" and "the" are intended to include both singular and plural forms, unless the context clearly dictates otherwise. Additionally, the terms "comprises," "comprising," and/or similar terms specify the presence of stated features, elements, and/or components, but do not preclude the presence or addition of one or more other features, elements, components, and/or groups thereof.

Unless otherwise noted, when an element or component is said to be "connected to," "coupled to," or "adjacent to" another element or component, it will be understood that the element or component can be directly connected or coupled to the other element or component, or intervening elements or components may be present. That is, these and similar terms encompass cases where one or more intermediate elements or components may be employed to connect two elements or components. However, when an element or component is said to be "directly connected" to another element or component, this encompasses only cases where the two elements or components are connected to each other without any intermediate or intervening elements or components.

The present disclosure, through one or more of its various aspects, embodiments and/or specific features or sub-components, is thus intended to bring out one or more of the advantages as specifically noted below. For purposes of explanation and not limitation, example embodiments disclosing specific details are set forth in order to provide a thorough understanding of an embodiment according to the present teachings. However, other embodiments consistent with the present disclosure that depart from specific details disclosed herein remain within the scope of the appended claims. Moreover, descriptions of well-known apparatuses and methods may be omitted so as to not obscure the description of the example embodiments. Such methods and apparatuses are within the scope of the present disclosure.

The various embodiments described herein streamline control of imaging and interventional devices using an imaging interface device operable by a single user. That is, the imaging interface device enables the single user to directly control the imaging modality, while simultaneously controlling parameters of the imaging modality, or other devices in the system, such as a robotic device inside the subject during an interventional procedure by manipulating the imaging interface device. The imaging interface device reduces redundant communication, is intuitive, and minimizes or potentially eliminates the need for a supporting imaging modality technician.

Generally, a system controller is configured to control at least one control parameter of a robotic device having at least one degree of freedom (DOF) of control in response to motion of an imaging interface device (e.g., ultrasound probe) held by the user. Operation of the imaging interface device enables command of a robot controller to follow the motion of the imaging interface device, where the motion is derived from the content in the images, which are also provided through imaging interface device. Thus, the imaging interface device itself is effectively utilized as a joystick to allow the user to control robotic device and any interventional devices operable via the robotic device, such as endovascular devices, for example, that are inside the subject and being tracked in the images (e.g., ultrasound volume). So, the user may image the anatomy of interest, and when motion of the robotic device is desired (e.g., advancing a guidewire inside a vessel), the user may enter the control (joystick) mode through operation of an actuator on the imaging interface device. Alternatively, or in addition, the system controller may be configured to control at least one control parameter of the imaging modality itself, such as frequency, pulse wave, color, and power, for example, in response to motion of the imaging interface device.

With regard to controlling a robotic device, the system controller takes a note of the position of a trackable feature (e.g., distal tip) of the robotic device in the imaging volume and creates a reference plane relative to the imaging interface device (and inherently the imaging volume) that is perpendicular to a longitudinal axis of the robotic device. When the user translates the imaging interface device along the direction of the robotic device, for example, the distance of the trackable feature of the robotic device to the reference plane is changing. The system controller uses this distance as an error input to servo (e.g., using PID servo control loop) the robotic device relative to the reference plane, such that the robotic device will appear to travel with the image. The servo motion continues until a tolerance error is satisfied. When the user rotates the imaging interface device, either during translation or not, the amount and direction of the rotation is detected through direct image processing or (not according to the claimed invention) positional data of the imaging interface device, e.g., using an onboard Inertial Measurement Unit (IMU), and the system controller causes the robotic device to rotate in a corresponding fashion. The user may then leave the control mode and return to the imaging mode when the actuator on the imaging interface device is inactivated.

<FIG> is a simplified block diagram of a system for controlling imaging and a robotic device insertable into an anatomical structure of a subject, according to a representative embodiment.

Referring to <FIG>, system <NUM> is configured to control placement and operation of a robotic device <NUM> inserted in an anatomical structure <NUM> of a subject (patient) <NUM> under direction of user (clinician) <NUM> during an interventional procedure. The anatomical structure <NUM> may be a vascular structure (e.g., artery), and the robotic device <NUM> may include or hold a catheter and/or a guidewire, for example, inserted into the vascular structure. The robotic device <NUM> may include or hold various other interventional devices, such as surgical devices (e.g., loop cutters, clamps), ablation devices, imaging and sensing devices, or the like, without departing from the scope of the present teachings.

In the depicted embodiment, the system <NUM> includes an ultrasound system <NUM> and a system controller <NUM>, which includes a robot controller <NUM>, and an image processing/device tracking system <NUM>. The ultrasound system <NUM> system is configured to receive ultrasound image data of the subject <NUM> through an imaging interface device <NUM>. The imaging interface device <NUM> may be a handheld ultrasound probe, for example, being manipulated by the user <NUM> while viewing a graphical user interface (GUI) <NUM> to obtain desired views of the anatomical structure <NUM> and/or the robotic device <NUM> while inside the anatomical structure <NUM>. Examples of the imaging interface device <NUM> as an ultrasound probe are shown in <FIG>, discussed below. In various embodiments, the imaging interface device <NUM> may be implemented by any other type of compatible interfaces, such as foot pedals, wheels and knobs, for example, in addition to or in place of an ultrasound probe.

When the imaging interface device <NUM> is implemented as an ultrasound probe, it may include multiple ultrasound transducers in a transducer array configured to emit acoustic waves into the subject <NUM>, and detecting elements in a detecting array configured to receive corresponding echoes under control of the ultrasound system <NUM> to provide ultrasound image data that includes imaging of the anatomical structure as well as the robotic device <NUM> inside the anatomical structure. The ultrasound system <NUM> provides the ultrasound image data to the GUI <NUM> to be viewed by the user <NUM>.

The system controller <NUM> is operable in two selectable modes, which may be selected by the user <NUM> through the imaging interface device <NUM>, for example, during the interventional procedure. As discussed in more detail below, the two selectable modes include an imaging (first) mode and a control (second) mode. In the imaging mode, the system controller <NUM> receives from the imaging interface device <NUM> the ultrasound image data of the anatomical structure <NUM> and the robotic device <NUM> in the anatomical structure <NUM>. In the control mode, the system controller <NUM> receives movement information from imaging interface device <NUM> for controlling movement of the robotic device <NUM> in the anatomical structure <NUM>. The movement information may include translational movement of the imaging interface device <NUM> across the surface of the subject <NUM> in a direction the user wishes to guide the robotic device <NUM>, and rotational movement of the imaging interface device <NUM> on the surface of the subject <NUM> at an angle the user wishes to rotate the robotic device <NUM>. The imaging interface device <NUM> may be configured to enable easy selection of the imaging mode and the control mode by the user while manipulating the imaging interface device <NUM>, as discussed below.

The robot controller <NUM> controls movement of the robotic device <NUM> within the anatomical structure <NUM> during the interventional procedure. Movement of the robotic device <NUM> includes translational movement for advancing and retracting the robotic device <NUM> into and out of anatomical structure <NUM>, rotational movement for rotating the robotic device <NUM> within anatomical structure <NUM>, e.g., in accordance with the available degrees of freedom (DOF) of the robotic device <NUM>, articulation movement for bending or articulating the robotic device <NUM>. The robotic device <NUM> may also control movement and operation of an active component, such as a laser. The robot controller <NUM> may control movement of the robotic device <NUM> through operation of servo motors (servos) integrated with the robotic device <NUM>, as is well known in the art. Thus, the movement may be referred to as servoing the robotic device <NUM>. The robot controller <NUM> may be implemented as a proportional-integral-derivative (PID) controller, for example.

The robot controller <NUM> and the robotic device <NUM> may have autonomous capability, in which case the user <NUM> may control a subset of actuated degrees of freedom. For example, the user <NUM> may control articulation or directional movement of the robotic device <NUM> by manipulation of the imaging interface device <NUM>, while the robot controller <NUM> advances (translates) the robotic device <NUM> at a constant rate. Likewise, the user <NUM> may control the amount of advancement or retraction of the robotic device <NUM> within the anatomical structure <NUM> using translation input, while the robot controller <NUM> simultaneously centers the robotic device <NUM> in the middle of the anatomical structure <NUM> cross-section (e.g., in robotic device <NUM>, which may be magnetically actuated).

The image processing/device tracking system <NUM> receives the ultrasound image data from the ultrasound system <NUM>, identifies movements of the imaging interface device <NUM> from the ultrasound image data relative to tracked anatomical features in the image data, such as speckle, vessels, for example, identifies the image of the robotic device <NUM> from the ultrasound image data, and calculates movements of the robotic device <NUM> responsive to the movements of the imaging interface device <NUM>.

As mentioned above, the imaging interface device <NUM> is configured to operate in an imaging mode and a control mode based on selection by the user while operating the imaging interface. In the imaging mode, the imaging interface device <NUM> outputs the ultrasound image data of the anatomical structure <NUM> and the robotic device <NUM> in the anatomical structure <NUM> to the ultrasound system <NUM>. The ultrasound image data is obtained in a known manner, for example, by the user <NUM> manually manipulating pressure and direction of the imaging interface device <NUM> (e.g., ultrasound probe) on the surface of the subject <NUM> while viewing corresponding ultrasound images on the GUI <NUM> in order to capture desired views. In alternative configurations, the imaging interface device <NUM> may be manipulated automatically or partially automatically using robotic control, without departing from the scope of the present teaches. In the control mode, the imaging interface device <NUM> controls movement of the robotic device <NUM>. That is, translational movements of the imaging interface device <NUM> across the surface of the subject <NUM> cause corresponding translational movements of the robotic device <NUM>. Likewise, rotational movements of the imaging interface device <NUM> cause corresponding rotational movements of the robotic device <NUM>. With regard to the rotational movements, in particular, the imaging interface device <NUM> effectively acts as a joy stick operable by the user <NUM> when in the control mode.

Notably, the user <NUM> is able to conveniently select the imaging and control modes using the imaging interface device <NUM>, while holding the imaging interface device <NUM> during the interventional procedure. For example, when the imaging interface device <NUM> is an ultrasound probe, the imaging interface device <NUM> includes a probe handle configured to be grasped by the user <NUM> for performing both ultrasound imaging and robotic device control without the need of an additional person for performing one or the other of the functions. The imaging interface device <NUM> includes an actuator, such as a push button or slider button, configured to be operated manually by the user <NUM> while holding the imaging interface device <NUM>. Again, when the imaging interface device <NUM> is an ultrasound probe, the actuator may be integrated in the probe handle, so that the actuator is easily accessed and operated by the user while grasping the probe handle to selectively activate the imaging mode and the control mode.

In an embodiment, a virtual probe representing the imaging interface device <NUM> may be rendered on the GUI <NUM> and/or on augmented or virtual reality displays (not shown). The GUI <NUM> enables multiple simultaneous inputs, with regard to selection of the imaging mode and the control mode, as well as other functionality, and may display different scales and values. In another embodiment, the imaging interface device <NUM> may be controlled by the user telerobotically, as opposed to being handheld. In this case, the user <NUM> may select the imaging and control modes by remote control.

<FIG> show perspective views of ultrasound probes with integrated actuators for selecting imaging and control modes, according to representative embodiments.

Referring to <FIG>, ultrasound probe 215A includes a push button <NUM> as the integrated actuator for selecting the imaging mode and the control mode. The ultrasound probe 215A includes a probe handle <NUM> for the user <NUM> to manipulate the ultrasound probe 215A, and a transducer/detector array <NUM> for transmitting acoustic waves into the subject <NUM>, and receiving corresponding echoes to provide the ultrasound image data. The push button <NUM> is configured so that, when grasping the probe handle <NUM> with their hand, the user <NUM> is able to easily depress the push button <NUM> in order to switch between the imaging mode and the control mode using the same hand while maintaining the grasp on the probe handle <NUM>. The push button <NUM> may be actuated using one or more fingers, thumb or other portions of the user's hand.

In an embodiment, ultrasound probe 215A enters the imaging mode when push button <NUM> is depressed, and enters the control mode when the push button <NUM> is released, or vice versa. Alternatively, the push button <NUM> may have one position, and simply switch back and forth between the imaging mode and the control mode each time the push button <NUM> is pressed. In this configuration, the push button <NUM> may simply be a touch sensitive button instead of a physically depressible push button.

Referring to <FIG>, ultrasound probe 215B includes a slider button <NUM> on the probe handle <NUM> as the integrated actuator for selecting the imaging mode and the control mode. The slider button <NUM> is configured so that, when grasping the probe handle <NUM>, the user is able to easily slide the slider button <NUM> between an upper position UP and a lower position LP in order to switch between the imaging mode and the control mode using the same hand while maintaining the grasp on the probe handle <NUM>. In an embodiment, ultrasound probe 215A enters the imaging mode when slider button <NUM> is in the upper position UP, and enters the control mode when the slider button <NUM> is in the lower position LP, or vice versa. The slider button <NUM> may be actuated by a sliding movement or a scrolling movement, like a scroll wheel, using one or more fingers, thumb or other portions of the user's hand.

Referring to <FIG>, ultrasound probe 215C includes a combination controller <NUM> on the probe handle <NUM> as the integrated actuator for selecting the imaging mode and the control mode. In the depicted embodiment, the combination controller includes upper push button 216U and lower push button <NUM>, which function as described above with reference to the push button <NUM>, and slider button <NUM> so that, when grasping the probe handle <NUM>, the user is able to easily operating either or all of the upper and lower push buttons 216U and <NUM> and the slider button <NUM>, as described above. Having both the upper and lower push buttons 216U and <NUM> enables choosing different parameters to be controlled by the motion of the ultrasound probe 215C, independently or simultaneously. Examples include robotic device translation, robotic device rotation, ultrasound imaging parameters, treatment device actuations, such as laser power in laser atherectomy. The combination controller <NUM> may be actuated using one or more fingers, thumb or other portions of the user's hand.

<FIG> shows a perspective view of an imaging interface translated while imaging an anatomical structure, according to a representative embodiment.

Referring to <FIG>, imaging interface device <NUM> (e.g., ultrasound probe 215A) is shown imaging the anatomical structure <NUM> and robotic device <NUM> by emitting acoustic waves <NUM> into the subject <NUM> and receiving corresponding echo signals (not shown). The imaging interface device <NUM> defines an image plane B in ultrasound space that, for purposes of illustration, extends from the transducer/detector array <NUM> to include the longitudinal axis of the robotic device <NUM> and/or the longitudinal section of the anatomical structure. That is, portions of the robotic device <NUM> located inside the anatomical structure <NUM> along the longitudinal axis are in the image plane B, in both the imaging mode and the control mode of the imaging interface device <NUM>. In the case of a matrix ultrasound probe, such imaging planes can be constructed using beam forming techniques or slicing of 3D volumetric images, both known in the art. In some cases, the image plane B may be located elsewhere relative to the imaging interface device <NUM>, e.g., parallel to longitudinal axis <NUM> and extending from a line on the transducer/ detector array, and/or may be designated by the user <NUM>, without departing from the scope of the present teachings.

In addition, a virtual reference plane R is defined in ultrasound space relative to the transducer/detector array <NUM>, e.g., whenever the push button <NUM> is actuated in a manner causing the imaging interface device <NUM> to enter the control mode. The reference plane R intersects the image plane B (e.g., at about a <NUM> degree angle) at a first position of a trackable feature of the robotic device <NUM> being tracked. The reference plane R is substantially perpendicular to a longitudinal axis of the robotic device <NUM>. In the depicted example, the trackable feature of the robotic device <NUM> being tracked is the distal tip <NUM> of the robotic device <NUM>. However, the trackable feature is not limited to the distal tip <NUM>, and may be another predetermined section of the robotic device <NUM>, a marker on the robotic device <NUM>, a bend in the robotic device <NUM>, or a predetermined distance from one of the distal tip <NUM>, the other predetermined section, the marker, or the bend, for example.

The control mode is selected to translate the trackable feature of the robotic device from the first position to a second position. In the control mode, a translational movement of the imaging interface device <NUM> is detected by the image processing/device tracking system <NUM> when the imaging interface device <NUM> is translated on the surface of the subject <NUM> relative to the anatomical structure <NUM>. The direction, distance and/or velocity of the image interface translation may be supplemented by data provided by IMUs on the imaging interface device <NUM>.

The reference plane R is defined relative to the imaging interface device <NUM>, thus moving the imaging interface device <NUM> moves the reference plane R relative to the anatomy. In particular, when the imaging interface device <NUM> is translated in the x direction as shown by arrow <NUM>, the reference plane R implicitly follows the translation, moving along the image plane B in the same direction by the same distance. In an embodiment, the translation of the imaging interface device <NUM> controls force input that is applied to the robotic device <NUM> towards the reference plane R.

Therefore, to move the robotic device <NUM> by maneuvering the imaging interface device <NUM>, the user <NUM> initially depresses the push button <NUM> to enter the control mode. In the control mode, the image processing/device tracking system <NUM> locates the trackable feature (e.g., the distal tip <NUM>) of the robotic device <NUM> in the ultrasound image provided by the ultrasound system <NUM> in the imaging mode. The image processing/device tracking system <NUM> may locate the robotic device <NUM> and its distal tip <NUM>, as well as the anatomical structure <NUM>, using artificial intelligence (AI) segmentation, for example, in which case the image plane B is created from the anatomical structure <NUM>. The image processing/device tracking system <NUM> creates the image plane B extending from the imaging interface device <NUM>, and further creates the reference plane R intersecting the image plane B at a first position of the trackable feature. This first reference plane R provides a reference datum relative to the imaging interface device <NUM> based on the first position of the trackable feature in the image plane B. The reference datum may be displayed on the GUI <NUM>, along with the ultrasound image of the anatomical structure <NUM> and the robotic device <NUM>.

Then, when the user <NUM> again activates (or continues to activate) the push button <NUM> to enter the control mode, the imaging interface device <NUM> can be used, e.g., as a joystick, to control movement of the robotic device <NUM>. In <FIG>, the user <NUM> translates the imaging interface device <NUM> across the surface of the subject <NUM> in the x direction indicated by the arrow <NUM> from a first location to a new second location. In response, the information from image processing/device tracking system <NUM> is used by robot controller <NUM> to set a new image plane B on the ultrasound system <NUM> in association with the movement of the imaging interface device <NUM> to the second location. The image plane B is defined so that tip and the distal longitudinal section of the robotic device <NUM> contained in the image plane B (in case there is inadvertent motion out of original image plane B). The imaging interface device <NUM> continues to collect images in the reference plane R, which is defined relative to the imaging interface device <NUM> probe such that it images in the same location relative to the imaging interface device <NUM> regardless of image content or tracking information.

The robot controller <NUM> receives information from the imaging processing/device tracking system <NUM> identifying the position of the robotic device <NUM> in the ultrasound image. The robot controller <NUM> determines positions of the trackable feature of the robotic device <NUM> in the image plane B. The robot controller <NUM> determines a distance within the image plane B between the current trackable feature of robotic device <NUM> and the intersection between the reference plane R and the image plane B, and servos the robotic device <NUM> so that the trackable feature of the robotic device <NUM> moves toward this intersection. The robot controller <NUM> then receives additional information from the imaging processing/device tracking system <NUM> identifying the next position of the robotic device <NUM> in the ultrasound image, and repeats the subsequent steps in a loop until a threshold is met.

In an embodiment, the robot controller <NUM> uses the determined distance as a position error, and translates the robotic device <NUM> toward the location of the reference plane R by correcting for the position error. In other words, as the user <NUM> translates the imaging interface device <NUM> in the direction x, the distance of the trackable feature of the robotic device <NUM> from the first position relative to the anatomy to the reference plane R increases. So, the robot controller <NUM> uses this distance as an error input to servo the robotic device <NUM> towards the reference plane R, for example, using a PID servo control loop, until the second position associated is aligned with the reference plane R. That is, the robot controller <NUM> continues the servo motion until a tolerance error is satisfied. The imaging interface device <NUM> may continue to provide ultrasound images during the translation, in which case the movement of the robotic device <NUM> is captured and displayed on the GUI <NUM> in real time. Thus, the robotic device <NUM> moving through the anatomical structure <NUM> will appear to travel with the ultrasound image.

The user <NUM> leaves the control mode and re-enters the imaging mode when the push button <NUM> is inactivated (e.g., released or depressed again). Then, the user <NUM> may again manipulate the imaging interface device <NUM> at the second location to obtain ultrasound images of the anatomical structure <NUM> and the robotic device <NUM>, which are displayed on the GUI <NUM>. This assumes that imaging is the default mode, which recurs when the push button <NUM> is not pressed. Alternatively, the button press may act as a toggle, it which case the push button <NUM> will need to be pressed again to disengage modes.

<FIG> shows a perspective view of an imaging interface rotated while imaging an anatomical structure, according to a representative embodiment.

Referring to <FIG>, imaging interface device <NUM> (e.g., ultrasound probe 215A), for example, is shown imaging the anatomical structure <NUM> and robotic device <NUM> by emitting acoustic waves <NUM> into the subject <NUM> and receiving corresponding echo signals (not shown). With regard to rotation of the imaging interface device <NUM>, the user <NUM> initially depresses the push button <NUM>, in response to which the imaging interface device <NUM> enters the control mode. In the control mode, the image processing/device tracking system <NUM> locates the trackable feature (e.g., the distal tip <NUM>) of the robotic device <NUM> in the ultrasound image provided by the ultrasound system <NUM> in the imaging mode. The image processing/device tracking system <NUM> may locate the robotic device <NUM> and its distal tip <NUM>, as well as the anatomical structure <NUM>, using AI segmentation, for example. The image processing/device tracking system <NUM> creates the image plane B extending from the imaging interface device <NUM>, and further creates the reference plane R intersecting the longitudinal axis <NUM> and an initial angle (e.g., <NUM> degrees to plane B), where plane R is virtually attached to one or more tracked anatomical features. The orientation of the reference plane R may provide a first reference image relative to the imaging interface device <NUM> and the image plane B, which may be displayed on the GUI <NUM>.

When the user <NUM> rotates the imaging interface device <NUM> while holding down the push button <NUM>, the imaging interface device <NUM> can be used, e.g., as a joystick, to control rotation of the robotic device <NUM>, indicated by arrow <NUM>. In some embodiments, consecutive button presses of the push button <NUM> may be used for clutching in and out. In the depicted example, the objective of rotating the imaging interface device <NUM> is to control directional movement (e.g., rotation about the longitudinal axis or articulation of the distal tip <NUM>) of the robotic device <NUM>, indicated by arrow <NUM> in <FIG>, for steering the robotic device <NUM> to a branch <NUM>' of the anatomical structure <NUM>. The magnitude of the rotation or articulation of the device is proportional to the rotational deviation of the probe relative to the reference frame R.

Alternatively, in a similar control fashion as above, during a particular control mode, the user <NUM> rotates the imaging interface device <NUM> to define the orientation of the reference plane R, which is virtually anchored to anatomical features in the images. Once the control mode is exited, the robot controller <NUM> uses the distance metric from the distal tip <NUM> of the robotic device <NUM> to the reference plane R (from the image processing) to actuate the robotic device <NUM> to move (e.g., rotate or articulate) towards the reference plane R location relative to the anatomy. Meanwhile, the robot controller <NUM> receives information from the imaging processing/device tracking system <NUM> identifying the new orientation the reference plane R and the corresponding angle of rotation, either during the rotation maneuver or after the rotation maneuver is complete. The robot controller <NUM> servos the robotic device <NUM> to move (e.g., articulate) by the angle of rotation relative to the image plane B, e.g., steering the distal tip <NUM> of the robotic device <NUM> from a main portion of the anatomical structure <NUM> into the branch <NUM>'. The imaging interface device <NUM> may continue to provide ultrasound images during the translation, such that the movement of the robotic device <NUM> is captured and displayed on the GUI <NUM> along with the rotation of the ultrasound probe 215A.

The GUI <NUM> may display the basic three-dimensional volume or, in the case of two-dimensional multiple plane imaging, the GUI <NUM> may display the image plane B in a static position and show reference plane R dynamically relative to plane B, in a 3D perspectives, as shown in <FIG>. That is, since this is essentially beam steering of the image plane B, the display of the GUI <NUM> is able to stay on the target region.

The amount and direction of rotation may be detected by direct image observation by the image processing/device tracking system <NUM> with image processing, such as speckle tracking, for example. The amount and direction of rotation may also be detected or supplemented from data provided by the IMUs on the imaging interface device <NUM>. The amount and direction of rotation relative to the first orientation of the reference plane R as determined by the image processing/device tracking system <NUM> is input to the robot controller <NUM>. Other compatible rotation detection techniques may be incorporated without departing from the scope of the present teachings.

The robot controller <NUM> may also apply one or more boundary conditions linked to physical attributes of the anatomical structure <NUM>, such as vessel walls, limiting the translational and/or rotational movements. For example, the one or more boundary conditions may be defined by an anatomical model and/or pre-op images of the anatomical structure, where physical boundaries are identified either automatically using AI segmentation of an ultrasound image, for example, or manually through user <NUM> interaction the anatomical model and/or the pre-op images via the GUI <NUM>. The boundary conditions are then implemented to avoid contact between the robotic device <NUM> and the anatomical structure <NUM>. For example, the boundary conditions may cause movement of the robotic device <NUM> associated with the translation or rotation of imaging interface device <NUM> to stop when the tracking feature or other portion of the robotic device <NUM> encounters or is about to encounter an interior wall of the anatomical structure <NUM>.

During the rotation, it is desirable for the point of view of the user <NUM> to have at least part of the ultrasound image of the robotic device <NUM> to appear to remain static in relation to the original reference image in the display on the GUI <NUM>, in a three-dimensional volume or two-dimensional planes. This provides a stable visual reference for the rotation and the area of interest in the ultrasound image.

In case of a three-dimensional volume, the image processing/device tracking system <NUM> maintains a static ultrasound image orientation by changing the perspective of the three-dimensional model during the rotation, and adjusting any virtual cut-away planes.

In case of two-dimensional images (x-planes), the image acquisition by the imaging interface device <NUM> may be controlled to change with the rotation. For example, when using IMUs, the two-dimensional image plane B is virtually steered via electronic beam steering around the longitudinal axis <NUM> in the equal and opposite direction of the corresponding rotation of the imaging interface device <NUM>, as measured by the IMUs. For example, when the IMUs sense <NUM> degrees of counterclockwise rotation, the image processing/device tracking system <NUM> causes the image plane B of the imaging interface device <NUM> to rotate <NUM> degrees clockwise. This keeps the image plane B in the same orientation relative to the anatomical structure <NUM> during the rotation. Alternatively, the image plane B can be anchored to the location of the robotic device <NUM> and/or an anatomical feature in the images while the imaging interface device <NUM> is manipulated, subject to constraints such that the image plane B rotates about the longitudinal axis <NUM>, and intersects the transducer elements, and the image formation is possible using beam forming techniques.

Further in case of two-dimensional images, the imaging interface device <NUM> may be controlled to create arbitrary x-planes by selective activation and deactivation of different sets of ultrasound transducers/detecting elements in the transducer/detector array <NUM>. Accordingly, as the imaging interface device <NUM> is rotated, the ultrasound system <NUM> activates the transducers and/or detecting elements of the imaging interface device <NUM> aligned with the initial position of the image plane B, and deactivates the remaining transducers and/or detecting elements, in order to keep the image plane B in the same orientation, thereby making the image plane B appear to be static relative to the anatomical structure <NUM>.

The user <NUM> leaves the control mode and re-enters the imaging mode when the push button <NUM> is inactivated (e.g., released or depressed again). Then, the user <NUM> may again manipulate the imaging interface device <NUM> to obtain ultrasound images of the anatomical structure <NUM> and the robotic device <NUM>, which are displayed on the GUI <NUM>, from the new rotated orientation. In an embodiment, the imaging interface device <NUM> may continue to provide ultrasound images during the translation, regardless of the position of the push button <NUM>, such that the movement of the robotic device <NUM> is captured and displayed on the GUI <NUM> along with the rotation of the ultrasound probe 215A.

Accordingly, the reference plane R may be used differently in the different rotation modes. In one rotation mode, the reference plane R is defined relative to arbitrary anatomy of the subject <NUM> for the purpose of rotation tracking, rotation controls articulation, or some other robotic or imaging parameter (in a continuous manner). In this scenario, the reference plane R is just a reference, and be substituted with IMU only. In another rotation mode, rotation of the imaging interface device <NUM> is used to place the reference plane R by rotation (e.g., about the distal tip <NUM> of the robotic device <NUM>) in a continuous manner. Once placed relative to the anatomy of the subject <NUM>, the new reference plane R may be used as a reference datum (travelling with the anatomy) for a subsequent servo loop controlled by the robot controller <NUM>. For example, the robot controller <NUM> may articulate the robotic device <NUM> towards the new reference plane R. The image plane B in both rotation modes is used generally for user feedback. That is, depiction of the image plane B need not be provided, and if it is, the image plane B rotates in the opposite direction as the reference plane R. Or, the image plane B stays in the same plane relative to the anatomical structure <NUM> or robotic device <NUM> by direct tracking.

In addition to or instead of directional movement of the distal tip <NUM>, the rotation input using the imaging interface device <NUM> may be detected and used to control other parameters associated with the imaging interface device <NUM> or the robotic device <NUM>. For example, the rotation of the imaging interface device <NUM> may be used to control rotation of the robotic device <NUM> around its longitudinal axis. That is, the imaging processing/device tracking system <NUM> may be configured to interpret the rotation of the imaging interface device <NUM> around the longitudinal axis <NUM> as corresponding rotation of the robotic device <NUM> around its longitudinal axis, matching the angle (amount) and direction of rotation.

Also, the rotation of the imaging interface device <NUM> may be used to control various operations of the robotic device <NUM>. For example, when the robotic device <NUM> comprises or controls a laser device, e.g., used for ablation or atherectomy procedures, the imaging processing/device tracking system <NUM> may be configured to interpret clockwise rotation of the imaging interface device <NUM> as an increase in laser power, and counterclockwise rotation as a decrease in laser power, or vice versa. The angle of rotation (clockwise or counterclockwise) may be interpreted as being proportional to the magnitude of the corresponding increase/decrease in laser power. Or, when the imaging interface device <NUM> is not rotating, translational movement of the imaging interface device <NUM> may control the desired speed setting of the laser. When the robotic device <NUM> comprises or controls a cutting tool, the imaging processing/device tracking system <NUM> may be configured to interpret clockwise rotation of the imaging interface device <NUM> as an increase in cutting speed or mechanical oscillation, and counterclockwise rotation as a decrease in cutting speed or mechanical oscillation, or vice versa. The angle of rotation may be interpreted as being proportional to the magnitude of the corresponding increase/decrease in cutting speed or mechanical oscillation. When the robotic device <NUM> comprises or controls a balloon catheter, the imaging processing/device tracking system <NUM> may be configured to interpret clockwise rotation of the imaging interface device <NUM> as an increase in balloon inflation rate, and counterclockwise rotation as a decrease in balloon inflation rate, or vice versa.

With regard to an atherectomy procedure in particular, the position of the imaging interface device <NUM> may control the orientation of the side-facing atherectomy cutter/laser of the robotic device <NUM> inside the anatomical structure <NUM>. This may be done, for example, by positioning the imaging interface device <NUM> transversally on the leg of the subject <NUM>. The robot controller <NUM> uses this orientation of the imaging interface device <NUM> to actuate the robotic device <NUM> to rotate/position the atherectomy body or cutting direction to align with the imaging interface device <NUM> as it is moved on the surface of the leg, thus facing the cutter towards the imaging interface device <NUM>.

The rotation of the imaging interface device <NUM> may also be used to control displacement of a secondary device relative to the robotic device <NUM>. For example, the robotic device <NUM> may be a catheter and the secondary device may be a guidewire within the catheter, in which case the imaging processing/device tracking system <NUM> may be configured to interpret clockwise rotation of the imaging interface device <NUM> as extending the secondary device, and counterclockwise rotation as retracting the secondary device, or vice versa. The angle of rotation may be interpreted as being proportional to the distance of the corresponding extension/retraction. Of course, the detected rotation may be used to adjust other parameters associated with the imaging interface device <NUM> and/or the robotic device <NUM> without departing from the scope of the present teachings.

In addition, the rotation of the imaging interface device <NUM> may be used to control operations of the ultrasound system <NUM>, the imaging interface device <NUM> itself, and/or the GUI <NUM>. For example, clockwise and counterclockwise rotation may be used to adjust image acquisition settings of the ultrasound system <NUM> or the imaging interface device <NUM>, such as frequency, pulse wave, color, and power, for example. Likewise, the clockwise and counterclockwise rotation may be used to adjust display settings of the GUI <NUM>. For example, the rotation may be used to control image contrast when displayed on the GUI <NUM>. Or, for an image review using the GUI <NUM>, the rotation may control a sequential scrolling through acquired ultrasound images.

In an embodiment, external tracking of the imaging interface device <NUM> and the robotic device <NUM> may be used to provide feedback, particularly when the robotic device <NUM> is not in the field of view of the ultrasound image or ultrasound volume. For example, one or more sensors may be provided on the imaging interface device <NUM> and/or the robotic device <NUM>, so that their respective positions may be determined using electromagnetic (EM) tracking, regardless of whether the robotic device <NUM> is present in the field of view of the imaging interface device <NUM>. Other examples of external tracking include optical tracking, acoustic tracking, electromagnetic tracking, ultrasound-sensed device tracking (e.g., Onvision® Needle Tip Tracking), and IMU based tracking.

The imaging interface device <NUM> may be implemented using imaging modalities other than ultrasound, which is, however, not according to the claimed invention. For example, the imaging interface device <NUM> may provide single photon emission computed tomography (SPECT) imaging, x-ray imaging, or endoscopic/laparoscopic imaging. In this case, the ultrasound system <NUM> would be replaced by an imaging system compatible with the alternative imaging modality, as would be apparent to one skilled in the art.

<FIG> is a simplified block diagram of a system controller for controlling imaging and a robotic device insertable into an anatomical structure of a subject, according to a representative embodiment.

Referring to <FIG>, the system controller <NUM> includes software instructions that can be executed to perform any of the methods or computer-based functions disclosed herein. The system controller <NUM> may operate as a standalone device or may be connected, for example, using a network <NUM>, to other computer systems or peripheral devices.

The system controller <NUM> includes a processor <NUM> configured to execute software instructions to perform functions as described in the various embodiments herein. The processor <NUM> is representative of one or more processing devices, and is configured to execute software instructions to perform functions as described in the various embodiments herein. The processor <NUM> may be implemented by field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), a general purpose computer, a central processing unit, a computer processor, a microprocessor, a microcontroller, a microcomputer, a state machine, programmable logic device, a processor chip, a digital signal processor (DSP), a graphics processing unit (GPU), or combinations thereof, using any combination of hardware, software, firmware, hard-wired logic circuits, or combinations thereof. Additionally, any processor described herein may include multiple processors, parallel processors, or both. Multiple processors may be included in, or coupled to, a single device or multiple devices.

The term "processor" as used herein encompasses an electronic component able to execute a program or machine executable instruction. References to a computing device comprising "a processor" should be interpreted to include more than one processor or processing core, as in a multi-core processor. A processor may also refer to a collection of processors within a single computer system or distributed among multiple computer systems. The term computing device should also be interpreted to include a collection or network of computing devices each including a processor or processors. Programs have software instructions performed by one or multiple processors that may be within the same computing device or which may be distributed across multiple computing devices.

The system controller <NUM> further includes a main memory <NUM> and a static memory <NUM>, where memories communicate with each other and the processor <NUM> via a bus <NUM>. Memories described herein are tangible storage mediums for storing data and executable software instructions and are non-transitory during the time software instructions are stored therein. As used herein, the term "non-transitory" is to be interpreted not as an eternal characteristic of a state, but as a characteristic of a state that will last for a period. The term "non-transitory" specifically disavows fleeting characteristics such as characteristics of a carrier wave or signal or other forms that exist only transitorily in any place at any time.

The main memory <NUM> and the static memory <NUM> are computer-readable mediums from which data and executable software instructions can be read by a computer (e.g., the processor <NUM>). Each of the main memory <NUM> and the static memory <NUM> may be implemented as one or more of random access memory (RAM), read only memory (ROM), flash memory, electrically programmable read only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, a hard disk, a removable disk, tape, compact disk read only memory (CD-ROM), digital versatile disk (DVD), floppy disk, Blu-ray disk, or any other form of storage medium known in the art. The memories may be volatile or non-volatile, secure and/or encrypted, unsecure and/or unencrypted. The main memory <NUM> and the static memory <NUM> may store various types of information, such as software algorithms, AI models including RNN and other neural network based models, and computer programs, all of which are executable by the processor <NUM>.

"Memory" is an example of a computer-readable storage medium. Computer memory is any memory which is directly accessible to a processor. Examples of computer memory include, but are not limited to RAM memory, registers, and register files. References to "computer memory" or "memory" should be interpreted as possibly being multiple memories. The memory may for instance be multiple memories within the same computer system. The memory may also be multiple memories distributed amongst multiple computer systems or computing devices.

As shown, the system controller <NUM> further includes a display <NUM>, such as a liquid crystal display (LCD), an organic light emitting diode (OLED), a flat panel display, a solid-state display, or a cathode ray tube (CRT), for example. The GUI <NUM> may be incorporated into the display <NUM>. Additionally, the system controller <NUM> includes an input device <NUM>, such as a keyboard/virtual keyboard or touch-sensitive input screen or speech input with speech recognition, and a cursor control device <NUM>, such as a mouse or touch-sensitive input screen or pad. The system controller also includes a robot interface <NUM> configured to control movement of the robotic device <NUM> in accordance with the embodiments described herein.

<FIG> is a flow diagram of a method for controlling imaging and a robotic device insertable into an anatomical structure of a subject, according to a representative embodiment. The various steps of the flow diagram may be performed by the system controller <NUM>, discussed above, for example.

Referring to <FIG>, ultrasound image data is received from an imaging interface device (e.g., imaging interface device <NUM>) in block S611. The ultrasound image data corresponds to an ultrasound image showing an anatomical structure (e.g., anatomical structure <NUM>) in the subject and a portion of a robotic device (e.g., robotic device <NUM>) within in the anatomical structure. The imaging interface device is maneuverable for providing the ultrasound image data. For example, the imaging interface device may be an ultrasound probe that is maneuverable by a user to obtain various images of the anatomical structure and the robotic device during an interventional procedure, where the images are viewable on a display (e.g., GUI <NUM>). The user may maneuver the ultrasound probe directly by making adjustments while holding the ultrasound probe, or remotely though a robotic interface, to obtain the desired ultrasound image.

In an embodiment, the imaging interface device has a built-in actuator enabling the user to select an imaging mode and a control mode of operation. In the imaging mode, image data received from the imaging interface device is processed to provide images of the anatomical structure and the portion of the robotic device within the anatomical structure. In the control mode, information indicative of translational and/or rotational movement of the imaging interface device on the surface of the subject received from the imaging interface device is processed to control movement of robotic device within the anatomical in association with the translation and/or rotational movement of the imaging interface device. When the imaging interface device is an ultrasound probe, the actuator may comprise one or more push buttons, slider buttons and/or touch sensitive buttons conveniently located on the handle of the ultrasound probe, as discussed above. This enables the user to select between the imaging and control modes with one hand, while holding the ultrasound probe, avoiding interruption of the interventional procedure and eliminating the need for another person to separately operate the robotic device (or the imaging device).

In block S612, a movement of the imaging interface device is detected. As discussed above, the movement of the imaging interface device may be a translational movement or a rotational movement, and is detectable in the control mode of operation. Translational movement may be detected based on direction, distance and/or velocity data received from IMU(s) on the imaging interface device. Rotational movement may be detected based on position data received from the IMU(s) on the imaging interface device, or may be detected by direct image observation using image processing, such as speckle tracking processing, for example.

In block S613, at least one command is output for controlling movement of the robotic device from a first position to a second position in response to the detected movement of the imaging interface device. The at least one command may be provided to a robot controller (e.g., a PID controller), for example, which in turn controls servos to move the robotic device in association with the detected movement of the imaging interface device. Alternatively the command may be stored in a non-transitory memory, for a delayed use or additional processing (e.g. formatting, additional check for debugging purpose. ), before any use by the robot controller.

When the movement of the imaging interface device is a translational movement, the method includes defining an image plane B and a reference plane R in the ultrasound image relative to the imaging interface device. The image plane B extends from the imaging interface device into the subject, and includes a longitudinal axis of the robotic device and/or a longitudinal axis or other features of the anatomical structure. The reference plane R intersects the image plane B at the first position of the robotic device. More particularly, the robotic device includes a tracking feature, such as the distal tip of the robotic device, a bend or a marker, for example, which defines the first position, and the reference plane R intersects the image plane B at this tracking feature. The reference plane R is translated in association with the translational movement of the imaging interface device, and at least one command for controlling the robotic device is output to the robot controller based on a distance between the trackable feature of the robotic device the translated reference plane R. In an embodiment, the robotic device may be controlled based on the distance between the trackable feature of the robotic device and the translated reference plane R, where the robotic device is servoed toward the reference plane R being translated using the distance as a position error. That is, the robot controller attempts to correct for the position error by reducing the distance between the trackable feature and the reference plane R.

When the movement of the imaging interface device is a rotational movement, the method includes defining an image plane B in the ultrasound image relative to the imaging interface device, the image plane B and a reference plane R in the ultrasound image relative to the imaging interface device. The image plane B extends from the imaging interface device into the subject, and includes a longitudinal axis of the robotic device. The reference plane R is defined in the ultrasound image relative to the imaging interface device, intersecting the image plane B at the first position of the robotic device. A direction and angle of the rotational movement of the imaging interface device are detected, e.g., from direct image observation using image processing, or position data provided by IMU(s) on the imaging interface device. The reference plane R is rotated in the same direction and angle as the rotational movement of the imaging interface device. At least one command for controlling the robotic device is output to the robot controller in response to the rotated reference plane R.

With regard to the rotational movement, it is desirable to maintain the image plane B in in a constant image plane as viewed on the display. For a three-dimensional volume, the image plane B remains visually static during the rotation by changing the perspective of the three-dimensional model and adjusting any virtual cut-away planes. For two-dimensional images, the image plane B may remain visually static during the rotation by virtually steering the image plane B around a longitudinal axis of the imaging interface device in the opposite direction as the corresponding rotation, using position data measured by one or more IMUs. Also for two-dimensional images, the image plane B may remain visually static during the rotation by causing the imaging interface device to selectively activate ultrasound transducers and/or detecting elements in the imaging interface device to maintain the ultrasound image in a constant image plane.

In accordance with various embodiments of the present disclosure, the methods described herein may be implemented using a hardware computer system that executes software programs. Virtual computer system processing may implement one or more of the methods or functionalities as described herein, and a processor described herein may be used to support a virtual processing environment.

Although developing adaptable predictive analytics has been described with reference to several exemplary embodiments, it is understood that the words that have been used are words of description and illustration, rather than words of limitation. Changes may be made within the purview of the appended claims, as presently stated and as amended, without departing from the scope and spirit of interventional procedure optimization in its aspects. Although developing adaptable predictive analytics has been described with reference to particular means, materials and embodiments, developing adaptable predictive analytics is not intended to be limited to the particulars disclosed; rather developing adaptable predictive analytics extends to all functionally equivalent structures, methods, and uses such as are within the scope of the appended claims.

The illustrations are not intended to serve as a complete description of all of the elements and features of the disclosure described herein.

Claim 1:
A controller for controlling a robotic device (<NUM>) configured for insertion into an anatomical structure (<NUM>) of a subject (<NUM>), the controller comprising:
at least one processor (<NUM>); and
a non-transitory memory for storing machine executable instructions that, when executed by the at least one processor (<NUM>), cause the at least one processor (<NUM>) to:
receive ultrasound image data from an imaging interface device (<NUM>), the ultrasound image data corresponding to an ultrasound image showing the anatomical structure (<NUM>) and a portion of the robotic device (<NUM>) within in the anatomical structure (<NUM>), wherein the imaging interface device (<NUM>) is maneuverable for providing the ultrasound image data;
detect a movement, or receive an information representative of the detected movement of the imaging interface device (<NUM>) derived from content of the ultrasound image data; and
output at least one command for controlling movement of the robotic device (<NUM>) from a first position to a second position in response to the detected movement of the imaging interface device (<NUM>).