ROBOTIC ASSISTED IMAGING

An imaging self-positioning system includes a robotic actuator for manipulating an imaging tool or medical probe and a sensory component for maintaining a normal orientation above patient a treatment site. The imaging tool, typically an US probe, is grasped by an end-effector or similar actuator, and a sensory component engaged with the imaging tool senses an orientation of the tool relative to the treatment surface, and the robotic actuator disposes the imaging tool for maintaining a normal or other predetermined angular alignment with the treatment surface. The treatment surface is a patient epidermal region adjacent an imaged region for identifying anatomical features and surgical targets. A medical probe such as a biopsy needle may accompany the end-effector for movement consistent with the probe, either manually or robotically advanced towards the surgical target.

BACKGROUND

Medical imaging has vastly improved medical diagnosis and treatment fields by allowing doctors and medical technicians to visualize internal anatomical structures. Among the many imaging capabilities available, ultrasound mediums are favored for their benign signals and portability. Ultrasound (US) imaging has been widely adopted for abnormality monitoring, obstetrics, guiding interventional, and radiotherapy procedures. US is acknowledged for being cost-effective, real-time, and safe. Nonetheless, the US examination is a physically demanding procedure. Sonographers needs to press the US probe firmly onto the patient's body and fine-tune the probe's image view in an un-ergonomic way. More importantly, the examination outcomes are heavily operator-dependent. The information contained in the US images can be easily affected by factors such as scan locations on the body, the probe orientations at the scan location and the contact force at the scan location. Obtaining consistent examination outcomes requires highly skilled personnel with substantial experience.

SUMMARY

An imaging self-positioning system includes a robotic actuator for manipulating an imaging tool or medical probe and a sensory component for maintaining a normal orientation adjacent patient a treatment site. The imaging tool, typically an US probe, is grasped by an end-effector or similar actuator, and a sensory component engaged with the imaging tool senses an orientation of the tool relative to the treatment surface, and the robotic actuator disposes the imaging tool for maintaining a normal or other predetermined angular alignment with the treatment surface. The treatment surface is a patient epidermal region adjacent an imaged region for identifying anatomical features and surgical targets. A medical probe such as a biopsy needle may accompany the end-effector for movement consistent with the probe, either manually or robotically advanced towards the surgical target.

Robotic members are often sought for performing repetitive object placement tasks such as assembly and sorting of various objects or parts. Robot-assisted imaging may include a procedure using an end-effector of a robot arm or mechanical actuators to manipulate an imaging probe (for ultrasound, optics, and photoacoustic imaging) to realize teleoperative or autonomous tasks. Such a procedure employs sensing of the surface terrain (e.g., skin) and controlling both the orientation and location of the probe by grasping the probe through the end-effector, typically a claw or similar actuator.

Configurations herein are based, in part, on the observation that convention medical imaging, and in particular US imaging, is often employed by skilled sonographers for obtaining visual imaging for diagnosis and real time feedback during minimally invasive procedures using a needle or probe. Unfortunately, conventional approaches to US imaging suffer from the shortcoming that it can be problematic to manipulate an imaging probe for an accurate depiction of a surgical target, particularly during concurrent insertion of the needle or instrument. US probes, while portable, are dependent on accurate positioning at the treatment surface for rendering positional guidance. Accordingly, configurations herein substantially overcome the shortcoming of conventional US procedures by providing a self-positioning robotic apparatus for positioning and maintaining an alignment of the probe at a predetermined angle with the treatment site. Typically a normal or substantially normal orientation to the surface is sought, however an angular tilt may be beneficial to avoid anatomical structures obscuring the surgical target.

In a particular use case of a needle or instrument, insertion force is another parameter that eludes automation. Insertion progression and depth may be measured by resistance, or the force needed for insertion. However, varied densities of anatomical tissue, as well as variances due to an insertion angle, can make depth sensing based on resistive force to insertion unreliable.

In an example configuration, the imaging device performs a method for robotic positioning of a by receiving, from each plurality of sensing elements disposed in proximity to a medical instrument, a signal indicative of a distance to a treatment site of a patient. The controller computes, based on each of the signals and an offset of the sensor from the medical instrument, a distance from each of the respective sensing elements to the treatment site. The medical instrument may be an imaging probe, such that the imaging device determines, based on the computed distances, an angle of the medical instrument relative to the treatment site for optimal imaging alignment of a surgical site,

DETAILED DESCRIPTION

Conventional manual ultrasound (US) imaging is a physically demanding requiring skilled operators for accurate positioning of the imaging sensor. A Robotic Ultrasound system (RUSS) has the potential to overcome this limitation by automating and standardizing the imaging procedure. It also extends ultrasound accessibility in resource-limited environments with the shortage of human operators by enabling remote diagnosis. During imaging, maintaining the US probe in a normal orientation to the skin surface largely benefits the US image quality. However, an autonomous, real-time, low-cost method to align the probe towards the direction orthogonal to the skin treatment without pre-operative information is absent in conventional RUSS.

FIG.1is a context diagram of the self-orienting sensor device100. The device100performs a method for robotic assisted medical imaging and procedures, including engaging an imaging probe with a robotic actuator such as an end-effector grasping the probe or instrument, and moving the robotic actuator to dispose the imaging sensor at a predetermined location relative to a patient imaging location. The actuator maintains the imaging probe at the predetermined relative location even during movement of the patient so that a trajectory or scan direction remains consistent.

Referring toFIG.1, a robotic arm110has a series of jointed segments112-1. . .112-4for movement of an end-effector or actuator114engaging an imaging probe116(probe)101in proximity over a treatment surface101. A sensory ring120defines a frame positioned to encircle the probe116and has a plurality of sensors for detecting a distance to the treatment surface. The sensory ring120forms a circular frame for disposing the sensors at a known radius from a longitudinal axis of the probe116.

A controller130includes a robotic positioning circuit132and logic and an image processor134, along with a processor136and memory138for containing instructions as described further below. The method for robotic positioning of a surgical instrument or probe116includes receiving, from each plurality of sensing elements disposed in proximity to the probe116, a signal indicative of a distance to a treatment site101of a patient, and computing, based on each of the signals and an offset of the sensor from the medical instrument, a distance from each of the respective sensing elements to the treatment site. This determines a normal or off-normal position of the sensor ring, and hence the probe, with the treatment surface. Based on the computed distances, the processor136computes an angle of the probe116relative to the treatment site101.

An autonomous RUSS has been explored to address the issues with the conventional US. RUSS utilizes robot arms to manipulate the US probe. The sonographers are thereby relieved of the physical burdens. The diagnosis can be done remotely, eliminating the need for direct contact with patients. The desired probe pose (position and orientation) and the applied force can be parameterized and executed by the robot arm with high motion precision. As a result, the examination accuracy and repeatability can be secured. The probe pose can also be precisely localized, which enables 3D reconstruction of human anatomy with 2D US images.

An autonomous scan may adopt a 2-step- strategy: First, a scan trajectory formed by a series of probe poses is defined using preoperative data such as Magnetic Resonance Imaging (MRI) of the patient or a vision-based point cloud of the patient body. Second, the robot travels along the trajectory while the probe pose and applied force are continuously updated according to intraoperative inputs (e.g., force/torque sensing, real-time US images, etc.). Yet, owing to factors including involuntary patient movements during scanning, inevitable errors in scan trajectory to patient registration, and a highly-deformable skin surface, which can be difficult to be measured preoperatively. The second step is of significance to the successful acquisition of diagnostically meaningful US images. The ability to update probe positioning and orientation in real-time is preferred to enhance the efficiency and safety of the scanning process. In particular, keeping the probe to an appropriate orientation assures a good acoustic coupling between the transducer and the body. A properly oriented probe position offers a clearer visualization of pathological clues in the US images. Real-time probe orientation adjustment is challenging and remains an open problem.

Configurations herein apply two aspects: i) a compact and cost-effective active-sensing end-effector (A-SEE) device that provides real-time information on the rotation adjustment required for achieving normal positioning. Conventional approaches do not achieve simultaneous in-plane and out-of-plane probe orientation control without relying on a passive contact mechanism; ii) the A-SEE approach integrates with the RUSS for implementing a complete US imaging workflow to demonstrate the A-SEE enabled probe self-normal-positioning capability. It should be further emphasized that normal positioning, meaning probe orientation locates a longitudinal axis of the probe at a normal, or perpendicular to a plane defined by the skin surface, is an example of a preferred orientation; other angular orientations may be determined.

FIG.1defines corresponding coordinate frames of reference. Coordinate frame Fbase103corresponds to the robot base frame; Fflange104is the flange frame to attach the end-effector; Fcam105is an RGB-D camera's frame adjacent the end effector and FA-SEE106is the US probe tip frame. The probe116orientation as controlled by the robot incorporates these frames as follows.

(1) depicts a transformation from Fbaseto Fflange, denoted as Tbase-flange

(2) denotes a transformation from Fflangeto FA-SEE, denoted as Tflange-A-see

(3) is the transformation from FA-SEEto Fcam, denoted as TA-see-cam.

Operation of the controller includes the implementation details of A-SEE and its integration with a RUSS to manipulate the actuator114according to the sensor ring120. A typical use case involves preoperative probe landing pose identification and intraoperative probe self-normal-positioning with contact force adaptation. During imaging, the shared control scheme can allow teleoperative sliding of the probe along the patient body surface, as well as rotating the probe about its axis. Of course, the normal (or other angle pose) can assist in in-person procedures as well.

FIGS.2A-2Care schematic diagrams of the imaging probe and end effector in the device ofFIG.1. Referring toFIGS.1and2A, a plurality of sensing elements122-1. . .122-4(122generally) are disposed in proximity to a medical instrument such as the probe116. The sensory ring120positions the sensing elements in a predetermined orientation with a robotic actuator114when the robotic actuator engages the medical instrument. The actuator114engages or grabs the probe116, and the sensory ring120attaches either to the probe116or the actuator114to define a predetermined orientation between the probe and sensors; in other words, the sensors122move with the probe116so that accurate positioning can be determined from the sensors. A particular configuration embeds four laser distance sensors122on the sensory ring120to estimate the desired positioning towards the normal direction, where the actuator is integrated with the RUSS system which allows the probe to be automatically and dynamically kept to a normal direction during US imaging. The actuator114, and hence the probe116, them occupies a known location relative to the sensors122-1. . .122-4(122generally). Each of the sensors122then determines a signal indicative of a distance to the treatment site101of a patient.

A typical scenario deploys the probe116to have an imaging field140capturing images of a surgical target150, usually a mass or anatomical region to be biopsied or pierced, although any suitable anatomical location may be sought. This usually involves identifying an axis124of the medical instrument or probe116, such that the axis124extends towards the treatment site101, and is based on an orientation of the axis124relative to the plane of the treatment site101. The probe axis124is defined by a longitudinal axis through the center of mass of the probe116, or other axis that denotes a middle of the sensed imaging field140. In a simplest case, seeking a normal orientation of the probe116to the surface101, each of the 4 distance sensors122returns an equal value. Differing values can give an angular orientation of the probe axis124relative to the treatment surface101, as the “tilt” or angle of the sensory ring120will be reflected in the relative distance122′-1. . .122′-4(122generally).

Either a sensory probe such as the US probe116, or a surgical medical instrument such as a needle may be grasped by the actuator114. The probe axis124therefore defines an approach angle of the medical instrument to the treatment site101, where the sensors122are used to dispose the medical instrument based on a target angle defined by intersection of the axis124with the treatment site101. The robotic arm110translates the surgical instrument along the axis124, and therefore disposes the robotic actuator114based on the determined angle of the medical instrument.

FIG.2Bshows a probe116in conjunction with a needle117or other medical or surgical instrument, or elongated shaft. When the needle117is attached via a bracket118or similar fixed support, the probe116and needle117share the same frame of reference for relative movement. Referring toFIGS.1-2B, such a procedure may include identifying the surgical target150, where the surgical target150is disposed on an opposed side of the plane defining the treatment surface101, meaning beneath the patients skin, The probe axis124aligns with an axis151leading to the surgical target, disposing the medical instrument117for aligning an axis125with the treatment site101, and advancing the medical instrument along the axis aligned with the treatment site and intersecting with the probe axis124at the surgical target140.

The probe axis124need not be normal to the treatment surface101. In general, the probe116receives a location of the surgical target150in the imaging region140. The sensors122may be used to compute the angle of the medical instrument based on an intersection with the surgical target150and the probe axis124. The medical instrument117may then be projected along the computed angle for attaining the surgical target150.

InFIG.2C, an example of the sensory ring120is shown. While three points define a plane, the use of 4 sensors allows a pair of sensors to align with a sensory plane of the imaging region140, and the unaligned pair of sensors (offset)90° then provides an angular position of the imaging plane. Additional sensors could, of course, be employed. A probe plane is defined by the plurality of sensors122and the sensory ring120. The sensory ring120encircles the probe116and at a known distance from an imaging tip116′ or US sensor. Once the actuator114grasps or engages the probe116, and the sensory ring120is secured around the probe, the controller130can determine an orientation of the medical instrument to the probe plane (sensor location). It then identifies a patient plane defined by the treatment site based on the sensor122distances. This allows computing an orientation of a probe plane160relative to the patient plane162based on the computed distances.122′.

Any suitable sensing medium may be employed for the sensors122. In an example configuration, optical based sensors such as infrared (IR) are a feasible option, however other mediums such as laser, electromagnetic or capacitance can suffice given appropriate power and distance considerations.

FIGS.3A-3Bare respective plan and side views of the integrated probe and position sensor ring ofFIGS.2A-2Bintegrated in an imaging device100as inFIG.1. Referring toFIGS.1-3B, the device100engages the medical instrument (probe)116with a robotic actuator114for advancing the medial instrument. Since the probe orientation is adjusted based on the sensor readings, the normal positioning performance depends largely on the distance sensing accuracy of the sensors. The purpose of sensor calibration is to model and compensate for the distance sensing error so that the accuracy can be enhanced. First, a trial is conducted to test the accuracy of each sensor, where a planar object was placed at different distances (from 50 mm to 200 mm with 10 mm intervals measured by a ruler). The sensing errors were calculated by subtracting the sensor readings from the actual distance. The 50 to 200 mm calibration range is experimentally determined to allow 0 to 60 degrees arbitrary tilting of A-SEE on a flat surface without letting the sensor distance readings exceed this range. Distance sensing beyond this range will be rejected. The results of the sensor accuracy test are shown inFIGS.4A-4D. Referring toFIGS.4A-4D, black curves indicate that the sensing error changes at different sensing distances with a distinctive distance-to-error mapping for each sensor. A sensor error compensator (SEC) is designed in the form of a look-up table that stores the sensing error versus the sensed distance data. SEC linearly interpolates the sensing error given arbitrary sensor distance input. The process of reading the look-up table is described by f:d_>∈R4→e_>∈R4, where d_> stores the raw sensor readings; e_> stores the sensing errors to be compensated. The sensor reading with SEC applied is given by:

where dminis 50 mm, dmax is 200 mm. With SEC, the same trials were repeated. The curves inFIGS.4A-4Dshow the sensing accuracy. The mean sensing error was 11.03±1.61 mm before adding SEC and 3.19±1.97 mm after adding SEC. A two-tailed t-test (95% confidence level) hypothesizing no significant difference in the sensing accuracy with and without SEC was performed. A p-value of 9.72×10−8suggests SEC can considerably improve the sensing accuracy.

The values inFIGS.4A-4Dshow curves for the respective sensors122-1. . .122-4(sensors1-4) for distance measurement error before and after adding sensor error compensator. Having accurate distance readings from the sensors in real-time, A-SEE can be integrated with the robot to enable “spontaneous” motion that tilts the US probe towards the normal direction of the skin surface. A moving average filter is applied to the estimated distances to ensure motion smoothness. As depicted inFIGS.2A-2C, upon normal positioning of the probe116, the distance differences between sensor1, and3, sensor2, and4are supposed to be minimized. This is facilitated by simultaneously applying in-plane rotation, which generates angular velocity about the y-axis of FA-SEE(ωny), and out- of-plane rotation, which generates angular velocity about the x-axis of FA-SEE(ωnx). The angular velocities about the two axes at timestamp t are given by a PD control law:

where:
d1to d4are the filtered distances from sensor1to4, respectively; Δt is the control interval. ωnxand ωnyare limited within 0.1 rad/s. The angular velocity adjustment rate can reach to 30 Hz.

To prevent a loose contact between the probe and the skin that may cause acoustic shadows in the image, a force control strategy is necessary to stabilize the probe by pressing force at an adequate level throughout the imaging process. This control strategy is also responsible for landing the probe gently on the body for the patient's safety. A force control strategy is formulated to adapt the linear velocity along the z-axis expressed in FA-SEE. The velocity adaptation is described by a two-stage process that manages the landing and the scanning motion separately: during landing, the probe velocity will decrease asymptotically as it gets closer to the body surface; during scanning, the probe velocity is altered based on the deviation of the measured force from the desired value.

Therefore, the velocity at time stamp t is calculated as:

where w is a constant between 0 to 1 to maintain the smoothness of the velocity profile; ν is computed by:

Where d_′ is the vector of the four sensor readings after error compensation and filtering, and F−z is the robot measured force along the z-axis of FA-SEE, internally estimated from joint torque readings. It is then processed using a moving average filter; F˜is the desired contact force; Kp1, Kp2are the empirically given gains; d˜is the single threshold to differentiate the landing stage from the scanning stage, which is set to be the length from the bottom of the sensor ring to the tip116′ of the probe (120 mm, in the example use case ofFIG.3B).

The combination of the self-normal-positioning and contact force control of the probe forms an autonomous pipeline that controls 3-DoF probe motion. A shared control scheme is implemented to give manual control of the translation along the x-, y-axis, and the rotation about the z-axis in concurrence with the three automated DoFs. A 3-DoF joystick may be used as an input source, whose movements in the three axes are mapped to the probe's linear velocity along the x-, y-axis (νtx, νty), and angular velocity about the z-axis (ωtz), expressed in FA-SEE.

A configuration of the imaging device100ofFIG.1for providing 6-DoF control of the US probe is built by incorporating self-normal-positioning, contact force control, and teleoperation of the probe116. In a use case, for a preoperative step, the patient lies on the bed next to the robot with the robot at its home configuration, allowing the RGB-D camera to capture the patient body. The operator selects a region of interest in a camera view as an initial probe landing position. By leveraging the camera's depth information, the landing position in 2D image space is converted to Tcamrepresenting the 3D landing pose above the patient body relative to Fcam. The landing pose relative to Fbaseis then obtained by:

Where TA-SEEflangeand TcamA-SEEand are calibrated from a CAD model or measurements of the device100. The robot then moves the probe100to a landing pose using a velocity-based PD controller. In the intraoperative step, the probe will be gradually attached to the skin using the landing stage force control strategy. Once the probe is in contact with the body, the operator can slide the probe on the body and rotate the probe about its long axis via the joystick. Meanwhile, commanding robot joint velocities generates probe velocities in FA-SEE, such that the probe will be dynamically held in the normal direction and pressed with constant force. The desired probe velocities are formed as:

Transforming them to velocities expressed in Fbaseyields:

Where RA-SEEbase∈ SO(3) is the rotational component of TA-SEEbase∈ SE(3);ris given by:

Lastly, the joint-space velocity command _q⋅that will be sent to the robot for execution is obtained by:

where J(_q)†is the Moore-Penrose pseudo-inverse of the robot Jacobian matrix. During the scanning, the US images are streamed and displayed to the operator. The operator decides when to terminate the procedure. The robot will move back to its home configuration after completing the scanning.

FIGS.5A-5Bshow an alternative sensor configuration employing video image sensors. When US imaging is robotically enabled, as in configurations above, tagged as A-SEE. A remote operation is enabled. When integrated with a robotic manipulator, the A-SEE enables simplified operation for telesonography tasks: the sonographer operator only needs to provide translational motion commands to the probe, whereas the probe's rotational motion is automatically generated using A-SEE. This largely reduces the spatial cognitive burden for the operators and allows them to focus on the image acquisition task.

The example A-SEE device100employs single-point distance sensors to provide sparse sensing of the local contact surface. Such sparse sensing is sufficient to enable probe rotational autonomy when scanning flat, less deformable surfaces. However, dense sensing capability is needed when dealing with more complicated scan surfaces. To this end, the sparsely configured single-point distance sensors can be replaced with short-range stereo cameras (e.g., RealSense D405, Intel, USA), allowing dense RGB-D data acquisition of the probe's surroundings. In general, the plurality of sensing elements122define a set of points, such that each point of the set of points has a position and corresponding distance122′ to the treatment site101. In the configuration ofFIGS.5A and5B, the distance122′ signal is a video signal and the set of points defines a pixelated grid, such that the pixelated grid has a two dimensional representation of the position of a respective point in the set of points, i.e. similar to the 4 points of the sensors122-1. . .122-4with greater granularity. A non-tissue background can be precisely filtered out according to the RGB data, providing more accurate probe orientation control. The dense depth information can be used for the reconstruction of complex surfaces, facilitating the imaging of highly curved surfaces such as neck and limbs. In addition, the temporal aggregation of the depth information makes it possible to continuously track tissue deformation, allowing the imaging of highly deformable surfaces like the abdomen. Moreover, tracked deformation can be utilized to determine the appropriate amount of pressure to be applied on the body to receive optimal image quality without causing pain to the patient.

A conceptual graph of the dense-sensing A-SEE is shown inFIG.5. Two short-range stereo cameras522-1. . .522-2are attached to the two sides of the probe116. Merging the left and right camera views allows for the creation of a comprehensive representation of the probe region on the treatment site101, including a panoramic color image and a panoramic depth map. Additionally, a light source is mounted in between the cameras to ensure adequate lighting, hence accurate depth map generation. The stereo camera based setup is approximately of the same dimension compared to the single-point distance sensor solution, and can be easily integrated with the robot.

FIGS.6A and6Bdepict comparisons of hand/manual scan and automated images, respectively, captured as inFIGS.1-4D. To assess the diagnostic quality of the acquired images, the contrast-noise-ratio (CNR) is employed to measure the image quality of the A-SEE tele-sonography system and is then compared to the images obtained through freehand scanning.FIGS.6A and6Bshow that lung images acquired with the A-SEE tele-sonography system (CNR: 4.86±2.03) (FIG.6B) are not significantly different compared with images obtained by freehand scans (CNR: 5.20±2.58) ofFIG.6A.