Patent Description:
Substantial progress has been made towards increasing the effectiveness of medical treatment while reducing trauma and risks to the patient. Many procedures that once required open surgery now may be done with less invasive techniques, thus providing for less recovery time and risks of infection for the patient. Certain procedures requiring biopsy, electrostimulation, tissue ablation, or removal of native or foreign bodies may be performed through minimally-invasive surgery.

In the field of urology, for example, renal calculi or kidney stones can accumulate in the urinary tract and become lodged in the kidney. Kidney stones are deposits of materials from the urine, typically minerals and acid salts. While smaller stones may pass from the body naturally, larger stones can require surgical intervention for removal. While open surgery was once the standard treatment for the removal of stones, other less invasive techniques, such as ureteroscopy and percutaneous nephrolithotomy/nephrolithotripsy (PCNL), have emerged as safer, effective alternatives. Additionally, advances in imaging technology have improved a medical professional's ability to identify and locate stones before and during procedures. Nevertheless, medical professionals still must analyze images to determine the location and size of stones and whether any stones are present. Moreover, the images are often obstructed, blurry, and/or otherwise difficult to evaluate, making the medical professional's task of discerning the presence or size of any stones challenging.

The systems, devices, and methods of the current disclosure may rectify some of the deficiencies described above, and/or address other aspects of the prior art.

Document <CIT> discloses a traffic signal lamp image processing method and a traffic signal lamp image processing device. The color and shape of an abnormal traffic signal lamp are recovered, and false detection of a red lamp and a yellow lamp are avoided. The accuracy of traffic signal lamp image processing is improved. The traffic signal lamp image processing method provided by the invention comprises the steps that the status of a current traffic signal lamp is determined, wherein the status of the traffic signal lamp comprises a red lamp status, a green lamp status and a yellow lamp status; the long exposure frame image and the short exposure frame image of the status of the current traffic signal lamp are collected; according to the long exposure frame image and the short exposure frame image of the status of the current traffic signal lamp, the shape of the current traffic signal lamp is recovered to acquire the shape recovery image of the traffic signal lamp; and color recovering is carried out on the shape recovery image of the traffic signal lamp.

Document <CIT> describes a vision-based vehicle detection, tracking and early warning method. The method comprises the steps of collecting the image and calibrating the road vanishing line; defining and graying the vehicle detection area, and then performing gray stretching on the vehicle detection area image according to the illumination intensity classification of the collected image; constructing the training sample images and labeling the images as positive sample images and negative sample images manually; extracting the haar feature and LBP feature of the positive sample images and negative sample images to train an Adaboost cascade classifier; dividing the vehicle detection area into different domains, and detecting the vehicle by the Adaboost cascade classifier after training, and judging the vehicle twice according to the illumination intensity. When the vehicle is detected, the KCF target tracking method is used to track the vehicle. The vehicle is tracked, and the distance between the vehicle and the vehicle is calculated by the distance estimation method based on the position, and the collision time is calculated according to the vehicle speed and the distance between the vehicle and the vehicle to give the early warning.

Document <CIT> describes an apparatus that is configured to deliver destructive energy to a stone. The apparatus includes a detector operative to obtain image data from the stone, a generator operating according to one or more producing parameters for producing the destructive energy, and a video processor unit receiving the image data from the detector. The video processor unit is operative to analyze the image data to determine a displacement of the stone relative to a previous location of the stone. A controller linked to the video processor unit and to the generator is operative to vary the one or more producing parameters of the generator responsively to the displacement of the stone.

The invention is set forth in the appended set of claims. Examples of the present disclosure relate to, among other things, medical systems and non-claimed methods. Each of the examples disclosed herein may include one or more of the features described in connection with any of the other disclosed examples.

In one example, the present disclosure includes a method not forming part of the invention as claimed for processing electronic images from a medical device comprising receiving an image frame from the medical device, and determining a first color channel and a second color channel in the image frame. A location of an electromagnetic beam halo may be identified by comparing the first color channel and second color channel. Edges of an electromagnetic beam may be determined based on the electromagnetic beam halo, and size metrics of the electromagnetic beam may be determined based on the edges of the electromagnetic beam. A visual indicator on the image frame may be displayed based on the size metrics of the electromagnetic beam.

The present invention provides a system for processing electronic images from a medical device, the system comprising at least one data storage device storing instructions for processing electronic images, and at least one processor configured to execute the instructions to perform operations for processing electronic images. The operations comprise processing electronic images from a medical device comprising, inter alia, receiving an image frame from the medical device, and determining a first color channel and a second color channel in the image frame. A location of an electromagnetic beam halo is identified by comparing the first color channel and second color channel. Edges of an electromagnetic beam are determined based on the electromagnetic beam halo, and size metrics of the electromagnetic beam are determined based on the edges of the electromagnetic beam. A visual indicator on the image frame may be displayed based on the size metrics of the electromagnetic beam.

In another example, the present invention includes a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform operations for processing electronic images from a medical device. The operations comprise processing electronic images from a medical device comprising, inter alia, receiving an image frame from the medical device, and determining a first color channel and a second color channel in the image frame. A location of an electromagnetic beam halo is identified by comparing the first color channel and second color channel. Edges of an electromagnetic beam is determined based on the electromagnetic beam halo, and size metrics of the electromagnetic beam is determined based on the edges of the electromagnetic beam. A visual indicator on the image frame is displayed based on the size metrics of the electromagnetic beam.

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various exemplary embodiments and together with the description, serve to explain the principles of the disclosure.

Examples of the present disclosure include systems and methods not forming part of the invention as claimed to facilitate, and improve the efficiency and safety of minimally-invasive surgeries. For example, aspects of the present disclosure may provide a user (e.g., a physician, medical technician, or other medical service provider) with the ability to more easily identify, size, and, thus, remove kidney stones or other material from a patient's kidney or other organ. In some embodiments, for example, the present disclosure may be used in planning and/or performing a flexible ureteroscope procedure, with or without laser lithotripsy. Techniques discussed herein may also be applicable in other medical techniques, such as any medical technique utilizing an endoscope.

Reference will now be made in detail to examples of the present disclosure described above and illustrated in the accompanying drawings.

The terms "proximal" and "distal" are used herein to refer to the relative positions of the components of an exemplary medical device or insertion device. When used herein, "proximal" refers to a position relatively closer to the exterior of the body or closer to an operator using the medical device or insertion device. In contrast, "distal" refers to a position relatively further away from the operator using the medical device or insertion device, or closer to the interior of the body.

Both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the features, as claimed. As used herein, the terms "comprises," "comprising," or other variations thereof, are intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements, but may include other elements not expressly listed or inherent to such a process, method, article, or apparatus. Additionally, the term "exemplary" is used herein in the sense of "example," rather than "ideal. " As used herein, the terms "about," "substantially," and "approximately," indicate a range of values within +/- <NUM>% of a stated value.

<FIG> illustrates a medical system <NUM> that includes a medical device such as an endoscope or other medical imaging device/medical device <NUM>, a network <NUM>, user device(s) <NUM> that may include display(s) <NUM> that may be viewed by a user/practitioner/physician/patient <NUM>, and server(s) <NUM> that may comprise a frame processor <NUM> that may execute techniques discussed herein. The endoscope <NUM>, user device(s) <NUM>, and/or server <NUM> may be wire connected (as shown), wirelessly connected, or otherwise communicatively coupled. Alternatively, functionality of the server <NUM> may be performed on endoscope <NUM>, user device <NUM>, etc. The server <NUM>, endoscope <NUM>, and/or user device <NUM> may further comprise a single electronic device.

As shown in <FIG>, endoscope <NUM> may be an insertion device such as, for example, a ureteroscope (e.g., LithoVue™ Digital Flexible Ureteroscope by Boston Scientific Corp. With endoscope <NUM> positioned within a patient, for example, through the patient's urethra to a patient's kidney, a retrieval device (not shown) may be inserted to retrieve and remove material such as, for example, a kidney stone, with or without using laser lithotripsy. The endoscope <NUM> may record and/or transmit image and/or video data when inserted into a patient, and may have a light or other imaging source that may act to display images of the interior of a patient's vessels, organs, etc. A fiber optic cable or other light source may illuminate the interior of the patient. The endoscope <NUM> is equipped with or receive a laser that projects at a lower power setting, such that it acts as an aiming beam. The aiming beam may act to inform the user of the endoscope where the laser is aiming, without illuminating at a high enough intensity to destroy tissue or the kidney stone. The laser may also, per signal from the user, emit electromagnetic waves at a higher intensity for performance of laser lithotripsy, which may be used to remove, break up, or otherwise destroy one or more organ obstructions, such as kidney stones.

Display <NUM> may be a single, or at least a dual display, with either multiple screens or multiple displays on one screen. In one example, one of the displays may show an image or images currently or previously obtained by endoscope <NUM>. The other display may show an image or video obtained from one or more additional imaging devices <NUM>, such as by X-ray, Magnetic Resonance Imaging, Computerized Tomography Scan, rotational angiography, ultrasound, or another appropriate internal imaging device. Alternatively, one of the displays <NUM> may show an image modified using one or more image enhancement techniques discussed herein, while another may display an unenhanced image. Alternatively, one of the displays <NUM> may show an image modified using one or more enhancement techniques discussed herein, while another of the displays <NUM> may show an image modified using one or more different enhancement techniques discussed herein.

The software or applications may manipulate, process, and interpret received images from imaging device <NUM> to identify the location, size, and characteristics of the aiming beam, kidney stone, or other material. As will be discussed further herein, the frame processor <NUM> may process and enhance received images from endoscope <NUM>.

The physician may insert endoscope <NUM> into a patient when performing a medical procedure such as a lithotripsy to remove a kidney stone. The display <NUM> may become partially or completely obscured by pieces of kidney stone or other floating particulate matter, for example, when illuminated by a light on the endoscope <NUM>. Pieces of the kidney stone may need to be removed via the exit channel used by the endoscope <NUM> (the access sheath, ureter, etc.). However, it may be difficult for the physician <NUM> to ascertain whether the kidney stone is too large to fit out the exit channel, and whether it should be broken up further in order to fit out the exit channel. The physician may attempt to remove the kidney stone in question via the exit channel, but the sharpness of the stone may cause tissue damage if it is too large, which may injure the patient, increase recovery time, etc. Techniques are needed to more effectively identify whether a kidney stone will fit out the exit channel.

<FIG> is a flow diagram of an exemplary method not forming part of the invention as claimed for processing medical images, according to aspects of the present disclosure. A source of video or image frames <NUM>, which may be any medical device, such as an endoscope <NUM> or imaging device <NUM>, may provide frames to signal in <NUM>. The frames may be provided to a frame handler <NUM>, which may store a plurality of frames. One or more frames <NUM> may be provided to a frame processor <NUM>, which may produce one or more processed frames <NUM> via techniques discussed herein. The processed frames <NUM> may be provided to the signal out <NUM>, which may be shown on display <NUM>.

The signal in <NUM> may be a software handler that may transmit that a new frame has been received. The frame handler <NUM> may either directly send a frame via the signal out <NUM> to a display <NUM>, or it may send one or more frames to the frame processor <NUM>. As will be discussed elsewhere herein, the frame processor <NUM> may perform object size determination techniques. The frame handler <NUM> may also send the original frame to the display <NUM>, and also send a copy of the frame to the frame processor <NUM>. The processed frame <NUM> may be received and also forwarded to the display <NUM>. This may allow for the original frame to be displayed alongside the processed frame <NUM> at the display <NUM>. Alternatively, the frame handler <NUM> may send the source frame <NUM> to the frame processor <NUM>, and the frame processor may return a processed frame <NUM> that comprises a dual display of the original and enhanced frame. Accordingly, the processed frame <NUM> may be larger than the source frame. The frame processor <NUM> may further add buttons or other user interface elements to the processed frame <NUM>.

Although techniques discussed herein are discussed as happening on the frame processor <NUM>, which may be depicted as being located on a single device, any of the functions of the frame processor may be spread across any number of devices, for example, any of the devices depicted in system <NUM>. Further, one or more of the signal in <NUM>, frame handler <NUM>, and/or signal out <NUM> may be housed on one or more servers <NUM>, or any of the other devices pictured on system <NUM>.

<FIG> is a flow diagram of an exemplary method not forming part of the invention as claimed for processing medical images to determine the size of objects, according to aspects of the present disclosure. A plurality of frames <NUM> may be received from a frame source. The frame source may comprise an endoscope <NUM> or other medical imaging device <NUM>. One or more frames <NUM> may be accumulated at a frame buffer. A frame may comprise an image of a kidney stone <NUM>, or other object. While the kidney stone <NUM> may be illuminated by a light source on the tip of the endoscope or other medical device <NUM>, such as by an optical fiber emitting white light, a laser beam <NUM>, or other electromagnetic beam, may also be present and allow for aiming the tip of the endoscope, either for purposes of retrieving the kidney stone or performing lithotripsy, etc. The beam <NUM>, when illuminated, may cause a halo <NUM> to be visible around the beam, which may be due to the intensity of the laser. Steps may then be taken to determine the size of the beam <NUM>.

The halo <NUM> may be distinguished from the beam <NUM>. As lasers are typically a particular color, such as red, in the blue or green "non-matching" channels <NUM> there may be no trace or faint trace of the beam <NUM>, and no trace of the halo <NUM>. Conversely, in the "matching" red channel <NUM>, the laser and the halo <NUM> might appear together as a large, bright, and indistinguishable entity. Similarly, a green laser and its halo might show up as a single bright indistinguishable entity in the green color channel, while at least the lower-intensity halo would not show up in the red channel or blue channel, etc. Thus, the color of the laser may be initially determined for purposes of separating color channels and performing techniques discussed herein. In addition, the endoscope may have a white light from an optical cable to help the user navigate. The white light may create reflections that might be confused by the algorithm as the aiming beam. By finding the aiming beam by comparing different color channels, this problem is avoided. A white light would show up equally in different color channels. A colored laser, or at least the halo, may show up primarily in the corresponding color channel.

The aiming beam may be intense in all color channels, even though the aiming beam may be a laser of a particular color. This may make distinguishing the aiming beam from other light reflections, for example a light reflection from an LED on the endoscope, difficult. However, the aiming beam may have an associated halo around it, which may allow for differentiation of the aiming beam from other light sources or reflections. In addition, the halo may only appear in a particular color channel, for example the channel of the color of the laser. Thus, the halo may be reliably identified by comparing different color channels. Once the beam of matching channel <NUM> (e.g. red channel for a red laser) and smaller beam <NUM> of the non-matching channel (green and/or blue channels for a red laser) <NUM> are determined, the two channels may be combined to form a mask area in order to more accurately determine the halo. This may be done by subtracting the non-matching channel <NUM> from the matching channel <NUM>, which may produce an image of the halo with the beam <NUM> removed at <NUM>. Alternatively, one of the channels may be inverted at <NUM> to form an inverted channel <NUM>. The inverted channel <NUM> may be added to the, e.g., non-matching channel to form the halo <NUM>. A bounding box or other boundary may be placed around the halo/mask area <NUM> for further image analysis.

After the bounding box <NUM> is determined, the aiming beam <NUM> in the center of the halo and within the bounding box may be determined. Image artifacts <NUM> may also be present. The artifacts <NUM> may show up, for example, if the laser reflects off of objects near the aiming beam itself. The algorithm, by comparing the various candidate aiming beams, may consider the largest object to be the true aiming beam <NUM>, and may discard or disregard the artifacts <NUM>. The edges of the true aiming beam <NUM> may be determined using an algorithm such as Canny edge detection, Hough transform, etc. The algorithm may then approximate the true shape and size of the aiming beam <NUM> by placing a circle or ellipse around the aiming beam to form an aiming beam ellipse <NUM>. This may be done dynamically, or the shape of the aiming beam may be previously known. This ellipse placement may be performed based on the determined aiming beam edges. Multiple candidate ellipses may be fit over the detected edges, and the best-fitting ellipse may be determined, e.g. the aiming beam ellipse <NUM>. Metrics may be determined for the aiming beam ellipse <NUM>, such as the measurements of the major and minor axes.

The aiming beam may be of a standard, predetermined size and shape. Since the aiming beam <NUM> is a laser or other highly directional or unidirectional electromagnetic light source, it does not get substantially larger with distance. Hence, whatever object upon which the aiming beam is projected may be measured by using the aiming beam and/or its halo as a standard metric. Thus, kidney stones or other objects may be measured by evaluating the aiming beam and/or halo. However, difficulties may arise that may complicate an accurate measurement. Kidney stones are often jagged and contain crevices or other irregularities that distort the apparent shape of the aiming beam. Thus, the aiming beam may appear abnormally small or abnormally large, which may cause incorrect estimates of the size of any object being measured based on the aiming beam.

To mitigate this problem, the image of surface upon which the aiming beam is being aimed may be evaluated by image analysis within the bounding box, or otherwise within a predetermined distance of the aiming beam <NUM> and/or halo <NUM>, may be extracted at <NUM>. The image features may comprise image lighting, texture, entropy, artifact detection, etc. The extracted image features plus the major and minor axes of the ellipse <NUM> may be provided to a trained machine-learning system, which may apply weights to alter the estimated size of the aiming beam <NUM> and/or halo <NUM>.

The machine learning system may be trained. A size (e.g., in pixels) of a ground truth indicator may be determined. The ground truth indicator may be a physically drawn circle or other shape onto an image of the kidney stone representing the true size of aiming beam. In the training process, the size of the physically drawn circle will be extracted from the image. A correlation between the size of the ground truth indicator and the aiming beam properties (major and minor axes of ellipse <NUM>, image lighting, texture, entropy, artifact detection, etc.) may be determined. In the inference step of the production version, the machine learning model may calculate the size of the aiming beam. This may be accurately performed despite distortions produced by the surface upon which the aiming beam is being projected.

As discussed above, the dimensions of the aiming beam may be known. For example, the aiming beam may be a laser that, when shined on a surface, creates a "dot" <NUM> across. As the laser is a directional beam, the dot may be <NUM> across no matter how far away the object is upon which the dot is being shined. Thus, based on the final estimated shape and/or size of the aiming beam, measurements of the surface on which the aiming beam is projected may be determined. This may be done by assessing the size of an object relative to the known size of the laser dot. For example, if a laser dot is known to be <NUM> across, and the kidney stone upon which it is shined is approximately double the width of the laser dot, the kidney stone may be determined to be <NUM> across.

Using this same technique, the size of the exit channel is determined and displayed to the user <NUM>. Thus, using this scaling technique, the size of objects may be determined, and the accurate size of projected virtual objects may also be determined. A visual indicator <NUM> representing the size of the exit channel relative to the surface upon which the aiming beam is projected is displayed on the display. The user <NUM> may thus be able to determine if the kidney stone or other object, upon which the aiming beam is shined, will fit out the exit channel visually and with minimal cognitive load. Alternatively or in addition, one or more rulers may be displayed, such as rulers <NUM> along the X or Y axis. Visual indicators of the bounding box <NUM> and/or final aiming beam ellipse <NUM> may also be displayed on a display to the user <NUM>.

<FIG> is a flow diagram of an exemplary method not forming part of the invention as claimed for determining medical image enhancement, according to techniques discussed herein. At step <NUM>, an image frame from the medical device may be received, and at step <NUM> a first color channel and a second color channel in the image frame may be determined. At step <NUM> a location of an electromagnetic beam halo may be identified by comparing the first color channel and second color channel. At step <NUM>, edges of an electromagnetic beam may be determined based on the electromagnetic beam halo, and at step <NUM> size metrics of the electromagnetic beam may be determined based on the edges of the electromagnetic beam. At step <NUM> a visual indicator or other projection on the image frame may be displayed based on the size metrics of the electromagnetic beam. As discussed elsewhere herein, the visual indicator may comprise a visual representation of the exit channel, a ruler, a bounding box, an electromagnetic beam halo, and/or an electromagnetic beam.

<FIG> illustrates an exemplary system that may be used in accordance with techniques discussed in <FIG>, according to aspects of the present disclosure. <FIG> is a simplified functional block diagram of a computer that may be configured as server <NUM>, endoscope <NUM>, imaging device <NUM>, and/or user device <NUM>, according to exemplary embodiments of the present disclosure. Specifically, in one embodiment, any of the user devices, servers, etc., discussed herein may be an assembly of hardware <NUM> including, for example, a data communication interface <NUM> for packet data communication. The platform also may include a central processing unit ("CPU") <NUM>, in the form of one or more processors, for executing program instructions. The platform may include an internal communication bus <NUM>, and a storage unit <NUM> (such as ROM, HDD, SDD, etc.) that may store data on a computer readable medium <NUM>, although the system <NUM> may receive programming and data via network communications. The system <NUM> may also have a memory <NUM> (such as RAM) storing instructions <NUM> for executing techniques presented herein, although the instructions <NUM> may be stored temporarily or permanently within other modules of system <NUM> (e.g., processor <NUM> and/or computer readable medium <NUM>). The system <NUM> also may include input and output ports <NUM> and/or a display <NUM> to connect with input and output devices such as keyboards, mice, touchscreens, monitors, displays, etc. The various system functions may be implemented in a distributed fashion on a number of similar platforms, to distribute the processing load. Alternatively, the systems may be implemented by appropriate programming of one computer hardware platform.

The disclosed techniques may help enable efficient and effective procedures to breakup and/or remove material from a patient's organ. In particular, the user may easily view the processed frames to assist with, for example, removing kidney stones within the patient's kidney.

Moreover, while examples discussed in this disclosure are commonly directed to ureteroscopic kidney stone removal, with or without lithotripsy, it is further contemplated that the systems and procedures discussed herein may be equally applicable to other material removal procedures. For example, the systems and methods discussed above may be used during a percutaneous nephrolithotomy/nephrolithotripsy (PCNL) to plan for a procedure and mid-procedure to locate any missed kidney stones. The systems and methods discussed above may also be used to plan for or conduct procedures to remove ureteral stones, gallstones, bile duct stones, etc..

Claim 1:
A system (<NUM>; <NUM>) for processing electronic images from a medical device (<NUM>), comprising:
at least one data storage device (<NUM>) storing instructions (<NUM>) for processing electronic images; and
at least one processor (<NUM>) configured to execute the instructions (<NUM>) to perform operations for processing electronic images, the operations comprising:
receiving an image frame from the medical device (<NUM>), wherein the image frame is acquired with the medical device (<NUM>) when performing a medical procedure to remove an object from a patient's organ via an exit channel;
determining a first color channel (<NUM>) and a second color channel (<NUM>) in the image frame;
identifying a location of an aiming beam halo (<NUM>) of an aiming beam (<NUM>) by comparing the first color channel (<NUM>) and second color channel (<NUM>); and
determining edges of the aiming beam (<NUM>) based on the aiming beam halo (<NUM>);
determining size metrics of the aiming beam (<NUM>) based on the edges of the aiming beam (<NUM>);
displaying a visual indicator (<NUM>) on the image frame based on the determined edges of the aiming beam halo (<NUM>);
wherein the size metrics correspond to a diameter or radius of the aiming beam (<NUM>), and wherein the visual indicator (<NUM>) corresponds to a size of the exit channel.