User-interface for visualization of endoscopy procedures

A user-interface for visualizing a colonoscopy procedure includes a video region and a navigational map upon which coverage annotations are displayed. A live video feed received from a colonoscope is displayed in the video region. The navigational map depicts longitudinal sections of a colon. The coverage annotations are presented on the navigation map and indicate whether one or more of the longitudinal sections is deemed adequately inspected or inadequately inspected during the colonoscopy procedure.

TECHNICAL FIELD

This disclosure relates generally to endoscopy, and in particular, but not exclusively, to user-interfaces to aid colonoscopy.

BACKGROUND INFORMATION

When an endoscopist performs a colonoscopy, one of the most important tasks during the withdrawal phase is to ensure that they have visualized every surface of the colon in order to detect all the polyps. 20% to 24% of polyps that have the potential to become cancerous (adenomas) are missed. Two major factors that may cause an endoscopist to miss a polyp are: (1) the polyp appears in the field of view, but the endoscopist misses it, perhaps due to its small size or flat shape; and (2) the polyp does not appear in the field of view, as the endoscopist has not fully covered the relevant area during the procedure.

Conventional products that assist clinicians/endoscopists with detecting polyps do not currently support features for coverage visualization.

DETAILED DESCRIPTION

Embodiments of a system, apparatus, and method for a user-interface (UI) to aid visualization of an endoscopy (particularly colonoscopy) procedure are described herein. In the following description numerous specific details are set forth to provide a thorough understanding of the embodiments. One skilled in the relevant art will recognize, however, that the techniques described herein can be practiced without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring certain aspects.

Conventional endoscopy and colonoscopy interfaces only display the live video feed on the screen without providing any other user aids. Embodiments of the colonoscopy user-interface (UI) described herein introduce additional on-screen elements to support and aid the endoscopist fully visualize every surface patch of a colon to improve polyp detection and the reliability of the overall colonoscopy procedure. In certain embodiments, machine learning (ML) models may be used to track relative position, depth, and angle of the colonoscope camera within the colon. Examples of these image analysis techniques are described in “Detecting Deficient Coverage in Colonoscopies,” Freedman et al., IEEE Transactions On Medical Imaging, Vol. 39, No. 11, November 2020. ML models may further be trained to provide polyp detection and/or optical biopsies. Position, depth, and angle tracking along with feature detection (polyp detection) and optical biopsy may all be performed based upon image analysis of the video output from the colon. In other embodiments, additional position sensors or real-time scanning techniques may be implemented to obtain position/depth tracking information of the distal end of the colonoscope.

The data obtained from the above image analysis of a live video feed from a colonoscope may be leveraged to display a number of beneficial on-screen visual aids in a colonoscopy UI. These visual aids provide improved operator context and visualization of the colonoscopy procedure. For example, these aids may include a navigational map that depicts longitudinal sections of a colon, a position marker indicating a position of a field of view (FOV) of a camera capturing the live video feed, annotations indicating inspection status of different longitudinal sections of a colon, a cross-sectional coverage map indicating whether portions or surface patches of a longitudinal section have been adequately inspected, guidance arrows prompting the endoscopist back to a longitudinal section deemed inadequately inspected, annotations highlighting detected polyps, and display of a variety of other valuable feedback data (e.g., estimated withdrawal time, polyp detected status, polyp detected history, important notifications, etc.). It should be appreciated that the terms “annotate” or “annotation” are broadly defined herein to include both textual markups (e.g., on screen textual prompts or dialog) and graphical/pictorial markups (e.g., on screen boxes, arrows, shading, coloring, highlighting, etc.).

Providing these visual aids on the colonoscopy UI in real-time and contemporaneously alongside the live video feed from the colonoscope provides a higher level of context and orientation to the endoscopist. The visual aids increase confidence that all surface patches of the colon (i.e., the internal surfaces of the colon) have been reviewed or provide actionable, real-time feedback to guide the endoscopist back to a missed surface patch. Ultimately, the visual aids improve the operator experience thus providing improved detection of polyps and improved confidence in the overall colonoscopy procedure.

FIG.1Aillustrates a colonoscopy tower system100, in accordance with an embodiment of the disclosure. System100illustrates an example hardware system in which embodiments of the improved colonoscopy UI described herein may be used. System100includes an endoscope or colonoscope105coupled to a display110for capturing images of a colon and displaying a live video feed of the colonoscopy procedure. In one embodiment, the image analysis and UI overlays described herein may be performed and generated by a processing box that plugs in between the colonoscope105and display110.FIG.1Billustrates an example endoscopy video assistant (EVA)115capable of generating the colonoscopy UI described herein. EVA115may include the necessary processing hardware and software, including ML models, to perform the real-time image processing and UI overlays. For example, EVA115may include a data storage, a general-purpose processor, graphics processor, and video input/output (I/O) interfaces to receive a live video feed from colonoscope105and output the live video feed within a UI that overlays various visual aids and data. In some embodiments, EVA115may further include a network connection for offloading some of the image processing and/or reporting and saving coverage data for individual patient recall and/or longitudinal, anonymized studies. The colonoscopy UI may include the live video feed reformatted, parsed, or scaled into a video region (e.g., video region205inFIG.2), or may be a UI overlay on top of the existing colonoscopy monitor feed to maintain the original format, resolution, and integrity of the colonoscopy live video feed as well as reduce any latency.

FIG.2illustrates a colonoscopy UI200for visualizing a colonoscopy procedure, in accordance with an embodiment of the disclosure. The illustrated embodiment of colonoscopy UI200includes a video region205for displaying a live video feed, a navigation map210with a position marker215, a cross-sectional coverage map220, and a region for procedure data225. The illustrated embodiment of procedure data225includes scope information230, procedure timer(s)235, withdrawal timer240, polyp detected status245, polyp detected history250, and notifications255.

As mentioned, video region205provides a region within colonoscopy UI200to display a live video feed of the interior of a colon captured during a colonoscopy procedure by a camera of colonoscope105. In other words, video region205may be used to display the real-time FOV captured by the camera of colonoscope105. Although video region205is illustrated as having a round FOV, in other embodiments, the FOV may be rectangular, square, or otherwise.

Navigation map210depicts longitudinal sections of the colon. Each longitudinal section represents a different depth into the colon (or large intestine) extending from the rectum or anal canal to the cecum. Navigation map210may be implemented as an anatomical atlas or caricature being representative of the colon, or an actual three-dimensional (3D) model of the colon. In the case of a 3D model, the 3D model of the colon may be generated during an insertion phase of the colonoscopy procedure as colonoscope105is inserted into the anal canal and moved towards the cecum. The live video feed during insertion may be analyzed and mapped into the 3D model. In the illustrated embodiment, navigation map210is annotated with position marker215to indicate a position of the FOV of the live video feed and by extension the distal end of colonoscope105within the colon. In one embodiment, position marker215does not appear on navigation map210until after the colon has been fully mapped or traversed during the insertion phase. After the insertion phase, position marker215moves in real-time tracking the position of the distal end of colonoscope105and the FOV of the live video feed during the withdrawal phase.

FIGS.3A-Dillustrate further details of navigational map210, in accordance with an embodiment of the disclosure. As illustrated inFIG.3A, navigational map210may be initially presented in a lighter shade or grayed out shade during the insertion phase of the colonoscopy procedure. In yet other embodiments, navigational map210may not be initially presented until the end of the insertion phase or beginning of the withdrawal phase. The insertion phase may be deemed complete once the cecum is reached and recognized as the end of the colon. The colon illustration may be withheld, grayed out, or presented in a lighter shade while the colon is being spatially mapped during the insertion phase. The spatial mapping may be achieved using a 3D visual mapping via image analysis of the live video feed during the insertion phase. In other embodiments, additional sensors and/or tracking devices may be used (alone or in conjunction with the image analysis) to facilitate spatial mapping or generation of a full 3D model of the colon. For example, ultrasound imaging, magnetic tracking, etc. may be used to track the distal tip of colonoscope105as it progresses through the colon.

InFIG.3B, upon commencement of the withdrawal phase, navigation map210is fully presented and position marker215displayed. Navigation map210along with position marker215present the endoscopist with a visual representation of the position of the FOV of the live video feed within the colon along with a visual estimation of the remaining distance to traverse during the withdrawal phase.

Referring toFIGS.3C and3D, as colonoscope105is withdrawn through the colon, navigation map210is annotated to illustrate the inspection status of each longitudinal section along the way. This annotation may be updated in real-time during the withdrawal phase. Longitudinal sections deemed fully inspected (i.e., all surface patches in those longitudinal sections have been adequately inspected) are annotated as such. For example, longitudinal sections that are deemed adequately inspected may be colored green (FIG.3C). Correspondingly, if the endoscopist withdrawals through a given longitudinal section without fully inspecting every surface patch within that longitudinal section, then the corresponding longitudinal section on navigation map210is annotated to represent an inadequate inspection. For example, the inadequately inspected section may be colored red (FIG.3D) to indicate that one or more surface patches of the colon in the longitudinal section has been deemed inadequately inspected. Of course, other colors, shades, or labels may be used to indicate adequate or inadequate inspection of a given longitudinal section.

FIG.4illustrates a guidance arrow405overlaying a live video feed400guiding an operator (endoscopist) of colonoscope105back to one of the longitudinal sections of the colon deemed inadequately inspected. As illustrated, navigational map410is annotated to depict that a longitudinal section413is deemed inadequately inspected. Since the distal end of colonoscope105has been withdrawn past longitudinal section413(see position marker415), guidance arrow405may be overlaid on live video feed400within video region205to visually guide the endoscopist back to longitudinal section413. Guidance arrow405can help the endoscopist navigate the twist, turns, and folded anatomical structures of the colon to return to longitudinal section413for a more thorough inspection of the missed surface patches.

The data of adequately vs inadequately inspected longitudinal sections and/or surface patches may be anonymized by EVA115and uploaded to a server or cloud-based service to collect coverage data from a multitude of colonoscopy procedures. As illustrated inFIG.5, a coverage map505may be generated based upon many colonoscopies to instruct practitioners in the field, which surface patches of a colon are more likely to be missed than others, where polyps are commonly found, etc. This feedback data may also be correlated with anonymized demographic data including age, race, sex, etc. to provide further insights and improve the success, reliability, and confidence in colonoscopy procedures.

Returning toFIG.2, the illustrated embodiment of colonoscopy UI200further includes a cross-sectional coverage map220. Cross-sectional coverage map220indicates whether angular portions of a cross-section of a given longitudinal section of the colon is deemed adequately or inadequately inspected. For example, cross-section coverage map220may display a cross-sectional map of the current longitudinal section indicated by position marker215. In the illustrated embodiment, cross-sectional coverage map220is indicating that only the surface patch of the colon residing in the upper left quadrant of the current longitudinal section has been adequately inspected and the remaining 76% of the perimeter surface patches of the current longitudinal section have not yet been adequately inspected. During the insertion phase, the image inspection software (e.g., trained neural networks) maps and orients itself to the colon. During the withdrawal phase, cross-sectional coverage map220may map surface patch inspection status relative to the frame of reference of the FOV of the camera during the insertion phase. In other embodiments, cross-sectional coverage map220maps surface patch inspections relative to a current frame of reference or other anatomical reference frames (e.g., sagittal, coronal, or median planes).

Turning toFIG.6, as the endoscopist commences withdrawal through a longitudinal section of the colon, cross-sectional coverage map620A initially displays 0% inspection coverage. While loitering in and inspecting surface patches of a given longitudinal section, EVA115tracks the inspection and begins to highlight cross-sectional coverage map620B to reflect an estimated inspection coverage. As the endoscopist continues to inspect a given longitudinal section, more of the circle of cross-sectional coverage map620C is highlighted until all surface patches of the current longitudinal section are deemed inspected, as represented by cross-sectional coverage map620D showing 100% inspection coverage. As colonoscope105is withdrawn to the next longitudinal section, the inspection status is reset to 0% and the process repeats. If colonoscope105is withdrawn past a longitudinal section before that section is fully inspected, when the endoscopist returns to the missed longitudinal section for reinspection, cross-sectional coverage map620E highlights (e.g., colored red) the missed area to quickly guide the endoscopist to the missed location/surface patch(es).

The inspection status may be determined or estimated using a combination or weighting of one or more of the following factors: (a) loitering time of a camera of colonoscope105within the given longitudinal section; (b) a determination of whether all surface patches of the colon within the given longitudinal section is observed by the camera (e.g., sweeps within the FOV of the camera for a threshold period of time); (c) a distance between each of the surface patches and the camera when each of the surface patches is observed by the camera; (d) an angle of viewing incidence between the camera and each of the surface patches when each of the surface patches is observed by the camera, or (e) an ML analysis of the colonoscopy video to determine whether any scene potentially included an anatomical fold or area where additional colon anatomy may have be hidden from the FOV. The distance and viewing angles may be thresholded such that surface patches that technically sweep into the FOV of the camera but are either too far away or occur at too steep of an angle may be deemed to not have been adequately observed even though the surface patch did pass within the FOV of the camera. When operating within threshold limits for viewing distance and angle of viewing incidence, loitering times may be adjusted depending upon the actual viewing distance and/or angle of viewing incidence. For example, a viewing distance that does not exceed a threshold maximum may still require twice the loitering time if its distance is considered longer than typical, but does not exceed a maximum distance permitted. Yet another factor that may be considered when determining inspection status is image quality while observing a given surface patch, which may include focus, contrast, sharpness or other image quality characteristics. Again, permissible thresholds may be enforced and loitering multipliers applied for sub-optimal conditions when observing a given surface patch. In some embodiments, any or all of the above factors may be used as ground truth data when training an ML model to estimate or otherwise “deem” an longitudinal section as adequately or inadequately inspected.

In one embodiment, cross-sectional coverage map220(or620A-620D) may visually indicate the angular portions observed/not observed for a given longitudinal section. In this manner, the endoscopist is quickly guided as to which perimeter surface patches still need to be observed for a given depth or longitudinal position. In yet another embodiment, cross-sectional coverage map220is merely an overall percentage estimate of the surface patches observed within a longitudinal section without specifically connoting angular directions of observed and unobserved portions.

Returning toFIG.2, colonoscopy UI200includes a region for displaying procedure data225. The illustrated embodiment of procedure data225includes scope information230, procedure timer235, withdrawal timer240, polyp detected status245, polyp detected history250, and notifications255. Scope information230may include metadata pertinent to the particular colonoscope105such as camera resolution, software/firmware version, frame rate, color space, etc.

Procedure timer(s)235may include one or more timers that track the overall procedure time since commencement of the insertion phase, track the procedure time of just the insertion phase, or track the procedure time since commencement of the withdrawal phase. Withdrawal timer240displays an estimated withdrawal time to complete the withdrawal phase of the colonoscopy procedure. The estimated withdrawal time may be calculated using a trained neural network upon inspecting the colon during the insertion phase and may further be updated as the withdrawal phase progresses. As such, the estimated withdrawal time may not be displayed until after completion of the insertion phase and represents a sort of countdown timer until completion of the withdrawal phase.

Polyp detect status245represents an indication of whether the image analysis and polyp detect software has detected a polyp in the current FOV or live image feed currently displayed in video region205. Referring toFIG.7, if a polyp is detected in live video feed705, then the detected polyp may be highlighted or accentuated with an annotation710clearly identifying its location within the displayed image. AlthoughFIG.7illustrates annotation710as a box outline, the annotation may be implemented using a variety of different shapes, colors, shadings, labels, etc.

Polyp detected history250represents a count of the overall number of detected polyps. Additionally, polyp detected history250may include a selectable menu for displaying further information regarding the particular detected polyps. For example, if an ML classifier is applied to perform optical biopsies on the detected polyps, then the results of the optical biopsy (seeFIG.8A) may be accessed via the polyp detected history250by selecting a given polyp. Alternatively, optical biopsy results and/or reference images for comparison may automatically appear when a polyp is identified in the FOV. The results may include a classification of benign, precancerous, cancerous, etc. along with display of a confidence interval. In yet other embodiments, the classification may include other classifications such as hyperplastic polyp (FIG.8B), adenomatous polyp (FIG.8C), etc. Reference images of polyps (FIGS.8B and8C) corresponding to the classification of a polyp may be linked and presented upon selection so that the endoscopist may compare the live video feed image against a reference image during real-time observation of a given polyp. Alternatively,FIG.8B or8Cmay represent or include static screen captures of the polyp enabling the endoscopist to more easily inspect the polyp without worrying about moving/controlling the colonoscope to maintain an adequately still image for live inspection. Finally, procedure data225may further include a section for notifications255where miscellaneous notifications including polyp types/classifications may also be presented.

Embodiments disclosed herein provide a colonoscopy UI200that contemporaneously presents the live video feed from colonoscope105alongside contextual/orientational data from navigation map210, cross-sectional coverage map220, and procedure data225. These contemporaneous visual aids provide a higher level of context and orientation to the endoscopist, thereby improving the reliability of the colonoscopy procedure and confidence that all polyps are detected.

FIG.9is a block diagram that illustrates aspects of a demonstrative computing device appropriate for implementing EVA115, in accordance with embodiments of the present disclosure. Those of ordinary skill in the art will recognize that computing device900may be implemented using currently available computing devices or yet to be developed devices.

In its most basic configuration, computing device900includes at least one processor902and a system memory904connected by a communication bus906. Depending on the exact configuration and type of device, system memory904may be volatile or nonvolatile memory, such as read only memory (“ROM”), random access memory (“RAM”), EEPROM, flash memory, or similar memory technology. Those of ordinary skill in the art will recognize that system memory904typically stores data and/or program modules that are immediately accessible to and/or currently being operated on by the processor902. In this regard, the processor902may serve as a computational center of computing device900by supporting the execution of instructions.

As further illustrated inFIG.9, computing device900may include a network interface910comprising one or more components for communicating with other devices over a network. Embodiments of the present disclosure may access basic services that utilize network interface910to perform communications using common network protocols. Network interface910may also include a wireless network interface configured to communicate via one or more wireless communication protocols, such as WiFi, 2G, 3G, 4G, LTE, WiMAX, Bluetooth, and/or the like.

In the exemplary embodiment depicted inFIG.9, computing device900also includes a storage medium908. However, services may be accessed using a computing device that does not include means for persisting data to a local storage medium. Therefore, the storage medium908may be omitted. In any event, the storage medium908may be volatile or nonvolatile, removable or nonremovable, implemented using any technology capable of storing information such as, but not limited to, a hard drive, solid state drive, CD-ROM, DVD, or other disk storage, magnetic cassettes, magnetic tape, magnetic disk storage, and/or the like.

The illustrated embodiment of computing device900further includes a video input/out interface911. Video I/O interface911may include an analog video input (e.g., composite video, component video, VGG connector, etc) or a digital video input (e.g., HDMI, DVI, DisplayPort, USB-A, USB-C, etc.) to receive the live video feed from colonoscope105and a similar type of video output port to output the live video feed within colonoscopy UI200to display110. In one embodiment, video I/O interface911may also represent a graphics processing unit capable of performing the necessary computational video processing to generate and render colonoscopy UI200.

As used herein, the term “computer-readable medium” includes volatile and non-volatile and removable and non-removable media implemented in any method or technology capable of storing information, such as computer-readable instructions, data structures, program modules, or other data. In this regard, the system memory904and storage medium908depicted inFIG.9are merely examples of computer-readable media.

Suitable implementations of computing devices that include a processor902, system memory904, communication bus906, storage medium908, and network interface910are known and commercially available. For ease of illustration and because it is not important for an understanding of the claimed subject matter,FIG.9does not show some of the typical components of many computing devices. In this regard, the computing device900may include input devices, such as a keyboard, keypad, mouse, microphone, touch input device, touch screen, tablet, and/or the like. Such input devices may be coupled to computing device900by wired or wireless connections including RF, infrared, serial, parallel, Bluetooth, USB, or other suitable connection protocols using wireless or physical connections. Since these devices are well known in the art, they are not illustrated or described further herein.

The above user-interface has been described in terms of a colonoscopy and is particularly well-suited as a colonoscopy user-interface to aid visualization of colonoscopy procedures. However, it should be appreciated that user-interface200may be more broadly/generically described as an endoscopy user-interface that may be used to visualize endoscopy procedures, in general, related to other anatomical structures. For example, the user-interface is applicable to aid visualization of other gastroenterological procedures including endoscopy procedures within the upper and lower gastrointestinal tracts. In yet other examples, the user-interface may be used to visualize exploratory endoscopy procedures of non-gastroenterological structures such as the esophagus, bronchial tubes, other tube-like anatomical structures, etc. When adapting the user-interface to visualize other endoscopy procedures, navigational map210would represent a map of the corresponding anatomical structure being explored and cross-sectional coverage map220would represent cross-sectional or perimeter inspection coverage of the corresponding anatomical structure. Similarly, coverage map505illustrated inFIG.5could be adapted to represent inspection coverage of a variety of different anatomical structures and the optical biopsy images and reference images illustrated inFIGS.8A—8C may be adapted for various types of anatomical structures, tissue types, tumors, etc. to support clinical workflow related to other endoscopy procedures.

The processes and user-interface described above are described in terms of computer software and hardware. The techniques described may constitute machine-executable instructions embodied within a tangible or non-transitory machine (e.g., computer) readable storage medium, that when executed by a machine will cause the machine to perform the operations described. Additionally, some of the processes or logic for implementing the user-interface may be embodied within hardware, such as an application specific integrated circuit (“ASIC”) or otherwise.