Patent ID: 12236499

The drawings are not necessarily to scale. The drawings are merely schematic representations, not intended to portray specific parameters of the invention. The drawings are intended to depict only typical embodiments of the invention, and therefore should not be considered as limiting the scope of the invention. In the drawings, like numbering represents like elements.

DETAILED DESCRIPTION

Various embodiments provided herein relate to a computing infrastructure for processing a video feed to detect, size, and analyze objects and image data. In certain aspects, the video feed includes video captured during a clinical procedure, such as colonoscopy. The video feed is outputted in real-time to a practitioner performing the procedure with calculated information, i.e., annotations, such as lesion detection, size, histology, etc.

Various techniques exist for processing video feeds to provide such information. For example, U.S. Pat. No. 10,957,043 entitled AI Systems for Detecting and Sizing Lesions, which is hereby incorporated by reference, discloses machine learning techniques for detecting and sizing lesions, which can be displayed to the practitioner along with the video feed during a procedure. Technical challenges however arise due to the computational overhead required to implement such algorithms. When processing a video stream, traditional sequential computational techniques often cannot guarantee real-time results back to the practitioner. In order to address this issue, the present approach uses a distributed, multithreaded computing infrastructure to process and annotate video during the procedure.

Referring toFIG.1, an illustrative computing infrastructure10is shown that processes video streams16,16′, e.g., captured during procedures from different medical procedure rooms11,11′ and outputs annotated video18,18′ back to the medical practitioners performing the procedure. Video streams16,16′ may be captured from a camera or scope used during the procedure and the resulting annotated video18,18′ may be output on a display, such as a computer screen or the like. Each room11,11′ includes a local processing system12,12′ (or client) that includes: (1) a main (i.e., first) processing thread20,20′ that handles the input, output and annotation of image frames; and (2) an object (i.e., second) processing thread22,22′ for handling image processing tasks including the generation of annotation information. In addition, a set of cloud services14, e.g., provided by a remote cloud system (i.e., server) are provided that perform one or more complex image processing tasks, e.g., object detection, diagnostics, etc., for each of the local processing systems12,12′.

In one illustrative embodiment, each local processing system12,12′ includes a central processing unit (CPU) having multiple cores or a single core divided into two virtual cores or threads. Each thread is created by the operating system and performs a specific set of tasks. The two threads operate at the same time independently of each other. In one illustrative embodiment, the main processing thread20provides input/output and annotation functions on the video stream16, while object processing thread22manages cloud services14(which, e.g., provides a first image processing task) and performs further image processing tasks to generate annotation information. A memory buffer24may be deployed to buffer video frames or annotation information being transported between the two threads20,22. Using this multithreaded approach, various advantages are gained, e.g., if image processing gets bogged down in the object processing thread22, it will not interfere with the main processing thread's20ability to display video to the practitioner.

Cloud services14include one or more functions that provide a further level of image processing. For example, computationally intensive processes such as object detection and diagnostics using machine learning algorithms or the like may be implemented by cloud services14. Each cloud service14can be configured to perform a service in a highly efficient manner that manages computational overhead required for certain image processing tasks. An additional advantage of performing such specialized services on the cloud is that data from multiple rooms11,11′ or sources can be collected at a single location and used for analytical purposes, e.g., to continuously train an associated machine learning model.

FIG.2depicts a detailed process flow of a computing infrastructure that annotates and outputs video with lesion detection and sizing information. Annotated video18may for example include a graphically overlayed rectangle, oval, or other graphics indicia showing the location of a lesion and/or text data displaying information such as a diameter or histology of the detected lesion. In this embodiment, the main processing thread20receives a video stream16from an image source32(e.g., a camera, scope, etc.) that processes a sequence of image frames. In operation, a next, i.e., “current,” image frame is inputted at P1, and at P2the current image frame is forwarded to the display manager40(thus reproducing the inputted video stream16) where the current image frame is rendered at P3along with either previous annotation information (P6) or new annotation information (P13). A typical video stream16may for example include 32 frames per second that flow through the main processing thread20to the display manager40. The resulting annotated video18is then output to a physical display33. In an illustrative embodiment, the frames are in a BMP (bitmap image) file format.

Processes P4-P13describe the functionality that adds annotations to each frame at the display manager40. At P4, the main processing thread20determines if the object processing thread22is busy, e.g., processing a previous frame. If yes, thread20outputs previous annotation information, i.e., detection and size information determined for a previous frame at P5. This may also include retrieving previous raw annotation information from memory buffer24, which can, e.g., include a collection of graphical data items that represent types and properties of graphical objects, e.g., rectangles, circles, and/or text. Drawing functions in the main processing thread20are used to, e.g., draw a frame and annotations (e.g., BMP data), based on the data in the buffer24. The resulting graphics data is outputted to the display manager40(alternatively, the display manager40can simply be instructed to keep displaying the previous annotation information). At P6, display manager40graphically overlays (or simply keeps) previous annotation information on the current frame. Accordingly, this ensures that each displayed frame has annotation information overlayed thereon, even if the object processing thread22is unable to provide new annotation information for the current frame.

If the object processing thread is not busy at P4, then the current frame is converted to a JPEG format at P7and loaded to memory buffer24for processing by the object processing thread22.

At P8, the object processing thread22checks to see if a new frame is available in the buffer24. If no, process P8waits until one is available. If a new frame is available, the JPEG image is encoded at P9, e.g., to Base64, which is a common encoding scheme for transporting images. The encoded image is then forwarded to an object detection service30in the cloud (which may include any type of server), where an object detection algorithm is applied to the image data to determine if a lesion exists in the image. As noted, various image processing/machine learning techniques exist to perform object detection. In certain embodiments, service30returns an XML file containing detection results (e.g., new lesion detected, no new lesion detected, location, etc.). Once the service30provides a result, process P10determines if the result is a new result, i.e., a new lesion is detected. If no, then process P10instructs the main processing thread20to use the previous annotation information at P5for the current frame being displayed. If at P10a new result is detected by the object detection service30(indicative of a new lesion), then object sizing is performed at P11to provide sizing information of the lesion. Sizing of a lesion may likewise be performed using any known image processing technique. The resulting information (e.g., location, size, etc.) is packaged as a new set of annotation information and is loaded to memory buffer24. In an illustrative embodiment, the object detection information is returned as XML and is converted to graphical data when loaded into the buffer24.

At P12, main processing thread20detects and forwards the new annotation information to the display manager40. This may also include using drawing functions to convert annotation data in the buffer to graphics BMP data. At P13, the display manager40graphically overlays the new annotation information onto the current frame. The current frame and annotations are drawn simultaneously in the same thread, or the current frame is drawn along with the annotation from the previous frame.

Accordingly, each frame from the image source32is transported through the main processing thread20and is outputted as annotated video18to display33(e.g., a monitor in the procedure room). If the object processing thread22is available, an inputted BMP frame is converted to JPEG and sent through memory buffer24to the object processing thread22, where it is encoded to Base64 and forwarded to the object detection service30on the cloud. The results come back as XML and are converted to a format supporting graphical annotation, and object sizing is performed. Finally, the results go back through the memory buffer24to the main processing thread20, where new or previous annotation information is graphically overlayed with the current frame via the display manager40.

If the object processing thread22is busy processing the last frame, the main processing thread20uses the annotation information of the previously analyzed frame from the memory buffer24. In this case, the display manager40outputs visual annotation information for the previous frame on top of the current frame, and the result shows up on the monitor.

FIG.3depicts an illustrative cloud service14implementation in which each of the various local processing systems12,12′ (i.e., clients) interfaces with a router/server/queue proxy/dealer platform50. Each client has a corresponding REP worker52in the cloud. In operation, the cloud server starts a set of worker threads. Each worker thread creates an REP socket and then processes requests on a socket. Each REP socket is used by the service to receive requests from and send replies to a client. This socket type allows an alternating sequence of receive and subsequent send calls. Each request received is fair-queued from among all clients, and each reply sent is routed to the client that issued the last request. Worker threads behave just like single-threaded servers. The only differences are the transport (inproc instead of tcp), and the bind-connect direction.

The server also creates a ROUTER socket to talk to clients and binds this to its external interface (e.g., over tcp). The server further creates a DEALER socket to talk to the workers and binds these to an internal interface (e.g., over inproc). The server also starts a proxy that connects the two sockets. The proxy pulls incoming requests from all clients and distributes those out to workers. It also routes replies to a corresponding client.

Each client uses the REQ socket to send requests and receive replies from a service running on the cloud server. This socket type allows only an alternating sequence of sends and subsequent receive calls. A REQ socket may be connected to n numbers of REP or ROUTER sockets. Each request sent is round-robin-ed among all connected services, and each reply is matched with the last issued request. If no services are available on the cloud server, any send operation on the socket will block until at least one service becomes available. The REQ socket will not discard any messages.

A REP socket is used by a service running on the cloud server to receive requests and send replies to a client. This socket type allows only an alternating receive and subsequent send calls sequence. Each request received is queued from among all clients, and each reply sent is routed to the client that issued the last request. If the original requester does not exist anymore, the response is silently discarded.

A DEALER socket type talks to a set of anonymous peers, sending and receiving messages using round-robin algorithms. The DEALER works as an asynchronous REQ replacement for clients who talk to REP or ROUTER servers. The messages received by a DEALER are queued from all connected peers.

The ROUTER socket type talks to a set of peers, using explicit addressing to send each outgoing message to a specific peer connection. ROUTER works as an asynchronous replacement for REP and is often used as the basis for servers that talk to DEALER clients.

FIG.4depicts an illustrative annotated frame showing the results of a lesion detection. This image depicts a detected polyp inside the square.FIG.5depicts an illustrative annotated frame showing sizing information of a detected polyp. In this case, the polyp is encircled in an oval and is shown to be 11.6 mm in diameter.

It is understood that aspects of the described video processing infrastructure can be implemented in any manner, e.g., as a stand-alone system, a distributed system, within a network environment, etc. Referring toFIG.6, a non-limiting network environment101in which various aspects of the disclosure may be implemented includes one or more client machines102A-102N, one or more remote machines106A-106N, one or more networks104,104′, and one or more appliances108installed within the computing environment101. The client machines102A-102N communicate with the remote machines106A-106N via the networks104,104′.

In some embodiments, the client machines102A-102N communicate with the remote machines106A-106N via an intermediary appliance108. The illustrated appliance108is positioned between the networks104,104′ and may also be referred to as a network interface or gateway. In some embodiments, the appliance108may operate as an application delivery controller (ADC) to provide clients with access to business applications and other data deployed in a datacenter, the cloud, or delivered as Software as a Service (SaaS) across a range of client devices, and/or provide other functionality such as load balancing, etc. In some embodiments, multiple appliances108may be used, and the appliance(s)108may be deployed as part of the network104and/or104′.

The client machines102A-102N may be generally referred to as client machines102, local machines102, clients102, client nodes102, client computers102, client devices102, computing devices102, endpoints102, or endpoint nodes102. The remote machines106A-106N may be generally referred to as servers106or a server farm106. In some embodiments, a client device102may have the capacity to function as both a client node seeking access to resources provided by a server106and as a server106providing access to hosted resources for other client devices102A-102N. The networks104,104′ may be generally referred to as a network104. The networks104may be configured in any combination of wired and wireless networks.

A server106may be any server type such as, for example: a file server; an application server; a web server; a proxy server; an appliance; a network appliance; a gateway; an application gateway; a gateway server; a virtualization server; a deployment server; a Secure Sockets Layer Virtual Private Network (SSL VPN) server; a firewall; a web server; a server executing an active directory; a cloud server; or a server executing an application acceleration program that provides firewall functionality, application functionality, or load balancing functionality.

A server106may execute, operate or otherwise provide an application that may be any one of the following: software; a program; executable instructions; a virtual machine; a hypervisor; a web browser; a web-based client; a client-server application; a thin-client computing client; an ActiveX control; a Java applet; software related to voice over internet protocol (VoIP) communications like a soft IP telephone; an application for streaming video and/or audio; an application for facilitating real-time-data communications; a HTTP client; a FTP client; an Oscar client; a Telnet client; or any other set of executable instructions.

In some embodiments, a server106may execute a remote presentation services program or other program that uses a thin-client or a remote-display protocol to capture display output generated by an application executing on a server106and transmit the application display output to a client device102.

In yet other embodiments, a server106may execute a virtual machine providing, to a user of a client device102, access to a computing environment. The client device102may be a virtual machine. The virtual machine may be managed by, for example, a hypervisor, a virtual machine manager (VMM), or any other hardware virtualization technique within the server106.

In some embodiments, the network104may be: a local-area network (LAN); a metropolitan area network (MAN); a wide area network (WAN); a primary public network104; and a primary private network104. Additional embodiments may include a network104of mobile telephone networks that use various protocols to communicate among mobile devices. For short range communications within a wireless local-area network (WLAN), the protocols may include 802.11, Bluetooth, and Near Field Communication (NFC).

Elements of the described solution may be embodied in a computing system, such as that shown inFIG.7in which a computing device300may include one or more processors302, volatile memory304(e.g., RAM), non-volatile memory308(e.g., one or more hard disk drives (HDDs) or other magnetic or optical storage media, one or more solid state drives (SSDs) such as a flash drive or other solid state storage media, one or more hybrid magnetic and solid state drives, and/or one or more virtual storage volumes, such as a cloud storage, or a combination of such physical storage volumes and virtual storage volumes or arrays thereof), user interface (UI)310, one or more communications interfaces306, and communication bus312. User interface310may include graphical user interface (GUI)320(e.g., a touchscreen, a display, etc.) and one or more input/output (I/O) devices322(e.g., a mouse, a keyboard, etc.). Non-volatile memory308stores operating system314, one or more applications316, and data318such that, for example, computer instructions of operating system314and/or applications316are executed by processor(s)302out of volatile memory304. Data may be entered using an input device of GUI320or received from I/O device(s)322. Various elements of computer300may communicate via communication bus312. Computer300as shown inFIG.7is shown merely as an example, as clients, servers and/or appliances and may be implemented by any computing or processing environment and with any type of machine or set of machines that may have suitable hardware and/or software capable of operating as described herein.

Processor(s)302may be implemented by one or more programmable processors executing one or more computer programs to perform the functions of the system. As used herein, the term “processor” describes an electronic circuit that performs a function, an operation, or a sequence of operations. The function, operation, or sequence of operations may be hard coded into the electronic circuit or soft coded by way of instructions held in a memory device. A “processor” may perform the function, operation, or sequence of operations using digital values or using analog signals. In some embodiments, the “processor” can be embodied in one or more application specific integrated circuits (ASICs), microprocessors, digital signal processors, microcontrollers, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), multi-core processors, or general-purpose computers with associated memory. The “processor” may be analog, digital or mixed-signal. In some embodiments, the “processor” may be one or more physical processors or one or more “virtual” (e.g., remotely located or “cloud”) processors.

Communications interfaces306may include one or more interfaces to enable computer300to access a computer network such as a LAN, a WAN, or the Internet through a variety of wired and/or wireless or cellular connections.

In described embodiments, a first computing device300may execute an application on behalf of a user of a client computing device (e.g., a client), may execute a virtual machine, which provides an execution session within which applications execute on behalf of a user or a client computing device (e.g., a client), such as a hosted desktop session, may execute a terminal services session to provide a hosted desktop environment, or may provide access to a computing environment including one or more of: one or more applications, one or more desktop applications, and one or more desktop sessions in which one or more applications may execute.

As will be appreciated by one of skill in the art upon reading the following disclosure, various aspects described herein may be embodied as a system, a device, a method or a computer program product (e.g., a non-transitory computer-readable medium having computer executable instruction for performing the noted operations or steps). Accordingly, those aspects may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, such aspects may take the form of a computer program product stored by one or more computer-readable storage media having computer-readable program code, or instructions, embodied in or on the storage media. Any suitable computer readable storage media may be utilized, including hard disks, CD-ROMs, optical storage devices, magnetic storage devices, and/or any combination thereof.

The foregoing drawings show some of the processing associated according to several embodiments of this disclosure. In this regard, each drawing or block within a flow diagram of the drawings represents a process associated with embodiments of the method described. It should also be noted that in some alternative implementations, the acts noted in the drawings or blocks may occur out of the order noted in the figure or, for example, may in fact be executed substantially concurrently or in the reverse order, depending upon the act involved. Also, one of ordinary skill in the art will recognize that additional blocks that describe the processing may be added.