Patent ID: 12232900

DETAILED DESCRIPTION OF THE DISCLOSED EMBODIMENTS

The following detailed description illustrates exemplary aspects of the disclosed embodiments and ways in which they can be implemented. Although some modes of carrying out the aspects of the disclosed embodiments have been disclosed, those skilled in the art would recognize that other embodiments for carrying out or practising the aspects of the disclosed embodiments are also possible.

Referring toFIG.1, a schematic block diagram of an exemplary apparatus100for automated contour adjustment for data annotation is illustrated. The aspects of the disclosed embodiments are generally directed to automatically adjusting a position of one or more control points and initial contour(s) based on user input and characteristics of an object or region of interest. The need for manual intervention is minimized which improves annotation efficiency.

In one embodiment, the apparatus100can be implemented as a tool in a medical image or medical image annotation apparatus or system110. The apparatus100can be communicatively coupled to the imaging system110as shown in the example ofFIG.1. In alternative embodiments, the apparatus100can be embodied in or part of the imaging system110. Examples of such imaging systems can include, but are not limited to x-ray imaging systems, medical resonance imaging (MRI) systems, and computed tomography (CT) systems. Although medical imaging systems are generally referred to herein, the aspects of the disclosed embodiments are not so limited. In alternate embodiments, the aspects of the disclosed embodiments can be implemented in any imaging system where annotation of contours of an object or region of interest is desired.

As used herein, the term “annotation” generally refers to defining the edges or boundaries of an object or region of interest in an image. Referring toFIG.2, the image200is the scan of a body organ, which in this example is a scan of brain tissue. An object or region of interest204in this example is shown as the white or lighter area within or against the backdrop of the overall shape of the organ202. For purposes of the description herein, the area or region204will be referred to as the “object of interest.”

As an example, for a given a sequence of CT/MRI scan images of a patient with a tumor, the annotation tool of the disclosed embodiments provides utilities for the annotator to mark the tumor region out in the scan images. In one embodiment, the output can be a binarized mask having a same size as the scan image200, with a “1” indicating a tumor region and a “0” indicating a normal, or non-tumor region.

As shown inFIG.2, a contour206identifies the boundary or edges of the object of interest204. As will be generally understood, identification of the boundaries or edges of the object of interest204is necessary for accurate assessment and labelling purposes.

In certain imaging processes, the boundaries or edges of the object of interest204will be marked with a line or other suitable marker. For the purposes of the description herein, this marking or definition of the boundary or edges will be referred to as “contour206.”

As will be described further herein, control points or markers can be commonly used to identify and annotate the contour206of the object of interest204. When a contour206is not accurately marked by a line or control point, the positions of the control points can be adjusted to more accurately define the contour206. The aspects of the disclosed embodiments are directed to the automatic adjustment of control point positions and definition of the contour206.

Referring again toFIG.1, in one embodiment the apparatus or system, includes at least one processor102. The processor102is configured to receive image data108as an input. As generally described herein, the image data108is medical image data such as the image200ofFIG.2.

In one embodiment, the image data108is received from an imaging system110. Although the processor102and apparatus100ofFIG.1are shown as external to the imaging system110, the aspects of the disclosed embodiments are not so limited. In alternate embodiments, the apparatus100and processor102can be a component of the imaging system110.

In one embodiment, the processor102is configured to initially partition the input image data108based on a segmentation mask. One example of a segmentation mask is shown inFIG.3. In this example, the segmented image300includes a dark, background region302, and a light or white region304. The white region304generally encompasses the object of interest204shown inFIG.2. As will be described further in conjunction withFIG.4A, the area304will be partitioned and control points generated. The segmentation process described herein generally comprises any suitable image segmentation process.

An example of partitioned input image data400is shown inFIG.4A. As can be seen from this example, the partitioning of the input image data108ofFIG.2generally results in a partitioned image400with a series of grid-like lines402. In the example ofFIG.4A, the grid-like lines402are non-linear and generated by a partitioning algorithm. The partitioning algorithm is generally configured to generate the grid-like lines402based on pixel features of the input image108, shown as scanned image404inFIG.4A. The white or lighter colored area406in the example ofFIG.4Ais the object of interest. Other considerations in the generation of the grid-like lines402can include, but are not limited to, the geometric shape and size of the object of interest406.

Referring also toFIG.4B, in one embodiment, an initial contour410identifies a boundary or edge region of the object of interest406. As shown inFIG.4B, one or more control points or markers412are generated and used to identify or mark the contour410based on the partitioning. The control points412are generally configured to provide a visual identification of the contour410to a user.

The number of control points412shown inFIG.4Bis merely exemplary. In alternate embodiments, the number of control points412can be any suitable number. For example, in one embodiment the number of control points412is set by the user.

In one embodiment, the partitioned input image400with the control points412can be presented on a display106of the apparatus100. In one embodiment, the display106can be part of a user interface of the apparatus100that allows the annotator to interact and annotate the image400as is generally described herein. In one embodiment, the apparatus100can include suitable tools, such a joystick, touch pen, mouse or other cursor device that will allow the annotator to reposition the control points412as is described herein. The aspects of the disclosed embodiments are configured to allow the annotator to click on points in the image400, draw lines on the image400, as well as drag or move points and lines on or in the image400. In one embodiment, the display or user interface106comprises a touch screen or touch sensitive device that allows the annotator to interact with the image400as is generally described herein.

For example, in one embodiment, the input image data108comprises CT/MRI scan images. The input image data108to be annotated will typically be in the form of a sequence of grayscale images. When the user or annotator starts to annotate the scanned images, these scanned images are loaded and shown to the annotator via a computer screen or user interface104, as is shown in the example ofFIG.4A. The user can select a suitable tool from a utility or toolbox provided by the apparatus100, and use the tool to annotate the image400as is generally described herein. For example, in one embodiment, the apparatus100can provide a menu from which the annotator can select a suitable tool or utility to annotate the image400, including the grid-lines402, control points412and contour410.

The processor102is configured to set the initial contour410and the control points412based on the partitioning. The partitioning process uses a suitable algorithm to identify the edges of the object of interest406. The initial contour410and control points412are used to provide a visual demarcation of the edges as identified by the partitioning algorithm.

In one embodiment, the granularity or fineness of the partitioning shown inFIG.4Acan be varied. The closer together the grid lines are positioned, the finer the granularity of the partitioning. Such granularity can be used to provide more definition to the contour406. For example, finer granularity in the grid lines can enhance the detection of edges of the object(s) of interest by the partitioning algorithm. In one embodiment, the granularity or fineness of the partitioning can be set or adjusted by the user.

As shown in the example ofFIG.4A and4B, the processor102is configured to generate an initial contour410for the object of interest406. In the example ofFIG.4B, the initial contour410is identified by one or more control points412positioned on or near the grid lines402. The aspects of the disclosed embodiments are configured to differentiate between the light and dark areas and identify the edges of the object of interest406. The control points412are then disposed on or in conjunction with the grid lines that one or more of form or are closest to the edges, such as grid line402aand402b.

The control points412, also referred to as markers412, are generally configured to provide identification points along the edges or boundaries of the object of interest406. In some cases, manual adjustment of one or more of the control points412is required in order to more accurately identify the edges. For example, as illustrated inFIG.4B, one or more of the control points412, such as control points412aand412b, may not be accurately located with respect to the edges or boundary of the object of interest. In this example, control points412aand412bare associated with or connected by grid line402b. However, a more accurate placement or connection of the control points412aand412bmay be grid line402c. The aspects of the disclosed embodiments are configured to enable the annotator to manually reposition one or more of the control points412as well as redefine the initial contour410relative to the grid lines402.

FIG.5illustrates one example of an image510showing the manual repositioning of a control point412. In this example, the control point412ais manually repositioned from an initial position502to a next position504. In this manner, the control point412ais more closely associated with grid line402c. In this example, the grid line402may be more accurately associated with the edge of the object of interest406than the grid line402b. Although only one control point412ais shown as manually adjusted, the aspects of the disclosed embodiments are not so limited. In alternate embodiments, any suitable number of control points412can be adjusted.

FIG.6illustrates the automatic adjustment of adjacent or nearby control points and the contour in accordance with the aspects of the disclosed embodiments. As noted with respect toFIG.5, control point412awas manually adjusted from position502to position504. In accordance with the aspects of the disclosed embodiments, the positions of control points that are one or more of adjacent to or within a pre-determined distance from the manually adjusted control point are automatically adjusted and updated.

An “adjacent control point” as that term is used herein, generally refers to a control point412that is within a certain distance or range of the manually adjusted control point, which is control point412ain the example ofFIGS.5and6. Although the term “adjacent” is used herein, the aspects of the disclosed embodiments are intended to apply to any control point that is within the predetermined range or area of the manually adjusted control point. In one embodiment, the predetermined distance, range or area can be manually set or adjusted by the user/annotator.

In the example ofFIG.6, area602is defined as the area of control points adjacent to the manually adjusted control point412a. In this example, control point412bis automatically adjusted from its position504shown inFIG.5to a new position604as shown inFIG.6. The new position604in this example is associated with grid line402c.

In one embodiment, the adjustment of the control point412bis based on a degree of movement of the manually adjusted control point412a. For example, the determination of the new location604for the control point412bcan be proportional to the distance of the movement of the manually adjusted control point412a.

In one embodiment, the movement of the control point412bwill be to the next closest grid line, relative to the movement of the manually adjusted control point412a. In the example ofFIG.6, control point412bmoves from grid line402bto grid line402c, in a direction and distance relative to the direction and distance of the adjustment of control point412a.

As shown inFIG.6, in addition to the adjustment of control point412b, additional control points412cand412dcan be generated. The additional control points412cand412dare disposed on the grid lines connecting the respective control points to the repositioned control point412a. For example, control point412cis positioned on grid line402cconnecting control point412aand412c.

The automatic repositioning of the adjacent control points results in the generation or definition of an adjusted or new contour line, generally illustrated as contour line608inFIG.6. As shown in this example, the aspects of the initial contour lines410ofFIG.4Bare modified or changed in the area602ofFIG.6, relative to the manual adjustment of control point412a. In the example ofFIG.4B, the initial contour line410was associated with grid line402b. As shown inFIG.6, the updated contour line608is now associated with grid line402c.

FIG.7is a flow chart illustrating one embodiment of a process incorporating aspects of the disclosed embodiments. In one embodiment, image data is input702. This can include a sequence of images. A segmentation mask is generated704. A partition is then applied706. Control points are generated706on an initial contour relative to the object of interest.

In one embodiment, a manual adjustment of at least one control point is identified or detected708. Adjacent or nearby control points are identified and positions of the adjacent control points are automatically adjusted710relative to the movement of the manually adjusted control point. Updated control points and contour are generated712for visualization by the user and/or annotator. This can include, for example, the display of an image with the updated control points and contour.

In one embodiment, the apparatus100shown inFIG.1, generally comprises a computing device. The computing device can comprise or include any suitable computer or computing arrangement.

In one embodiment, the processor102comprises a hardware processor. Although only one processor102is generally described herein, the aspects of the disclosed embodiments are not so limited. In alternate embodiments, the apparatus100can include any suitable number of processors102.

Referring again toFIG.1, the apparatus100generally includes suitable logic, circuitry, interfaces and/or code that is configured to receive the input image data108and process the image data108as is generally described herein. In some embodiments, the processor102can be configured to receive a sequence of image frames (e.g., one or more video) of the patient from the imaging system110. The imaging system110will generally include suitable image capture devices or sensors.

The processor102generally includes suitable logic, circuitry, interfaces and/or code that is configured to process the image input data108as is generally described herein. The processor102is configured to respond to and process instructions that drive the apparatus100. Examples of the processor102include, but are not limited to, a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or any other type of processing circuit. Optionally, the processor102may be one or more individual processors, processing devices and various elements associated with a processing device that may be shared by other processing devices. Additionally, the one or more individual processors, processing devices and elements are arranged in various architectures for responding to and processing the instructions that drive the system100. The apparatus100can include any suitable components or devices that are needed to carry out the processes described herein, such as a memory or storage, for example.

In one embodiment, the apparatus100can comprise or be part of a standalone computing device, in communication with, or part of, the imaging system110. In one embodiment, the apparatus100will include or be connected to the machine learning models needed to carry out the aspects of the disclosed embodiments described herein.

In the example ofFIG.1, the apparatus100also includes or is communicatively coupled to a memory104. Although not shown, the apparatus100could be communicatively coupled to network or network interface to enable communication with the components and devices of the apparatus100and image system110.

The memory104may comprise suitable logic, circuitry, interfaces, and/or code that may be configured to store instructions executable by the processor102. The memory104is further configured to store the image data108. The memory104may be further configured to store operating systems and associated applications of the processor102. Examples of implementation of the memory104may include, but are not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Hard Disk Drive (HDD), Flash memory, and/or a Secure Digital (SD) card. A computer readable storage medium of a computer program product for providing a non-transient memory may include, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.

The aspects of the disclosed embodiments are directed to an interactive contour refinement process for efficient data annotation. The positions of one or more control points on a contour are automatically adjusted relative to a manual adjustment of another control point on the contour. Implementation of the aspects of the disclosed embodiments can be in the form of a portal or software installed in a computer that can read/load/store sensor data (e.g., CT/MRI scans), display images and provide tools to the annotator(user) for them to annotate images. The output can be a binarized mask generated by the control points and contour lines.

Various embodiments and variants disclosed above, with respect to the aforementioned system100, apply mutatis mutandis to the method. The method described herein is computationally efficient and does not cause processing burden on the processor102.

Modifications to embodiments of the aspects of the disclosed embodiments described in the foregoing are possible without departing from the scope of the aspects of the disclosed embodiments as defined by the accompanying claims. Expressions such as “including”, “comprising”, “incorporating”, “have”, “is” used to describe and claim the aspects of the disclosed embodiments are intended to be construed in a non-exclusive manner, namely allowing for items, components or elements not explicitly described also to be present. Reference to the singular is also to be construed to relate to the plural.

Thus, while there have been shown, described and pointed out, fundamental novel features of the invention as applied to the exemplary embodiments thereof, it will be understood that various omissions, substitutions and changes in the form and details of devices and methods illustrated, and in their operation, may be made by those skilled in the art without departing from the spirit and scope of the presently disclosed invention. Further, it is expressly intended that all combinations of those elements, which perform substantially the same function in substantially the same way to achieve the same results, are within the scope of the invention. Moreover, it should be recognized that structures and/or elements shown and/or described in connection with any disclosed form or embodiment of the invention may be incorporated in any other disclosed or described or suggested form or embodiment as a general matter of design choice. It is the intention, therefore, to be limited only as indicated by the scope of the claims appended hereto.