METHOD AND SYSTEM FOR GENERATING LABEL OF MEDICAL IMAGE

A medical image processing method includes following steps. A first medical image about a first patient under a first examination condition is obtained. A second medical image about the first patient under a second examination condition is obtained. A first label corresponding to the first medical image is collected. The first label marks a lesion within the first medical image. A transformation function between the first medical image and the second medical image is calculated by aligning the first medical image with the second medical image. The transformation function is applied to convert the first label into a second label corresponding to the second medical image.

BACKGROUND

Field of Invention

The disclosure relates to a medical image processing system and medical image processing method. More particularly, the disclosure relates to a medical image processing system and medical image processing method capable of converting image labels between medical examinations.

Description of Related Art

Several medical imaging technologies are widely used in diagnosing diseases or examining health conditions on patients. For example, X-ray imaging, computed tomography (CT) imaging and a magnetic resonance imaging (MRI) can provide critical information while diagnosing a cancer, a fracture, an internal bleeding and other symptoms.

Normally, it requires an experienced doctor or an expert to look into outcome images generated by these medical imaging technologies, and to determine whether the outcome images are normal or abnormal. Recently, some artificial intelligence (AI) systems are developed to classify examination outcomes of the medical images. In order to train an artificial intelligence agent for classifying medical images, it requires a lot of medical images and label data corresponding to these medical images. Producing correct label data on these medical images is a professional task, which is time-consuming and must be executed by personnel with medical expertise. Therefore, the label data on medical images are always recognized as scarce resources.

SUMMARY

An embodiment of the disclosure provides a medical image processing method, which includes following steps. A first medical image about a first patient under a first examination condition is obtained. A second medical image about the first patient under a second examination condition is obtained. A first label corresponding to the first medical image is collected. The first label marks a lesion within the first medical image. A transformation function between the first medical image and the second medical image is calculated by aligning the first medical image with the second medical image. The transformation function is applied to convert the first label into a second label corresponding to the second medical image.

An embodiment of the disclosure provides a medical image processing system, which includes a memory, an interface and a processor. The memory is configured to store a first medical image and a second medical image. The first medical image is captured under a first examination condition about a first patient. The second medical image is captured under a second examination condition about the first patient. The second examination condition is different from the first examination condition. The interface is configured to collect a first label corresponding to the first medical image. The first label is configured to mark a lesion within the first medical image. The processor is coupled with the interface and the memory. The processor is configured to calculate a transformation function between the first medical image and the second medical image by aligning the first medical image with the second medical image. The processor is configured to apply the transformation function for converting the first label into a second label corresponding to the second medical image.

DETAILED DESCRIPTION

Reference is made toFIG.1, which is a block diagram illustrating a medical image processing system100according to some embodiments of this disclosure. In some embodiments, the medical image processing system100is configured to convert labels between different medical images captured under different examination conditions.

As shown inFIG.1, the medical image processing system100includes a memory120, a processor140and an interface160. The memory120is configured to store some data (e.g., medical images) and computer-executable instructions. In some embodiments, the memory120can include a dynamic memory, a static memory, a hard-drive and/or a flash memory. The interface160is configured to receive input data (e.g., an input medical image, an instruction, a voice command or a keyboard input) and/or display output content. In some embodiments, the interface160may include a keyboard, a displayer, a touch panel, a microphone, a network transceiver, a speaker, etc. The processor140is coupled with the memory120and the interface160. In some embodiments, the processor140can include a central processing unit (CPU), a graphic processing unit (GPU), a tensor processing unit (TPU), an application specific integrated circuit (ASIC) or any equivalent processing circuit.

In the medical field, in order to examine a health condition of a patient, the same patient may undergo different medical imaging examinations during a visit to the hospital. These different medical imaging examinations may generate different medical images about the same patient. As shown inFIG.1, the memory120is configured to store a first medical image IMG1and a second medical image IMG2. In some embodiments, the first medical image IMG1is captured under a first examination condition about a first patient, and the second medical image IMG2is captured under a second examination condition about the first patient. In other words, the first medical image IMG1and the second medical image IMG2are about the same patient under different examination conditions. Reference is further made toFIG.2AandFIG.2B.FIG.2AandFIG.2Bare schematic diagrams illustrating an exemplary pair of the first medical image IMG1and the second medical image IMG2according to some embodiments of the disclosure.

In some embodiments, the first medical image IMG1under the first examination condition is captured on the first patient with dosing a contrast medium. As shown inFIG.2A, the first medical image IMG1is a computed tomography perfusion image about a head portion of the first patient in the example. In practical applications, the computed tomography perfusion image is a three-dimensional image. For brevity, the computed tomography perfusion image shown inFIG.2Ais one sectional view of the three-dimensional computed tomography perfusion image for demonstration.

On the other hand, the second medical image IMG2under the second examination condition is captured on the first patient without dosing the contrast medium. As shown inFIG.2B, the second medical image IMG2is a non-contrast computed tomography image about the head portion of the first patient in the example. In practical applications, the non-contrast computed tomography image is a three-dimensional image. For brevity, the non-contrast computed tomography image shown inFIG.2Bis one sectional view of the three-dimensional non-contrast computed tomography image for demonstration.

Based on the computed tomography perfusion image with dosing the contrast medium, it is relatively easier for medical personnel (e.g., a doctor, a radiologist or a medical expert) to locate a lesion within the first medical image IMG1. For example, with dosing the contrast medium, it is easier to locate a blood leakage/blockage region within the first medical image IMG1by the medical personnel, and the blood leakage/blockage region may lead to an ischemic stroke. Reference is further made toFIG.3, which is a schematic diagram illustrating a first label LB1corresponding to the first medical image IMG1to some embodiments of the disclosure.

In some embodiments, the medical personnel can observe the first medical image IMG1on the interface160and manually assign the first label LB1on a specific region of the first medical image IMG1based on their medical knowledges. The first label LB1, manually inputted by the medical personnel about the lesion in the first medical image IMG1, can be collected by the interface160of the medical image processing system100and transmitted to the processor140. In some other embodiments, the first label LB1can be extracted from existed medical records and imported through the interface160to the processor140.

Based on the non-contrast computed tomography image without dosing the contrast medium, it is relatively harder for the medical personnel to locate a lesion within the second medical image IMG2. Because the blood leakage/blockage region is hardly visible in the second medical image IMG2, it may consume a lot of time to locate a blood leakage/blockage region within the second medical image IMG2. Producing label data manually on the second medical image IMG2takes a lot of time and costs a lot of money. Therefore, it is hard to find any reliable label data marking the lesion on the second medical image IMG2.

In practical applications, equipment and procedures for capturing the computed tomography perfusion image (e.g., the first medical image IMG1as shown inFIG.2A) are relatively complicated and expensive, in comparison with equipment and procedures for capturing the non-contrast computed tomography image (e.g., the second medical image IMG2as shown inFIG.2B). Some small hospitals do not have equipment for capturing computed tomography perfusion image. On the other hand, the equipment and procedures for capturing the non-contrast computed tomography image (e.g., the second medical image IMG2as shown inFIG.2B) are relatively accessible and common in hospitals. In some cases, it is faster and cheaper for the patient to perform a non-contrast computed tomography examination compared with a computed tomography perfusion examination. It is desired to be able to detect the lesion based on the non-contrast computed tomography image (i.e., the second medical image IMG2as shown inFIG.2B).

In some embodiments, the processor140of the medical image processing system100is able to calculate a transformation function between the first medical image IMG1and the second medical image IMG2, and utilize the transformation function to convert the first label LB1corresponding to the first medical image IMG1into a second label LB2corresponding to the second medical image IMG2. Further details about how to convert the first label LB1into the second label LB2will be discussed in following paragraphs.

Reference is further made toFIG.4, which is a flowchart diagram illustrating a medical image processing method200according to some embodiments of the disclosure. In some embodiments, the medical image processing method200inFIG.4can be executed by the medical image processing system100as shown inFIG.1.

As shown inFIG.1andFIG.4, step S210is executed by the medical image processing system100for obtaining the first medical image IMG1(referring toFIG.2A) about the first patient under the first examination condition. In embodiments shown inFIG.2A, the first medical image IMG1is obtained from an external examination scanner (not shown in figures) performing a computed tomography perfusion scanning.

As shown inFIG.1andFIG.4, step S220is executed by the medical image processing system100for obtaining the second medical image IMG2(referring toFIG.2B) about the first patient under the second examination condition. In embodiments shown inFIG.2B, the second medical image IMG2is obtained from an external examination scanner (not shown in figures) performing a non-contrast computed tomography scanning.

As shown inFIG.1andFIG.4, step S230is executed by the interface160for collecting the first label LB1(referring toFIG.3) corresponding to the first medical image IMG1, and the first label LB1marks the lesion within the first medical image IMG1. The first label LB1can be manually inputted by a doctor, a radiologist or a medical expert through the interface160.

In some embodiments, before calculating the transformation function between the first medical image IMG1and the second medical image IMG2, step S240is executed by the processor140for pre-processing the first medical image IMG1and the second medical image IMG2. Reference is further made toFIG.5.FIG.5is a schematic diagram illustrating pre-processing on the first medical image IMG1and the second medical image IMG2in step S240according to some embodiments in the disclosure.

As shown inFIG.5, in some embodiments, the first medical image IMG1(i.e., the computed tomography perfusion image) may include several computed tomography perfusion subgraphs, which includes computed tomography perfusion subgraphs IMG1a, IMG1b, IMG1c, IMG1dand IMG1e. These computed tomography perfusion subgraphs IMG1a-IMG1eshow brain examination results under different perfusion modalities. In some embodiments, each of the computed tomography perfusion subgraphs IMG1a-IMG1eincludes a brain perfusion result and also a color index bar located on the right side.

As shown inFIG.5, step S240includes two sub-steps S241and S242. Step S241is executed by the processor140for pre-processing the first medical image IMG1for cancelling a first noise feature in the first medical image IMG1while maintaining a first target feature in the first medical image IMG1. In some embodiments, during step S241, the color index bars in the computed tomography perfusion subgraphs IMG1a-IMG1eare regarded as noise features and cancelled by the processor140. In some embodiments, the processor140can utilize a rectangle mask (e.g., with a width of one pixel and a height equal to IMG1a-IMG1e) to scan over different parts on these computed tomography perfusion subgraphs IMG1a-IMG1e. The processor140detects the pixel values located in the rectangle mask in different scanning rounds. When pixel values located in the rectangle mask in three adjacent scanning rounds are no image data (or all dark pixels), with colored data and no image data (or all dark pixels) in sequence, the processor140is able to detect the color index bar. The processor140can cancel the color index bar in the computed tomography perfusion subgraphs IMG1a-IMG1eby filling the color index bars with the background color. On the other hand, the brain perfusion results within the computed tomography perfusion subgraphs IMG1a-IMG1eare regarded as target features and maintained by the processor140.

Besides cancelling the color index bars, during step S241, the processor140further performs a union operation to combine the computed tomography perfusion subgraphs IMG1a-IMG1einto a processed first medical image IMG1p. The union operation is performed by applying a logical “OR” to combine the computed tomography perfusion subgraphs IMG1a-IMG1e. In this case, pixel data of the brain perfusion results in five computed tomography perfusion subgraphs IMG1a-IMG1eare integrated into the processed first medical image IMG1pas shown inFIG.5.

On the other hand, the non-contrast computed tomography image may include image data about skull, scalp and brain tissues. As shown inFIG.5, step S242is executed by the processor140for pre-processing the second medical image IMG2for cancelling a second noise feature in the second medical image IMG2while maintaining a second target feature in the second medical image IMG2. In this case, the image data about skull and the image data about scalp are regard as the noise features, and cancelled by the processor in step S242. In this case, the image data about brain tissues are regard as the target features and maintained by the processor140in step S242, so as to generate a processed second medical image IMG2pas shown inFIG.5.

As shown inFIG.4, step S250is executed by the processor140, to calculating a transformation function between the first medical image IMG1and the second medical image IMG2. Reference is further made toFIG.6, which is a schematic diagram illustrating operations in step S250for calculating the transformation function according to some embodiments in the disclosure. Step S250includes sub-steps S251to S255. As shown inFIG.6, in step S250, the processor140calculate the transformation function by aligning the processed first medical image IMG1pand the processed second medical image IMG2p.

Step S251is executed by the processor140for generating a first transformed medical image IMG1ptby rotating, shifting and/or re-sizing the processed first medical image IMG1paccording to variable parameters. For example, the variable parameters include rotating angles (θx1, θy1, θz1), shifting displacements (Dx1, Dy1, Dz1) and/or a re-sizing ratio (R1). The processor140is configured to rotate the processed first medical image IMG1paccording to the rotating angles (θx1, θy1, θz1), shift the processed first medical image IMG1paccording to the shifting displacements (Dx1, Dy1, Dz1) and/or re-size the processed first medical image IMG1paccording to the re-sizing ratio (R1), so as to generate the first transformed medical image IMG1pt.

Afterward, step S252is executed by the processor140to calculate a similarity score by comparing the first transformed medical image IMG1ptand the processed second medical image IMG2p. In this case, when the target features in the first transformed medical image IMG1ptare highly overlapped with the target features in the processed second medical image IMG2p, the similarity score will be higher. On the other hand, when the target features in the first transformed medical image IMG1ptare mismatched from the target features in the processed second medical image IMG2p, the similarity score will be lower. In some embodiments, the similarity score between the first transformed medical image IMG1ptand the processed second medical image IMG2pcan be calculated by a Mattes Mutual Information algorithm.

Step S253is executed by the processor140to check whether the similarity score satisfies a similarity threshold or not. The similarity threshold can be predetermined (e.g., 50%, 70% or 90%) or dynamically assigned (e.g., a relatively high similarity score in three minutes).

If the current similarity score is below the similarity threshold, step S254is executed by the processor140to update the variable parameters. For example, the processor140updates the rotating angles into (θx2, θy2, θz2), updates the shifting displacements into (Dx2, Dy2, Dz2) and/or updates the re-sizing ratio into (R2). In some embodiments, updating the variable parameters in step S254can be performed based on a Gradient Descend algorithm. Step S251and step S252is executed again based on the updated variable parameters. In step S251, the processor140re-generates the first transformed medical image IMG1ptaccording to the update variable parameters, including the rotating angles (θx2, θy2, θz2), the shifting displacements (Dx2, Dy2, Dz2) and/or updates the re-sizing ratio into (R2). In step S252, the processor140re-calculates the similarity score between the first transformed medical image IMG1ptand the processed second medical image IMG2p.

Steps S251to S254will be repeated until the similarity score exceeds the similarity threshold. By repeating the steps S251to S254, the target features in the first transformed medical image IMG1ptare rotated, shifted, re-sized to be overlapped with the target features in the processed second medical image IMG2p. If the similarity score is high, it means that the locations and sizes of target features shown in the first transformed medical image IMG1ptare highly overlapped with the target features shown in the processed second medical image IMG2p. If the similarity score exceeds the similarity threshold, step S255is executed by the processor140to recording the current variable parameters as the transformation function.

It is noticed that, the first medical image IMG1and the second medical image IMG2are images about the same patient captured under different examination conditions. Therefore, it may include a rotation difference, a displacement, a sizing difference while capturing the first medical image IMG1and the second medical image IMG2. If the first label LB1on the first medical image IMG1is directly duplicated to the same position onto the second medical image IMG2, the duplicated label may be placed at a wrong position away from the lesion in the second medical image IMG2. In this case, the transformation function is able to calibrate a rotation difference, a displacement, a sizing difference between the first medical image IMG1and the second medical image IMG2.

In other embodiments of step S250, it is also feasible to generate the transformation function by inverting an inverse transform function. The inverse transform function can be generated through processes similar to steps S251through255, but with the roles of the processed first medical image IMG1pand the processed second medical image IMG2pbeing swapped in the flowchart ofFIG.6. For example, the processor140can generate a second transformed medical image by rotating, shifting and/or re-sizing the processed second medical image IMG2paccording to variable parameters. Then, the similarity score is calculated by comparing the target features in second transformed medical image and the target features in the processed first medical image IMG1p. The processor140updates the variable parameters and regenerates the second transformed medical image according to the update variable parameters, until the similarity score exceeds the similarity threshold. As the similarity score exceeds the similarity threshold, the processor140records the current variable parameters as the inverse transformation function. In this situation, the processor executes an additional step to invert the inverse transformation function in order to generate the transformation function.

Reference is further made toFIG.7.FIG.7is a schematic diagram illustrating the transformation of labels between the first medical image IMG1and the second medical image IMG2according to some embodiments in the disclosure. As shown inFIG.1,FIG.4andFIG.7, step S260is executed by the processor140for applying the transformation function to convert the first label LB1into a second label LB2corresponding to the second medical image IMG2. As shown inFIG.7, an orientation, a location and a size of the second label LB2on the second medical image IMG2is adjusted by the transformation function and are different from the first label LB1on the first medical image IMG1. In this case shown inFIG.7, the second label LB2is slightly larger than the first label LB1, and the second label LB2is moved toward a top side of the second medical image IMG2compared to the first label LB1.

The relationships between the first label LB1and the second label LB2are not limited to the example shown inFIG.7. The relationships between the first label LB1and the second label LB2are decided by the transformation function calculated in step S250.

It is noticed that, in aforesaid embodiments, the first medical image IMG1is demonstrated as the computed tomography perfusion image with dosing the contrast medium and the second medical image IMG2is demonstrated as the non-contrast computed tomography image without dosing the contrast medium. This disclosure is not limited thereto. In some other embodiments, the first medical image IMG1under the first examination condition and the second medical image IMG2under the second examination condition are captured by different examination scanners. In an example, the first medical image IMG1can be a computed tomography (CT) image captured by a computed tomography scanner, and the second medical image IMG2can be an X-ray image captured by an X-ray scanner. In another example, the first medical image IMG1can be a magnetic resonance imaging (MRI) image captured by a magnetic resonance imaging scanner, and the second medical image IMG2can be the computed tomography image captured by the computed tomography scanner. Similarly to embodiments shown inFIG.4toFIG.7, the processor140of the medical image processing system100is able to convert the first label LB1(manually assigned by the medical personnel) on the first medical image IMG1into the second label LB2on the second medical image IMG2.

Based on the medical image processing system100and the medical image processing method200shown in aforesaid embodiments inFIG.1toFIG.7, the medical image processing system100and the medical image processing method200are able convert labels between different medical images captured under different examination conditions, and produce a reliable label on the second medical image IMG2under the second examination condition. Even though it is hard to collect/produce a manual label of the second medical image IMG2by the medical personnel, the medical image processing system100and the medical image processing method200can convert the second label LB2from the first label LB1, which is relatively easier to label by the medical personnel. In this case, the medical image processing system100and the medical image processing method200are able to solve a scarcity issue of the second medical label LB2on the second medical image IMG2. In some embodiments, the medical image processing system100is able to repeat the medical image processing method200shown inFIG.4to produce multiple second labels LB2corresponding to different second medical images IMG2(e.g., the non-contrast computed tomography image) captured under the second examination condition. In some embodiments, the second labels LB2and the second medical images IMG2can be utilized to train a neural network model for predicting a lesion based on non-contrast computed tomography images.

Reference is further made toFIG.8andFIG.9.FIG.8is a block diagram illustrating a medical image processing system300according to some embodiments of this disclosure.FIG.9is a flowchart diagram illustrating a medical image processing method400according to some embodiments of the disclosure. As shown inFIG.8, the medical image processing system300includes a memory320, a processor340and an interface360.

Similar to aforesaid embodiments inFIG.1toFIG.7, the processor340of the medical image processing system300inFIG.8is able to executed steps S410to S460shown inFIG.9for calculating a transformation function between the first medical image IMG1and the second medical image IMG2, and utilizing the transformation function to convert the first label LB1corresponding to the first medical image IMG1into a second label LB2corresponding to the second medical image IMG2. The details about the calculating the transformation function and converting the first label LB1into the second label LB2in steps S410to S460can be referred the aforesaid embodiments in steps S210to S260, and are not repeated here.

In some embodiments, the processor340is configured to repeat the steps S410to S460inFIG.9multiple times to produce multiple second labels LB2corresponding to different second medical images IMG2(e.g., the non-contrast computed tomography image) captured under the second examination condition. The second medical images IMG2and corresponding second labels LB2are utilized as training data. As shown inFIG.8andFIG.9, step S470is executed by the processor340for training the neural network model342according to the second medical images IMG2and the second labels LB2.

The neural network model342can be implemented by a convolutional neural network for lesion classification or lesion detection. The second medical images IMG2and corresponding second labels are utilized as ground truth to train the convolutional neural network. When a prediction label generated by the neural network model342matches with the second label LB2, a reward signal can be provided to the neural network model342. When a prediction label generated by the neural network model342fails to match with the second label LB2, a punish signal can provided to the neural network model342.

When the training is complete, the neural network model342is capable of predicting a lesion label from a medical image captured under the second examination condition. For example, the neural network model342is capable of predicting a lesion label based on a non-contrast computed tomography image.

As shown inFIG.8andFIG.9, in step S480, the interface360is configured to receive a third medical image IMG3captured the second examination condition. For example, the third medical image IMG3can be a non-contrast computed tomography image about a second patient. The second patient can be different from the first patient corresponding to the first medical image IMG1and the second medical IMG2.

As shown inFIG.8andFIG.9, step S490is executed by the processor340, and the processor340utilizes the neural network model342to generate a prediction label LBp attached on the third medical image IMG3. The prediction label LBp and the third medical image IMG3can be displayed on the interface360, such that the second patient or the medical personnel can acknowledge the prediction label LBp generated by the neural network model342.

In practical applications, the equipment and procedures for capturing the non-contrast computed tomography image (e.g., the second medical image IMG2as shown inFIG.2B) are relatively accessible and common in hospitals. In some cases, it is faster and cheaper for the patient to perform a non-contrast computed tomography examination compared with a computed tomography perfusion examination. The neural network model342of the medical image processing system300is able to generate the prediction label LBp based on the non-contrast computed tomography image. It is beneficial for the patient and the hospital to diagnosis the second patient based on a faster and cheaper medical examination.