Patent ID: 12217852

DETAILED DESCRIPTION

In the following, an embodiment of the present disclosure will be described with reference to the drawings.FIG.1is a hardware configuration diagram showing an outline of a diagnosis support system to which an image processing device according to the embodiment of the present disclosure is applied. As shown inFIG.1, in the diagnosis support system, an image processing device1according to the present embodiment, a modality2, and an image storage server3are connected in a communicable state via a network4.

The modality2is an apparatus that images a diagnosis target site of a subject to generate a three-dimensional image showing the diagnosis target site, and specifically, is a CT apparatus, an MRI apparatus, a positron emission tomography (PET) apparatus, and the like. The three-dimensional image including of a plurality of slice images generated by the modality2is transmitted to and stored in the image storage server3. Note that in the present embodiment, it is assumed that the modality2includes a CT apparatus2A and an MRI apparatus2B. In addition, it is assumed that the CT apparatus2A is a dual energy CT apparatus or a photon counting CT apparatus capable of imaging a subject by using radiation having different energy distributions. It is assumed that the MRI apparatus2B can derive a tissue eigenvalue of MRI by imaging the subject, which is described in JP2015-525604A and Rapid magnetic resonance quantification on the brain: Optimization for clinical usage, Magn Reson Med 2008. 60(2), 320-329, for example.

The image storage server3is a computer that stores and manages various data, and comprises a large capacity external storage device and software for database management. The image storage server3performs communication with other devices via the wired or wireless network4to transmit and receive image data and the like. Specifically, the image storage server3acquires various data including the image data of a medical image generated by the modality2via the network, and stores and manages the image data in a recording medium, such as the large capacity external storage device. Note that a storage format of the image data and the communication between the devices via the network4are based on a protocol, such as digital imaging and communication in medicine (DICOM). In addition, in the present embodiment, the image storage server3also stores and manages a plurality of teacher data to be described below.

The image processing device1according to the present embodiment is a computer in which an image processing program and a learning program according to the present embodiment is installed. The computer may be a workstation or a personal computer directly operated by a doctor who makes a diagnosis, or a server computer connected to the workstation or the personal computer via the network. Alternatively, the image processing program and the learning program are stored in a storage device of the server computer connected to the network or a network storage in a state of being accessible from the outside, and are downloaded and installed in the computer used by the doctor in response to a request. Alternatively, the image processing program and the learning program are distributed in a state of being recorded on a recording medium, such as a digital versatile disc (DVD) or a compact disc read only memory (CD-ROM), and are installed in the computer from the recording medium.

FIG.2is a diagram showing a schematic configuration of the image processing device realized by installing the image processing program and the learning program in the computer. As shown inFIG.2, the image processing device1comprises a central processing unit (CPU)11, a memory12, and a storage13, as a configuration of a standard workstation. In addition, the image processing device1is connected with a display unit14, such as a liquid crystal display, and an input unit15, such as a keyboard or a mouse.

The storage13is configured by a hard disk drive or the like, and stores the CT images, which is a processing target, a plurality of teacher data, and various pieces of information including information necessary for processing, which are acquired from the image storage server3via the network4.

In addition, the image processing program and the learning program are stored in the memory12. The image processing program defines, as processing to be executed by the CPU11, image acquisition processing of acquiring a target medical image having the representation format different from the MRI image, tissue eigenvalue derivation processing of deriving the tissue eigenvalue of MRI for the target medical image, MRI image derivation processing of deriving the MRI image of a desired representation format by using the tissue eigenvalue, display control processing of displaying the derived MRI image on the display unit14. The learning program defines, as processing to be executed by the CPU11, learning processing of constructing a derivation model by performing machine learning using the plurality of teacher data to, in a case in which at least one medical image having the representation format different from the MRI image is input, output the tissue eigenvalue of MRI for the medical image.

Moreover, the CPU11executes the processing according to the image processing program and the learning program, so that the computer functions as an image acquisition unit21, a tissue eigenvalue derivation unit22, an MRI image derivation unit23, a display control unit24, and a learning unit25.

The image acquisition unit21acquires the target medical image having the representation format different from the MRI image from the image storage server3via an interface (not shown) connected to the network. In addition, the plurality of teacher data used for learning are acquired from the image storage server3. Note that in the present embodiment, the target medical image is the CT image acquired by the CT apparatus2A. In addition, in the present embodiment, the CT apparatus2A is the dual energy CT apparatus or the photon counting CT apparatus capable of imaging the subject by using radiation having different energy distribution. Therefore, in the present embodiment, it is assumed that one or more CT images Ci (i=1 to n) acquired by imaging the same site of the same subject are used as the target medical image.

The tissue eigenvalue derivation unit22has a derivation model30constructed by machine learning to, in a case in which the CT image is input, output the tissue eigenvalue of MRI for the input CT image. Moreover, the tissue eigenvalue derivation unit22derives the tissue eigenvalue by inputting the target CT image Ci into the derivation model30and outputting the tissue eigenvalue of MRI for the CT image Ci from the derivation model30. In the present embodiment, it is assumed that a T1 value, a T2 value, and a PD value are derived as the tissue eigenvalues.

In the following, machine learning for constructing the derivation model30will be described. The learning unit25performs machine learning for constructing the derivation model30. The learning unit25constructs the derivation model30by performing machine learning using the teacher data including a combination of a CT image for learning and a tissue eigenvalue for learning. In the present embodiment, it is assumed that the derivation model30is constructed by machine learning a convolutional neural network (hereinafter, referred to as CNN), which is one of a multi neural network in which a plurality of processing layers are hierarchically connected and deep learning is performed.

The CNN includes a plurality of convolutional layers and pooling layers. The convolutional layer performs convolution processing using various kernels on the input image, and outputs a feature amount map including feature amount data obtained by the convolution processing. The kernel has an n×n pixel size (for example, n=3), and a weight is set for each element. Specifically, the weight, such as a differential filter that emphasizes the edge of the input image, are set. The convolutional layer applies the kernel to the entire input image or the feature amount map output from the processing layer in the previous stage while shifting an attention pixel of the kernel. Further, the convolutional layer applies an activation function, such as a sigmoid function, to a convolved value, and outputs the feature amount map.

The pooling layer reduces an amount of data in the feature amount map by pooling the feature amount map output by the convolutional layer, and outputs the feature amount map with the reduced amount of data.

Moreover, by repeating the outputting and pooling of the feature amount map in each processing layer, the tissue eigenvalue for each pixel of the input CT image is output from the final layer of the CNN.

FIG.3is a conceptual diagram of machine learning performed in the present embodiment. Note that although machine learning using two CT images will be described here, one CT image may be used or three or more CT images may be used.

First, the teacher data will be described. In the present embodiment, teacher data40includes a combination of CT images for learning41A and41B having different energy distributions, which are acquired by irradiating the subject with the radiation having different energy distributions in the CT apparatus2A, and the tissue eigenvalue for learning (that is, the T1 value, T2 value, and the PD value) Us acquired by imaging the same subject as the subject for which the CT images for learning41A and41B in the MRI apparatus2B and acquired by using the method described in JP2019-005557A, JP2015-525604A, and Rapid magnetic resonance quantification on the brain: Optimization for clinical usage, Magn Reson Med 2008. 60(2), 320-329.

As shown inFIG.3, the learning unit25inputs the CT images for learning41A and41B included in the teacher data to the CNN, and outputs a tissue eigenvalue Ux for the CT images for learning41A and41B from the CNN. The learning unit25derives a loss L0 based on a difference between the output tissue eigenvalue Ux and the tissue eigenvalue for learning Us. The loss L0 is the difference between the T1 value, T2 value, and PD value of the output tissue eigenvalue Ux and the T1 value, T2 value, and PD value of the tissue eigenvalue for learning Us.

Moreover, the learning unit25learns the CNN by using a large number of teacher data such that the loss L0 is equal to or less than a predetermined threshold value. Specifically, learning of the CNN is performed by deriving the number of convolutional layers and the number of pooling layers, which configure the CNN, and a coefficient of the kernel and magnitude of the kernel in the convolutional layer, such that the loss L0 is equal to or less than the predetermined threshold value. As a result, in a case in which the CT image Ci is input to the trained CNN, the CNN outputs the tissue eigenvalue for each pixel of the CT image Ci. Note that the learning unit25may perform learning a predetermined number of times instead of learning such that the loss L0 is equal to or less than the predetermined threshold value.

As described above, the learning unit25performs machine learning of the CNN, so that the derivation model30that outputs the tissue eigenvalue for each pixel of the CT image Ci in a case in which the CT image Ci is input is constructed.

Returning toFIG.2, the MRI image derivation unit23derives the MRI image by using the tissue eigenvalue derived by the tissue eigenvalue derivation unit22. Here, the MRI image has various representation formats, such as a T1-weighted image, a T2-weighted image, a fat suppression image, and a diffusion-weighted image, and the contrast, that is, the appearance differs depending on the representation format. For example, on the T1-weighted image, mostly, a fat tissue appears white, water, a humoral component, and a cyst appear black, and a tumor appears slightly black. In addition, on the T2-weighted image, water, a humoral component, and a cyst appear white, as well as the fat tissue.

These various representation formats of the MRI images can be derived by calculating the tissue eigenvalues, that is, the T1 value, the T2 value, and the PD value, by using a predetermined parameter. Specifically, the MRI image having a desired representation format can be generated by calculating the T1 value, the T2 value, and the PD value by using a predetermined arithmetic expression to which parameters, such as an inversion time TI, an echo time TE, and a repetition time TR, depending on the representation format are applied.

Here, in the present embodiment, a table and an arithmetic expression defining a relationship between the representation formats of various MRI images and various parameters are stored in the storage13. The MRI image derivation unit23reads out the parameters corresponding to the representation format of the MRI image previously input from the input unit15from the storage13. Moreover, an MRI image MO having the input representation format is derived by calculating the T1 value, the T2 value, and the PD value by using the read out parameters.

The display control unit24displays the MRI image MO derived by the MRI image derivation unit23on the display unit14.FIG.4is a diagram showing a display screen of the MRI image. As shown inFIG.4, on a display screen50, two CT images C1 and C2 used for generating the MRI image MO are also displayed together with the MRI image MO. As a result, it is possible to perform image interpretation with the feature of the representation format of each image by using two types of CT images C1 and C2, and the MRI image MO for the same patient.

Then, processing performed in the present embodiment will be described.FIG.5is a flowchart showing the processing performed in the present embodiment. Note that it is assumed that the CT image Ci, which is the processing target, is acquired from the image storage server3and stored in the storage13. In a case in which a processing start instruction is input from the input unit15, the image acquisition unit21acquires the CT image Ci, which is the processing target, from the storage13(step ST1). Then, the tissue eigenvalue derivation unit22derives the tissue eigenvalue of MRI for the CT image Ci by the derivation model30(step ST2). Moreover, the MRI image derivation unit23derives the MRI image MO having a desired representation format by using the tissue eigenvalue (step ST3). Further, the display control unit24displays the MRI image MO on the display unit14(step ST4), and terminates the processing.

As described above, in the present embodiment, the tissue eigenvalue of MRI for the CT image Ci is derived by inputting the CT image Ci having the representation format different from the MRI image. Here, by using the tissue eigenvalue, it is possible to generate the MRI image having any representation format. Therefore, even in a case in which the patient has contraindications to MRI, such as having a pacemaker implanted or having claustrophobia, it is possible to acquire the MRI image having a desired representation format for the patient. In addition, it is not necessary to prepare a conversion model for converting the image corresponding to each of the representation formats of the MRI image. Therefore, according to the present embodiment, it is possible to easily acquire the MRI image having a desired representation format for the patient.

Here, since a maintenance cost of the MRI apparatus is high, it is difficult to introduce the MRI apparatus in a small hospital, such as a private hospital. On the other hand, a maintenance cost of the CT apparatus is not so high, it can be easily introduced in a small hospital, such as a private hospital. In addition, a medical cost for the CT image are cheaper than the MRI image. Therefore, by using the CT image as the medical image, which is the processing target, it is possible to acquire the MRI image by using the image that can be acquired relatively easily and cheaply.

In addition, by using a plurality of the CT images having different representation formats, an amount of information for deriving the tissue eigenvalue can be increased, so that the tissue eigenvalue of MRI can be derived more accurately.

Note that in the embodiment described above, the tissue eigenvalue of MRI is derived from the CT image Ci, but the present disclosure is not limited to this. The tissue eigenvalue of MRI may be derived from other medical image (e.g. PET image) other than the CT image Ci. In this case, the derivation model30need only be constructed by performing learning of the CNN by using a combination of the medical image for learning of the other medical image (i.e. PET image), and the tissue eigenvalue for learning for the same subject as the subject for which the image is acquired, as the teacher data.

In addition, in the embodiment described above, the CNN is used as the derivation model30, but the present disclosure is not limited to this. In addition to the CNN, a support vector machine (SVM), a deep neural network (DNN), a recurrent neural network (RNN), and the like can be used.

In addition, in the embodiment described above, the image processing device1is assumed to include the learning unit25, but the present disclosure is not limited to this. The derivation model30may be constructed by a learning device separate from the image processing device1, and the constructed derivation model30may be applied to the tissue eigenvalue derivation unit22of the image processing device1.

In addition, in the embodiment described above, the image processing device1includes the MRI image derivation unit23and the display control unit24, but the present disclosure is not limited to this. The MRI image may be derived or the derived MRI image may be displayed by the MRI image derivation unit23provided in a device different from the image processing device1. In this case, the tissue eigenvalue derived by the image processing device1is stored in an external device, such as the image storage server3, is read out from the external device as needed, and is used for deriving the MRI image or the like.

In addition, in the embodiment described above, for example, various processors shown below can be used as the hardware structures of processing units that execute various pieces of processing, such as the image acquisition unit21, the tissue eigenvalue derivation unit22, the MRI image derivation unit23, the display control unit24, and the learning unit25. As described above, the various processors include, in addition to the CPU that is a general-purpose processor which executes software (program) and functions as various processing units, a programmable logic device (PLD) that is a processor whose circuit configuration can be changed after manufacture, such as a field programmable gate array (FPGA), and a dedicated electric circuit that is a processor having a circuit configuration which is designed for exclusive use in order to execute a specific processing, such as an application specific integrated circuit (ASIC).

One processing unit may be configured by one of these various processors, or may be a combination of two or more processors of the same type or different types (for example, a combination of a plurality of FPGAs or a combination of the CPU and the FPGA). In addition, a plurality of the processing units may be configured by one processor.

As an example of configuring the plurality of processing units by one processor, first, as represented by a computer, such as a client and a server, there is an aspect in which one processor is configured by a combination of one or more CPUs and software and this processor functions as a plurality of processing units. Second, as represented by a system on chip (SoC) or the like, there is an aspect of using a processor that realizes the function of the entire system including the plurality of processing units by one integrated circuit (IC) chip. As described above, as the hardware structure, various processing units are configured by one or more of various processors described above.

Moreover, as the hardware structures of these various processors, more specifically, it is possible to use an electric circuit (circuitry) in which circuit elements such as semiconductor elements are combined.