Patent ID: 12254553

DETAILED DESCRIPTION

In the following, an embodiment of the present disclosure will be described with reference to the drawings.FIG.1is a hardware configuration diagram showing an outline of a diagnosis support system to which a learning device and an image generation device according to the embodiment of the present disclosure are applied. As shown inFIG.1, in the diagnosis support system, the learning device and the image generation device (hereinafter, represented by an image processing device)1according to the present embodiment, a modality2, and an image storage server3are connected in a communicable state via a communication network4.

The modality2is an apparatus that images a site including a diagnosis target structure of a human as a subject to generate a three-dimensional image representing the diagnosis target site, and specifically, is a CT apparatus, an MRI apparatus, a positron emission tomography (PET) apparatus, and the like. The three-dimensional image including of a plurality of slice images generated by the modality2is transmitted to and stored in the image storage server3. Note that in the present embodiment, it is assumed that the modality2includes a CT apparatus2A and an MRI apparatus2B. It is assumed that the CT apparatus2A and the MRI apparatus2B can inject a contrast medium into a blood vessel of a patient and perform contrast imaging for confirming the spread of the contrast medium. In addition, it is assumed that the MRI apparatus2B can generate an MRI image having any representation format, such as a T1-weighted image and a T2-weighted image.

Here, in a medical image, a representation format of the image differs in a case in which a type of image is different, such as the CT image and the MRI image. For example, even in a case in which a tissue of a human body included in the image is the same, the density differs between the CT image and the MRI image. In addition, even in a case in which the same MRI image is used, the representation format differs between the T1-weighted image and the T2-weighted image. Specifically, on the T1-weighted image, mostly, a fat tissue appears white, water, a humoral component, and a cyst appear black, and a tumor appears slightly black. In addition, on the T2-weighted image, water, a humoral component, and a cyst appear white, as well as the fat tissue. Therefore, the CT image, the T1-weighted image, and the T2-weighted image are images having different representation formats, respectively.

In addition, depending on the presence or absence of the contrast medium, the appearance of the image differs between the CT image acquired by performing imaging by using the contrast medium and a non-contrast CT image acquired by performing imaging without using the contrast medium. Therefore, the representation format of the image differs depending on the presence or absence of the contrast medium. In addition, in a case in which the image is captured by using the contrast medium, the spread of the contrast medium is changed with the elapse of time. Therefore, the representation format of the image differs depending on an elapsed time (contrast phase) after the contrast medium is injected. In addition, since the size, the density, and the like of an abnormal site are changed with the elapse of time, the appearance of the abnormal site, such as a lesion included in the same structural part of the same subject, is different. Therefore, the representation format of the image differs in the time before and after the current time.

The image storage server3is a computer that stores and manages various data, and comprises a large capacity external storage device and software for database management. The image storage server3performs communication with other devices via the wired or wireless communication network4to transmit and receive image data and the like. Specifically, the image storage server3acquires various data including the image data of a medical image generated by the modality2via the network, and stores and manages the image data in a recording medium, such as the large capacity external storage device. Note that a storage format of the image data and the communication between the devices via the communication network4are based on a protocol, such as digital imaging and communication in medicine (DICOM). In addition, in the present embodiment, the image storage server3also stores and manages a plurality of teacher data to be described below.

The image generation device1including the learning device according to the present embodiment is a computer in which an image generation program and a learning program according to the present embodiment are installed. The computer may be a workstation or a personal computer directly operated by a doctor who makes a diagnosis, or a server computer connected to the workstation or the personal computer via the network. Alternatively, the image generation program and the learning program are stored in a storage device of the server computer connected to the network or a network storage in a state of being accessible from the outside, and are downloaded and installed in the computer used by the doctor in response to a request. Alternatively, the image processing program and the learning program are distributed in a state of being recorded on a recording medium, such as a digital versatile disc (DVD) or a compact disc read only memory (CD-ROM), and are installed in the computer from the recording medium.

FIG.2is a diagram showing a schematic configuration of the image generation device realized by installing the image generation program and the learning program in the computer. As shown inFIG.2, the image generation device1comprises a central processing unit (CPU)11, a memory12, and a storage13, as a configuration of a standard workstation. In addition, the image generation device1is connected with a display unit14, such as a liquid crystal display, and an input unit15, such as a keyboard or a mouse.

The storage13is configured by a hard disk drive or the like, and stores various pieces of information including at least one target image, which is a generation target of the virtual image, the teacher data for learning the network configuring the image generation device as described below, and information necessary for processing, which are acquired from the image storage server3via the communication network4.

In addition, the image generation program and the learning program are stored in the memory12. The image generation program causes the CPU11to execute image generation processing of, in a case in which at least one target image for the subject, which includes a specific structure, having at least one representation format and target information representing a target representation format of the target image are input, deriving the virtual image having the target representation format from the target image. Specifically, the image generation program defines, as processing to be executed by the CPU11, information acquisition processing of acquiring at least one target image and the target information, subject model derivation processing of deriving the subject model representing the subject by deriving the feature amount from at least one target image and combining the feature amounts, latent variable derivation processing of deriving a latent variable obtained by dimensionally compressing a feature of the subject model according to the target information based on the target information and the subject model, virtual image derivation processing of deriving the virtual image having the target representation format based on the target information, the subject model, and the latent variable, and display control processing of displaying the virtual image on the display unit14.

As the processing to be executed by the CPU11, the learning program defines an information acquisition processing of acquiring various pieces of information including the teacher data for learning an image generation model included in the image generation device, and a learning processing of learning the image generation model.

Moreover, the CPU11executes the processing according to the image generation program and the learning program, so that the computer functions as an information acquisition unit20, a subject model derivation unit21, a latent variable derivation unit22, a virtual image derivation unit23, a display control unit24, and a learning unit25.

The information acquisition unit20acquires information ti representing the representation format for each of at least one target image Gi (i=1 to n) having at least one representation format and a target image Gi from the image storage server3via an interface (not shown) connected to the communication network4. In addition, the information acquisition unit20acquires target information A0representing the target representation format of the target image Gi by input from the input unit15or the like. In addition, the information acquisition unit20acquires a plurality of teacher images having different representation formats for the subject including the specific structure, and a plurality of teacher data including specific teacher information representing the specific representation format among the representation formats of the plurality of teacher images. Note that in a case in which a plurality of the target images Gi are used in one processing, the plurality of target images Gi input to the image generation device1are images including the same structure for the same patient and having different representation formats. In addition, the target information A0is information representing the target representation format of a virtual image V0to be generated. As the target representation format, for example, at least one of the type of image, the presence or absence of the contrast medium, the contrast phase, or the time before and after the current time can be used.

Here, the specific structure of the subject included in the target image and the teacher image is the same structure. For example, in a case in which the structure included in the target image is the liver, the structure included in the teacher image is also the liver. In the following, the specific structure will be described as being the liver.

The subject model derivation unit21derives a subject model M0representing the specific structure in the subject by deriving the feature amounts and combining the feature amounts based on the target image Gi and the information ti representing the representation format of the target image Gi. Therefore, the subject model derivation unit21includes a first network31that outputs the subject model M0representing the subject by deriving the feature amount of the input target image Gi in a case in which at least one target image Gi and the information ti representing the representation format of the target image Gi are input, further combining a plurality of feature amounts in a case in which the plurality of target images Gi are input and the plurality of feature amounts are derived. In the present embodiment, since the subject is the human body, the subject model M0can be said to be a human body model.

The latent variable derivation unit22derives a latent variable z1obtained by dimensionally compressing a feature of the subject model M0according to the target information A0based on the target information A0and the subject model M0. For this purpose, the latent variable derivation unit22includes a second network32that outputs the latent variable z1in a case in which the target information A0and the subject model M0are input. The latent variable z1will be described below.

The virtual image derivation unit23derives the virtual image V0having the target representation format represented by the target information A0based on the target information A0, the subject model M0, and the latent variable z1. For this purpose, the virtual image derivation unit23includes a third network33that derives the virtual image V0in a case in which the target information A0, the subject model M0, and the latent variable z1are input.

Note that, inFIG.3, the first network31, the second network32, and the third network33are separately shown as being included in the subject model derivation unit21, the latent variable derivation unit22, and the virtual image derivation unit23, respectively, but the first network31, the second network32, and the third network33configure the image generation model according to the present disclosure.

FIG.3is a schematic diagram showing a configuration of the image generation model. As shown inFIG.3, an image generation model30includes the first network31, the second network32, and the third network33. The first network31includes a convolutional neural network (CNN)31A and a combining unit31B. The CNN31A is hierarchically connected with a plurality of convolutional layers and pooling layers. The convolutional layer performs convolution processing using various kernels on the input image, and outputs a feature amount map including the feature amount obtained by the convolution processing. The kernel has an n×n pixel size (for example, n=3), and a weight is set for each element. Specifically, the weight, such as a differential filter that emphasizes the edge of the input image, are set. The convolutional layer applies the kernel to the entire input image or the feature amount map output from the processing layer in the previous stage while shifting an attention pixel of the kernel. Further, the convolutional layer applies an activation function, such as a sigmoid function, to a convolved value, and outputs the feature amount map.

The pooling layer reduces an amount of data in the feature amount map by pooling the feature amount map output by the convolutional layer, and outputs the feature amount map with the reduced amount of data.

Note that the subsequent processing layer outputs the feature amount map while up-sampling the feature amount map.

Moreover, by repeating the outputting, pooling, and up-sampling of the feature amount map in each processing layer, the feature amount for each pixel of the input target image Gi is output as a feature vector from the final layer of the CNN31A. The feature vector is a one-dimensional vector having n elements. In the present embodiment, in a case in which only one target image Gi is input to the first network31, the output feature vector itself is the subject model M0.

On the other hand, in a case in which two target images Gi are input to the first network31, the subject model M0is derived by combining feature vectors r1and r2output for each of two target images (referred to as a first target image G1and a second target image G2) by the combining unit31B.FIG.4is a diagram for describing the generation of the subject model M0by combining. As shown inFIG.4, it is assumed that the first and second target images G1and G2, and the information t1and t2representing the representation formats thereof are input to the CNN31A of the first network31, the first feature vector r1(a1, a2, . . . , an) is derived in certain pixel x of the first target image G1, and the second feature vector r2(b1, b2, . . . , bn) is derived in the pixel of the second target image G2corresponding to the pixel x.

The combining unit31B derives the subject model M0by adding the corresponding elements of the first feature vector r1and the second feature vector r2between the corresponding pixels of the first target image G1and the second target image G2. The subject model M0has the same number of pixels as the input target image Gi, and a composite feature vector is assigned to each pixel. Note that, instead of addition, two feature vectors r1and r2may be combined by deriving representative values, such as an average value and a median value, between the corresponding elements of two feature vectors r1and r2. Here, inFIG.4, two CNNs31A are shown side by side, but the number of CNNs31A included in the first network31may be only one or plural. In a case in which there are a plurality of the CNNs31A, each of the plurality of CNNs31A is constructed by the same learning.

Note that, in the present embodiment, in a case in which the plurality of target images Gi are used, the plurality of target images Gi are normalized. That is, registration processing of aligning the sizes and spatial positions of the subjects included in the target images Gi, smoothing processing of removing fine structural differences and noise, and the like are performed.

In a case in which the target information A0and the subject model M0are input, the second network32outputs the first latent variable z1obtained by dimensionally compressing the feature of the subject model M0according to the target information A0. The second network32includes a convolutional neural network, which is one of the multi-layer neural networks in which a plurality of processing layers are hierarchically connected, but unlike the CNN31A of the first network31, has a function as encoder that dimensionally compresses the feature of the input subject model M0according to the target information A0.FIG.5is a diagram for describing the second network. As shown inFIG.5, the second network32includes an input layer32A, at least one interlayer32B, and an output layer32C, and the dimension of the output layer32C is smaller than the dimension of the input layer32A.

Moreover, in a case in which the target information A0and the subject model M0are input to the input layer32A, the second network32performs processing of reducing (compressing) an information amount of the information representing the feature of the subject model M0such that the virtual image V0having the target representation format represented by the target information A0can be derived, and outputs the latent variable z1from the output layer32C. The latent variable z1represents the feature of the subject model M0, but includes the information having a smaller number of dimensions than the subject model M0. As a result, the latent variable z1obtained by dimensionally compressing the feature of the subject model M0according to the input target information A0is output from the second network32.

In a case in which the target information A0, the subject model M0, and the latent variable z1are input, the third network33outputs the virtual image V0having the target representation format represented by the target information A0. The third network33includes a convolutional neural network, which is one of a multi-layer neural network in which a plurality of processing layers are hierarchically connected, and has a function as decoder that reconstructs the virtual image V0by reconstructing the input subject model M0and the latent variable z1.FIG.6is a diagram for describing the third network. As shown inFIG.6, the third network33includes an input layer33A, at least one interlayer33B, and an output layer33C, and the dimension of the output layer33C is larger than the dimension of the input layer33A.

Moreover, in a case in which the target information A0, the subject model M0, and the latent variable z1are input, the third network33performs processing of reconstructing the virtual image V0, and outputs the virtual image V0from the output layer33C. As a result, the virtual image V0having the target representation format is output from the third network33.

The learning unit25trains the image generation model30by using the plurality of teacher data. That is, the learning unit25trains the first network31of the subject model derivation unit21, the second network32of the latent variable derivation unit22, and the third network33of the virtual image derivation unit23. For this purpose, the learning unit25includes a fourth network34that, in a case in which an image of a certain representation format is input for learning, outputs a latent variable z2obtained by dimensionally compressing the feature of the image of the representation format. The fourth network34has a function as the encoder and has a configuration similar to that of the second network32. The latent variable z2represents the feature of the input image, but includes the information having a smaller number of dimensions than the input image.

In the present embodiment, the fourth network34is used only at the time of learning. Therefore, in the present embodiment, the learning unit25includes the fourth network34, but the learning unit25is not limited to this. Note that it is assumed that the latent variable z1output by the second network32is referred to as a first latent variable, and the latent variable z2output by the fourth network34is referred to as a second latent variable. In addition, it is assumed that the dimensions of the first latent variable z1and the second latent variable z2are the same.

FIG.7is a diagram showing an example of the teacher data. Teacher data40shown inFIG.7includes three teacher images K1to K3as an example. The type of image of the teacher image K1is the CT image, the type of image of the teacher image K2is the T1-weighted image of MRI, and the type of image of the teacher image K3is the T2-weighted image of MRI. In addition, the teacher data40includes teacher information KJ representing the representation format for the teacher image K3. In the present embodiment, since the teacher information KJ is the T2-weighted image, the teacher information KJ represents the type of image of the T2-weighted image as the representation format. Note that the teacher data40may include information representing the type of image of the teacher images K1and K2, that is, the representation formats. In the present embodiment, the teacher data40includes the information representing the type of image of the teacher images K1and K2, that is, the representation formats. The plurality of teacher images included in one teacher data are acquired by imaging the same site of the same subject in the modality2such that images having different representation formats are acquired. For example, the teacher image K1which is the CT image is acquired by the CT apparatus2A, and the teacher image K2which is the T1-weighted image and the teacher image K3which is the T2-weighted image are acquired by the MRI apparatus2B, respectively. Here, inFIG.7, a difference in the representation format of the teacher images K1to K3is represented by giving different hatching to the teacher images K1to K3.

Note that a plurality of the teacher images K1to K3are normalized for learning. That is, registration processing of aligning the spatial positions of the plurality of teacher images K1to K3, smoothing processing of removing fine structural differences and noise, and the like are performed.

FIG.8is a conceptual diagram of learning of the image generation model. First, the teacher image included in the teacher data40is input to the first network31at the time of learning. Specifically, the teacher image other than the teacher image having the representation format represented by the teacher information KJ is input. For example, in the teacher data40shown inFIG.7, the teacher information KJ represents the representation format of the T2-weighted image. Therefore, the teacher image K1which is the CT image and the teacher image K2which is the T1-weighted image other than the teacher image K3which is the T2-weighted image included in the teacher data40are input to the first network31. Note that information Kt1and Kt2representing the representation formats of the teacher images K1and K2are also input to the first network31. As a result, a feature vector for the teacher image K1and a feature vector for the teacher image K2are output from the CNN31A of the first network31. Moreover, two feature vectors are combined by the combining unit31B, and a teacher subject model KM is derived.

In addition, the teacher information KJ included in the teacher data40shown inFIG.7and the teacher subject model KM output from the first network31are input to the second network32. As a result, the second network32outputs a first teacher latent variable Kz1, which is the first latent variable z1obtained by dimensionally compressing the feature of the teacher subject model KM according to the teacher information KJ.

In addition, the teacher information KJ included in the teacher data40shown inFIG.7, the teacher subject model KM output from the first network31, and the first teacher latent variable Kz1output from the second network32are input to the third network33. As a result, the third network33outputs a teacher virtual image KV0having the representation format represented by the teacher information KJ, that is, the representation format of the T2-weighted image.

In addition, the teacher image K3(here, the T2-weighted image) having the representation format corresponding to the teacher information KJ included in the teacher data40shown inFIG.7is input to the fourth network34. As a result, the fourth network34outputs a second teacher latent variable Kz2, which is the second latent variable z2obtained by dimensionally compressing the feature of the teacher image K3.

Moreover, the learning unit25derives a difference between the first teacher latent variable Kz1and the second teacher latent variable Kz2as a first loss L1. Moreover, the first network31and the second network32are trained by using the first loss L1. Here, the first teacher latent variable Kz1output from the second network32is derived based on the teacher information KJ and the teacher subject model KM. Therefore, the first teacher latent variable Kz1is different from the second teacher latent variable Kz2output from the fourth network34based on the teacher image K3having the representation format represented by the teacher information KJ, but a more preferable virtual image V0can be output from the third network33as the difference between the first teacher latent variable Kz1and the second teacher latent variable Kz2is smaller.

For this purpose, in the present embodiment, the learning unit25trains the CNN31A of the first network31and the second network32to reduce the first loss L1. Specifically, regarding the CNN31A, the learning unit25trains the CNN31A by deriving the number of convolutional layers and the number of pooling layers, which configure the CNN31A, a coefficient of the kernel, magnitude of the kernel, and a weight of the bond between the layers in the convolutional layer such that the first loss L1is equal to or less than a predetermined threshold value Th1. In addition, regarding the second network32, the learning unit25trains the second network32by deriving the number of convolutional layers and the number of pooling layers, which configure the second network32, a coefficient of the kernel, magnitude of the kernel, and a weight of the bond between the layers in the convolutional layer such that the first loss L1is equal to or less than the predetermined threshold value Th1.

As a result, in a case in which at least one target image Gi is input, the first network31can output the subject model M0in which the second network32can output the first latent variable z1capable of deriving the virtual image V0having the target representation format. In addition, in a case in which the subject model M0output by the first network31is input, the second network32outputs the first latent variable z1capable of outputting the virtual image V0having the target representation format by the third network33. Note that the learning unit25may perform learning a predetermined number of times instead of learning such that the first loss L1is equal to or less than the predetermined threshold value Th1.

In addition, the learning unit25derives a difference between the teacher virtual image KV0output by the third network33and the teacher image K3having the representation format represented by the teacher information KJ as a second loss L2. Moreover, the first network31, the second network32, and the third network33are trained by using the second loss L2. Here, the teacher virtual image KV0output from the third network33is derived based on the teacher information KJ, the teacher subject model KM, and the first teacher latent variable Kz1. Therefore, the teacher virtual image KV0is different from the teacher image K3having the representation format represented by the teacher information KJ, but a more preferable virtual image V0can be output from the third network33as the difference between the teacher virtual image KV0and the teacher image K3is smaller.

For this purpose, in the present embodiment, the learning unit25trains the CNN31A of the first network31, the second network32, and the third network33to reduce the second loss L2. Specifically, regarding the CNN31A, the learning unit25trains the CNN31A by deriving the number of convolutional layers and the number of pooling layers, which configure the CNN31A, a coefficient of the kernel, magnitude of the kernel, and a weight of the bond between the layers in the convolutional layer such that the second loss L2is equal to or less than a predetermined threshold value Th2. Note that the CNN31A is trained based on both the first loss L1and the second loss L2.

In addition, regarding the second network32, the learning unit25trains the second network32by deriving the number of convolutional layers and the number of pooling layers, which configure the second network32, a coefficient of the kernel, magnitude of the kernel, and a weight of the bond between the layers in the convolutional layer such that the second loss L2is equal to or less than the predetermined threshold value Th2. Note that the second network32is also trained based on both the first loss L1and the second loss L2.

In addition, regarding the third network33, the learning unit25trains the third network33by deriving the number of convolutional layers and the number of pooling layers, which configure the third network33, a coefficient of the kernel, magnitude of the kernel, and a weight of the bond between the layers in the convolutional layer such that the second loss L2is equal to or less than the predetermined threshold value Th2.

As a result, the CNN31A of the first network31outputs the subject model M0in which the second network32can output the first latent variable z1capable of deriving the virtual image V0having the representation format represented by the target information A0and the third network33can output the virtual image V0having the target representation format. In addition, the second network32outputs the first latent variable z1capable of outputting the virtual image V0having the target representation format by the third network33. In addition, in a case in which the target information A0, the subject model M0output by the first network31, and the first latent variable z1output by the second network32are input, the third network33outputs the virtual image V0having the target representation format.

Here, examples of the teacher image used as the teacher data include the CT image acquired by the CT apparatus2A as described above, the T1-weighted image and the T2-weighted image acquired by the MRI apparatus2B, and an image of any other type. For example, examples of the MRI image included in one teacher data include, in addition to the T1-weighted image and the T2-weighted image, the MRI image of any type, such as a diffusion-weighted image, a fat suppression image, an FLAIR image, a pre-contrast T1-weighted image, a post-contrast T1-weighted image, a T1-weighted image (in phase), a T1-weighted image (out phase), and a T2-fat suppression image. In this case, the teacher information KJ representing the type of image, such as the CT image and the MRI image, as the representation format need only be used.

By using such CT images and MRI images having various representation formats as the teacher images and using the teacher information KJ representing the type of image, such as the CT image and the MRI image as the representation format, in a case in which at least one target image Gi having any representation format and the target information A0representing the representation format used as the teacher information KJ are input to the image generation device1, the virtual image V0having the representation format represented by the target information A0is generated. For example, in a case in which the target image Gi is the CT image and the T1-weighted image of MRI, and the representation format represented by the target information A0is the T2-weighted image of MRI, the virtual image V0having the representation format of the T2-weighted image of MRI can be generated from the CT image and the T1-weighted image.

FIG.9is a diagram showing a representation format of the target image to be input and the representation format of the virtual image to be output. Note that, inFIG.9, the left column is the target image to be input, and the types of image as the representation format are, in order from the top, the post-contrast T1-weighted image, the T1-weighted image (out phase), the T2-weighted image, and the post-contrast T1-weighted image. In addition, the third to eighth columns from the left show the virtual images V0having the converted representation format. The types of image as the representation format are the CT image, the post-contrast T1-weighted image, a T1 non-contrast image, the T1-weighted image (in phase), the T1-weighted image (out phase), and the T2-fat suppression image, respectively, in order from the third column on the left side. According to the present embodiment, as shown inFIG.9, by using the teacher data in various representation formats, the virtual image V0having the target representation format is generated regardless of the representation format of the target image Gi to be input.

In addition, in a case in which performing CT imaging, there are a case in which the contrast medium is used and a case in which the contrast medium is not used. Therefore, as shown inFIG.10, a CT image K11acquired by using the contrast medium and a CT image K12acquired without using the contrast medium can be included in one teacher data41. In this case, the teacher information KJ representing the representation format of the presence or absence of the contrast medium in the CT image can be included in the teacher data41. InFIG.10, the teacher information KJ representing the representation format in which the contrast medium is present is included in the teacher data41. In addition, inFIG.10, a region of the contrast medium in the teacher image K11is shown by hatching.

As described above, by learning the image generation model30by using the teacher data41including the CT image acquired by using the contrast medium and the CT image acquired without using the contrast medium as the teacher images K11and K12, and including the teacher information KJ representing the representation format of the presence or absence of the contrast medium, in a case in which at least one target image Gi having any representation format and the target information A0representing the representation format of the presence or absence of the contrast medium are input to the image generation device1, the virtual image V0having the representation format of contrast or non-contrast is generated according to the target information A0. For example, in a case in which the target image Gi is one non-contrast MRI image and the target information A0represents that the contrast medium of CT is present, the image generation device1can generate the virtual image V0having the representation format of the MRI image obtained by performing imaging using the contrast medium.

In addition, in a case in which imaging is performed by using the contrast medium, as shown inFIG.11, the plurality of CT images having different elapsed time after the injection of the contrast medium may be included in one teacher data42as teacher images K21to K23. In this case, the teacher information KJ representing the representation format of the contrast phase, which represents the elapsed time after injection of the contrast medium in the CT image, need only be used. InFIG.11, the teacher information KJ representing the representation format of 30 seconds as the contrast phase is included in the teacher data42. In addition, inFIG.11, the teacher image K21is before contrast, the contrast phase of the teacher image K22is, for example, 10 seconds, and the contrast phase of the teacher image K23is 30 seconds.

As described above, by learning the image generation model30by using the teacher data42including the CT images having different elapsed time after the injection of the contrast medium as the teacher images K21to K23and including the teacher information KJ representing the representation format of the contrast phase, in a case in which at least one target image Gi having any representation format and the target information A0representing the representation format of the contrast phase are input to the image generation device1, the virtual image V0having the representation format of the contrast phase according to the target information A0is generated. For example, in a case in which the target image Gi is one non-contrast MRI image and the contrast phase represented by the target information A0is 30 seconds, the image generation device1can generate the virtual image V0having the representation format of the MRI image of 30 seconds after the injection of the contrast medium.

In addition, in the present embodiment, as shown inFIG.12, teacher data43including teacher images K31to K33having different imaging date and time can be used for the same site of the same subject. The teacher data43shown inFIG.12includes the CT image acquired by imaging on the same day, the CT image acquired by imaging one year ago, and the CT image acquired three years ago, for the same site of the same subject, as the teacher images K31to K33, respectively. In this case, the teacher information KJ representing the representation format of the time before and after the current time (for example, one year ago, three years ago, one year later, or three years later) need only be used.

By learning the image generation model30by using the teacher data43including such images having different imaging date and time as the teacher images K31to K33and including the teacher information KJ representing the representation format of the time before and after the current time, in a case in which at least one target image Gi having any representation format and the target information A0representing the representation format of the time before and after the current time are input to the image generation device1, the virtual image V0having the representation format of imaging period according to the target information A0is generated. For example, in a case in which the target image Gi is one current MRI image and the representation format represented by the target information A0is three years ago, the image generation device1can generate the virtual image V0having the representation format of the MRI image three years ago.

Note that it is also possible to use a two-dimensional radiation image acquired by simple radiation as the teacher image used as the teacher data. However, all the teacher data need to be acquired by imaging the same site of the same subject.

The display control unit24displays the virtual image V0output by the virtual image derivation unit23on the display unit14.FIG.13is a diagram showing a display screen of the virtual image V0. Note that, here, the description will be made that one virtual image V0is generated from two target images G1and G2. As shown inFIG.13, two target images G1and G2, and the virtual image V0are displayed on a display screen50. The target images G1and G2are, for example, the CT image and the T1-weighted image, respectively, and the virtual image V0is the T2-weighted image.

Note that by using the target information A0representing the representation format of a plurality of the contrast phases, it is possible to generate a plurality of the virtual images V0representing a state in which the contrast medium spreads. For example, as shown inFIG.14, it is possible to generate the plurality of virtual images V0in which an aspect of the spread of the contrast medium with the elapse of time of 30 seconds, one minute, and two minutes can be confirmed.

Then, processing performed in the present embodiment will be described.FIG.15is a flowchart showing the learning processing performed in the present embodiment. Note that it is assumed that the plurality of teacher data are acquired from the image storage server3and stored in the storage13. First, the learning unit25acquires one teacher data40from the storage13(step ST1), and inputs the teacher images K1and K2, which are included in the teacher data40, having the representation format other than the teacher image K3having the representation format represented by the teacher information KJ, and the information Kt1and Kt2representing the representation format of the teacher images K1and K2to the first network31. The first network31outputs the teacher subject model KM by deriving the feature amounts from the teacher images K1and K2and combining the feature amounts (step ST2). In addition, the learning unit25inputs the teacher subject model KM and the teacher information KJ to the second network32. The second network32outputs the first teacher latent variable Kz1, which is the first latent variable z1obtained by dimensionally compressing the feature of the teacher subject model KM according to the teacher information KJ (step ST3).

In addition, the learning unit25inputs the teacher image K3in the representation format represented by the teacher information KJ to the fourth network34. The fourth network34outputs the second teacher latent variable Kz2, which is the second latent variable z2obtained by dimensionally compressing the feature of the teacher image K3(step ST4). Further, the learning unit25inputs the teacher information KJ, the teacher subject model KM, and the first teacher latent variable Kz1to the third network33. The third network33outputs the teacher virtual image KV0having the representation format represented by the teacher information KJ (step ST5). Note that the processing of step ST4may be performed in parallel with or before or after any of the processing of steps ST1to ST3, and step ST5.

Then, the learning unit25derives the difference between the first teacher latent variable Kz1and the second teacher latent variable Kz2as the first loss L1(step ST6). In addition, the learning unit25derives the difference between the teacher virtual image KV0and the teacher image K3as the second loss L2(step ST7). Moreover, the learning unit25determines whether or not the first loss L1and the second loss L2are equal to or less than the predetermined threshold values Th1and Th2, respectively (equal to or less than a loss threshold value; step ST8). In a case in which a negative determination is made in step ST8, the learning unit25acquires new teacher data from the storage13(step ST9), returns to the processing of step ST1, and repeats the processing of steps ST1to ST8by using the new teacher data. In a case in which a positive determination is made in step ST8, the learning unit25terminates the learning processing. As a result, the image generation model30is constructed.

FIG.16is a flowchart showing the image generation processing performed in the present embodiment. Note that it is assumed that the target image Gi and the target information A0are input from the input unit15or acquired from the image storage server3, and stored in the storage13. In response to the instruction to start the image generation processing, the information acquisition unit20acquires at least one target image Gi and the target information A0from the storage13(step ST11). The subject model derivation unit21inputs at least one target image Gi and the information ti representing the representation format of the target image Gi to the first network31. The first network31outputs the subject model M0by deriving the feature amount of the target image Gi ad combining the feature amounts. As a result, the subject model derivation unit21derives the subject model M0(step ST12).

In addition, the latent variable derivation unit22inputs the target information A0and the subject model M0to the second network32. The second network32outputs the first latent variable z1obtained by dimensionally compressing the feature of the subject model M0according to the target information A0in a case in which the target information A0and the subject model M0are input. As a result, the latent variable derivation unit22derives the first latent variable z1(step ST13).

The virtual image derivation unit23inputs the target information A0, the subject model M0, and the first latent variable z1to the third network33. The third network33outputs the virtual image V0having the representation format represented by the target information A0. As a result, the virtual image derivation unit23derives the virtual image V0(step ST14). Moreover, the display control unit24displays the virtual image V0on the display unit14(step ST15), and terminates the processing.

As described above, in the present embodiment, the first network31included in the subject model derivation unit21is trained such that the subject model M0is output in which the second network32can output the first latent variable z1capable of deriving the virtual image V0having the representation format represented by the target information A0and the third network33can output the virtual image V0having the target representation format. In addition, the second network32included in the latent variable derivation unit22is trained such that the first latent variable z1capable of outputting the virtual image V0having the target representation format by the third network33is output in a case in which the subject model M0output by the first network31is input. In addition, the third network33included in the virtual image derivation unit23is trained such that the virtual image V0having the target representation format is output in a case in which the target information A0, the subject model M0output by the first network31, and the first latent variable z1output by the second network32are input.

As a result, the first network31of the subject model derivation unit21can be constructed such that the subject model M0is output in which the second network32can output the first latent variable z1capable of deriving the virtual image V0having the representation format represented by the target information A0and the third network33can output the virtual image V0having the target representation format. In addition, the second network32of the latent variable derivation unit22can be constructed such that the first latent variable z1capable of outputting the virtual image V0having the target representation format by the third network33is output in a case in which the subject model M0output by the first network31is input. In addition, the third network33of the virtual image derivation unit23can be constructed such that the virtual image V0having the representation format represented by the target information A0can be output from the target information A0, the subject model M0, and the first latent variable z1.

Therefore, according to the image generation device1according to the present embodiment, the virtual image V0having the target representation format represented by the target information A0can be derived from at least one target image Gi and the target information A0.

By the way, in a case in which the representation format of the image is converted to another representation format, there is a possibility that a unique feature of the original image is impaired. For example, in a case in which the target image Gi is the MRI image, in a case in which the representation format thereof is converted to the CT image, there is a possibility that a fine lesion and the like included in the MRI image disappear in the CT image. Here, it is possible to increase the information of the image that is the source for deriving the virtual image V0by inputting the plurality of target images Gi to the image generation device1according to the present embodiment. Therefore, by using the plurality of target images Gi, it is possible to reduce a possibility of information loss due to the conversion of the representation format.

Note that, in the embodiment described above, the target information A0represents any of the type of image, the presence or absence of the contrast medium, the contrast phase in a case in which the contrast medium is present, or the time before and after the current time, but the present disclosure is not limited to this. The target information A0representing at least one of the above as the representation format may be used. In addition, the target information A0further representing the representation format, such as the gender of the subject and the age of the subject, may be used. In this case, the teacher information KJ included in the teacher data need only further include at least one of the gender of the subject or the age of the subject. As a result, an information amount of the target information A0can be increased, so that a more preferable virtual image V0can be generated. Note that as the target information A0, only information representing the gender of the subject or the age of the subject as the representation format may be used.

Note that, in the embodiment described above, the first network31outputs the subject model M0by inputting the target image Gi and the information ti representing the representation format thereof, but the present disclosure is not limited to this. The first network31may be constructed such that the subject model M0is output only by inputting the target image Gi.

In addition, in the embodiment described above, the subject model derivation unit21, the latent variable derivation unit22, and the virtual image derivation unit23include the first network31, the second network32, and the third network33, which are trained by the learning unit25, respectively, but the present disclosure is not limited to this. For example, it is possible to execute the processing in the subject model derivation unit21, the latent variable derivation unit22, and the virtual image derivation unit23by software, which is not the network constructed by learning.

In addition, in the embodiment described above, the image generation device1includes the learning unit25, but the present disclosure is not limited to this. The learning device comprising the learning unit25may be provided separately from the image generation device1, and the image generation model may be trained by the learning device provided separately. In this case, the image generation model constructed by learning is installed in the image generation device1.

In addition, in the embodiment described above, the liver is the specific structure, but the present disclosure is not limited to this. In addition to the liver, a structure, such as lungs, heart, kidneys, and brain, can be used as the target image. Note that at the time of learning, the first network31, the second network32, and the third network33specialized for the target image Gi including the specific structure are constructed by using the teacher image including the specific structure included in the target image.

In addition, in the embodiment described above, for example, various processors shown below can be used as the hardware structures of processing units that execute various pieces of processing, such as the information acquisition unit20, the subject model derivation unit21, the latent variable derivation unit22, the virtual image derivation unit23, the display control unit24, and the learning unit25. As described above, the various processors include, in addition to the CPU that is a general-purpose processor which executes software (program) and functions as various processing units, a programmable logic device (PLD) that is a processor whose circuit configuration can be changed after manufacture, such as a field programmable gate array (FPGA), and a dedicated electric circuit that is a processor having a circuit configuration which is designed for exclusive use in order to execute a specific processing, such as an application specific integrated circuit (ASIC).

One processing unit may be configured by one of these various processors, or may be a combination of two or more processors of the same type or different types (for example, a combination of a plurality of FPGAs or a combination of the CPU and the FPGA). In addition, a plurality of the processing units may be configured by one processor.

As an example of configuring the plurality of processing units by one processor, first, as represented by a computer, such as a client and a server, there is an aspect in which one processor is configured by a combination of one or more CPUs and software and this processor functions as a plurality of processing units. Second, as represented by a system on chip (SoC) or the like, there is an aspect of using a processor that realizes the function of the entire system including the plurality of processing units by one integrated circuit (IC) chip. As described above, as the hardware structure, various processing units are configured by one or more of various processors described above.

Further, as the hardware structures of these various processors, more specifically, it is possible to use an electrical circuit (circuitry) in which circuit elements such as semiconductor elements are combined.