Patent ID: 12217387

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Preferred embodiments of the present invention will be described below in detail with reference to the accompanying drawings.

First Embodiment

A super resolution image generating device according to an embodiment of the present invention generates virtual high resolution image data from low resolution image data. “Generate” includes the concept of the term “estimate”. Here, as a specific example of the image data, data of a computed tomography (CT) image acquired by using a CT device is targeted, and a super resolution image generating device is described as an example, which generates virtual thin slice image data from thick slice image data acquired by using the CT device.

The thick slice image data refers to low resolution CT image data having a relatively large slice interval and slice thickness. For example, the CT image data having a slice interval and a slice thickness of more than 4 mm corresponds to the thick slice image data. The thick slice image data may be referred to as “thick slice image”, “thick slice data”, or “thick data”.

The thin slice image data is high resolution CT image data having a small slice interval and slice thickness. For example, the CT image data having the slice interval and the slice thickness of about 1 mm corresponds to the thin slice image data. The thin slice image data may be referred to as “thin slice image”, “thin slice data”, or “thin data”.

In the present embodiment, the virtual thin slice image generated from the thick slice image is referred to as a virtual thin slice (VTS) image. On the other hand, a real thin slice image acquired by imaging with the CT device is referred to as a real thin slice (RTS) image.

Explanation of CT Image Data

FIG.1is an image diagram of each data of the thick slice image and the VTS image. The left side ofFIG.1is the thick slice image, and the right side ofFIG.1is the VTS image. The VTS image can generate a reconstructed image having a high quality as compared with the thick slice image. InFIG.1, the Z-axis direction is the body axis direction.

As the CT data, data having various slice intervals and slice thicknesses may be present depending on the model of the CT device used for imaging and depending on the setting of output slice conditions and the like.

FIGS.2to5are diagrams for explaining examples of the slice interval and the slice thickness of the CT image. The slice interval is the distance between the center positions of the thicknesses of a slice and the adjacent slice. The slice interval is synonymous with the distance between the slices. The slice thickness refers to the length in the thickness direction of one slice at the center position of an imaging region. The slice thickness is synonymous with a slice thickness. InFIGS.2to5, the slice interval is displayed as SD and the slice thickness is displayed as ST. The thickness direction of the slice is the Z-axis direction.

FIG.2is an explanatory diagram schematically showing a CT image IM1in a case in which a slice interval SD is 4 mm and a slice thickness ST is 4 mm. Here, for the sake of simplicity, a three-layer tomographic image group is shown schematically.

FIG.3is an explanatory diagram schematically showing a CT image IM2in a case in which the slice interval SD is 4 mm and the slice thickness ST is 6 mm. In the case ofFIG.3, the slice thickness ranges overlap between the adjacent slices.

FIG.4is an explanatory diagram schematically showing a CT image IM3in a case in which the slice interval SD is 8 mm and the slice thickness ST is 4 mm. In the case of the example ofFIG.4, the slice interval SD is larger than the slice thickness ST, and thus adjacent tomographic images are separated from each other and there is a gap between the layers.

FIG.5is an explanatory diagram schematically showing a CT image IM4in which the slice interval SD is 1 mm and the slice thickness ST is 1 mm. The CT image Im4shown inFIG.5has a larger amount of information in the Z direction than the other CT images IM1to IM3shown inFIGS.2to4. That is, the CT image IM4has a relatively higher resolution than any of the CT images IM1, IM2, and IM3in the Z direction.

The slice interval and slice thickness of the CT image are set under various conditions depending on the facility at which the CT device is used, the preference of a doctor, and the like. It is preferable that the CT image have a high resolution for diagnosis, but there is a problem that in a case in which the slice interval is reduced, the amount of exposure to a subject is increased. In addition, the high resolution CT image has a large amount of data and presses a storage capacity of a storage, so that the CT image may be stored at a lower resolution in order to reduce the capacity. For example, the old CT data is stored in a database by reducing the number of imaged slices.

However, the thick slice image has a problem that it is difficult to use for sufficient observation and analysis because the quality of the reconstructed image or a volume rendered image viewed from a side surface with a surface parallel to the body axis as a cross section is poor.

The super resolution image generating device according to the present embodiment performs image generation processing of generating the high resolution VTS image having, for example, the slice interval of 1 mm and the slice thickness of 1 mm as shown inFIG.5from the low resolution CT image of various slice conditions (slice interval and slice thickness) as shown inFIGS.3to5.

Example of Image Generation Algorithm in Super Resolution Image Generating Device

FIG.6is a functional block diagram showing an example of the super resolution image generating device according to the embodiment of the present invention. The super resolution image generating device10includes an interpolation processing unit12, a generator14which is a learned model of a multilayered neural network, and an addition unit16. The “neural network” is a mathematical model of information processing that simulates a mechanism of a cranial nerve system. Processing using the neural network can be realized by using a computer. The neural network can be configured as a program module. In the present specification, the neural network may be simply referred to as a “network”.

The interpolation processing unit12performs spline interpolation on an input low resolution thick slice image TCK to generate an interpolation image IPT. The interpolation image IPT output from the interpolation processing unit12is an image blurred in the Z direction, and is an example of a low resolution image. It is preferable that the number of pixels of the interpolation image IPT match the number of pixels of a finally generated virtual thin slice image VT.

The interpolation image IPT output from the interpolation processing unit12is input to the generator14. The generator14is a generative model learned by machine learning by using a generative adversarial network (GAN). A learning method for obtaining the generator14will be described below. The learned model may be also referred to as the program module.

The generator14generates (estimates) high-frequency component information required to generate a high resolution image from the input image, and outputs the high-frequency component information.

The addition unit16adds a map of the high-frequency component information output from the generator14and the interpolation image IPT itself which is the input data of the generator14to generate the virtual thin slice image VT.

FIG.6shows an example in which the input to the generator14is the interpolation image IPT and the output of the generator14is the high-frequency component information, but a form in which the input to the generator14is the thick slice image TCK can also be adopted. A form in which the output of the generator14is the virtual thin slice image VT can also be adopted. Since the high-frequency component information is information that can generate the high resolution image by adding the original image, the map of the high-frequency component information is called the “high-frequency component image”. The high-frequency component image is an image including high resolution image information, and can be understood as substantially the same as the “high resolution image”.

InFIG.6, the thick slice image TCK is an example of a “third image” in the present disclosure. The virtual thin slice image VT is an example of a “fourth image” in the present disclosure. The high-frequency component output from the generator14is an example of “higher resolution image information than the third image” in the present disclosure. The interpolation processing unit12is an example of a “first interpolation processing unit” in the present disclosure. The addition unit16is an example of a “first addition unit” in the present disclosure.

Configuration Example of Learning System

Next, the learning method for generating the generator14will be described.

FIG.7is a block diagram showing a configuration example of a learning system20according to the embodiment of the present invention. The learning system20includes an image storage unit24, a learning data generating unit30, and a learning unit40. The learning system20can be realized by a computer system including one or a plurality of the computers. That is, the functions of the image storage unit24, the learning data generating unit30, and the learning unit40can be realized by a combination of a hardware and a software of the computer. Here, an example in which the image storage unit24, the learning data generating unit30, and the learning unit40are configured as separate devices will be described, but these functions may be realized by one computer, or the processing functions may be allocated and realized by two or more computers. For example, the image storage unit24, the learning data generating unit30, and the learning unit40may be connected to each other via a communication line. The term “connection” is not limited to wired connection, and also includes the concept of wireless connection. The communication line may be a local area network, or may be a wide area network.

With this configuration, the generation of the learning data and the learning of the generative model can be performed without being physically and temporally restricted by each other.

The image storage unit24includes a large-capacity storage device that stores a CT reconstructed image (CT image) captured by a medical X-ray CT device. The image storage unit24may be, for example, a storage in a medical image management system represented by a picture archiving and communication system (PACS). The image storage unit24stores data of a plurality of thin slice images, which are real high resolution images captured by using the CT device (not shown).

The CT image stored in the image storage unit24is a medical image obtained by imaging a human body (subject), and is a three-dimensional tomographic image including a plurality of tomographic images. Here, each tomographic image is an image parallel to the X direction and the Y direction which are orthogonal to each other. The Z direction orthogonal to the X direction and the Y direction is the body axis direction of the subject, and is also referred to as a slice thickness direction. The CT image stored in the image storage unit24may be an image for each part of the human body or an image obtained by imaging the whole body.

The learning data generating unit30generates the learning data required for the learning unit40to perform learning. The learning data is data for training used for machine learning, and is synonymous with “data for learning” or “training data”. In the machine learning of the present embodiment, a large number of learning data of an image pair in which a low resolution image for input and a high resolution correct image corresponding to the low resolution image are associated are used. Such an image pair can be artificially generated by image processing based on the thin slice data which is the real high resolution image.

The learning data generating unit30acquires an original real high resolution image from the image storage unit24and performs down-sampling processing on the real high resolution image to artificially generate various low resolution images (pseudo thick slice images). The learning data generating unit30performs posture conversion on, for example, the original thin slice data equalized to 1 mm, randomly cuts out a fixed-size region, and then generates virtual 4 mm slice data having the slice interval of 4 mm and virtual 8 mm slice data having the slice interval of 8 mm. The fixed-size region may be a three-dimensional region in which the number of pixels in X-axis direction×Y-axis direction×Z-axis direction is, for example, “160×160×160”. The learning data generating unit30generates an image pair of a fixed-size low resolution image LQ for learning and a real high resolution image RH corresponding to the fixed-size low resolution image LQ.

In order to perform the learning processing by the learning unit40, it is preferable that a plurality of pieces of the learning data be generated in advance from the original real high resolution image by using the learning data generating unit30, and stored in the storage as a learning data set.

The low resolution image LQ and the real high resolution image RH which are generated by the learning data generating unit30are input to the learning unit40.

The learning unit40includes a generative adversarial network (GAN)41as a learning model. The architecture of the learning unit40is based on a structure obtained by extending the architecture disclosed in Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros “Image-to-Image Translation with Conditional Adversarial Networks”, CVPR2016 from two-dimensional data to three-dimensional data. The GAN41is configured to include the generation network called a generator42G that produces the data and the identification network called a discriminator44D that identifies the input data. That is, the generator42G is a generative model that generates the image data, and the discriminator44D is an identification model that identifies the data. The term “generator” is synonymous with terms such as “generating unit”, “generating device”, and “generative model”. The term “discriminator” is synonymous with terms such as “identification unit”, “identification device”, and “identification model”.

By repeating adversarial learning by using the generator42G and the discriminator44based on the input learning data, the learning unit40learns the generator42G while improving the performance of both models.

A self-attention mechanism is implemented in the discriminator44D in this example. The layer to which the self-attention mechanism is introduced in the network of the discriminator44D may be a part or all of a plurality of convolutional layers. Details of the configuration and the operation of the discriminator44D including the self-attention mechanism and an example of the learning method of the GAN41will be described below.

The learning unit40includes an error calculating unit50and an optimizer52. The error calculating unit50evaluates an error between the output of the discriminator44D and the correct answer using the loss function. The optimizer52performs processing of updating network parameters based on the calculation result of the error calculating unit50. The network parameters include a filter coefficient (weight of connection between nodes) of filters used to process each layer, node bias, and the like.

The optimizer52performs parameter calculation processing of calculating the update amount of the parameters of each network of the generator42G and the discriminator44D from the calculation result of the error calculating unit50, and parameter update processing of updating the parameters of each network of the generator42G and the discriminator44D depending on the calculation result of the parameter calculation processing. The optimizer52updates the parameters based on the algorithm such as the gradient descent method.

About Generation of Learning Data

FIG.8is a functional block diagram showing a configuration example of the learning data generating unit30. The learning data generating unit30includes a fixed-size region cutout unit31, a down-sampling processing unit32, an up-sampling processing unit34, and a learning data storage unit38.

The fixed-size region cutout unit31performs processing of randomly cutting out the fixed-size region from an input original real high resolution image ORH1. A real high resolution image RH1of the fixed-size region cut out by the fixed-size region cutout unit31is sent to the down-sampling processing unit32.

The down-sampling processing unit32performs down-sampling of the real high resolution image RH1in the Z-axis direction to generate a low resolution thick slice image LK1. As the down-sampling processing, for example, thinning processing need only be performed so as to simply reduce the slices in the Z-axis direction at a certain rate. In this example, only the down-sampling in the Z-axis direction is performed, and the down-sampling is not performed in the X-axis direction and the Y-axis direction, but a form in which the down-sampling is performed in the X-axis direction and the Y-axis direction can be adopted.

The thick slice image LK1generated by the down-sampling processing unit32is input to the up-sampling processing unit34.

The up-sampling processing unit34performs up-sampling of the thick slice image LK1in the Z-axis direction to generate a low resolution image LQ1which is a low quality thin slice image. The up-sampling processing may be, for example, a combination of spline interpolation and Gaussian filter processing. The up-sampling processing unit34includes an interpolation processing unit35and a Gaussian filter processing unit36. The interpolation processing unit35performs spline interpolation on, for example, the thick slice image LK1. The interpolation processing unit35may be the same processing unit as the interpolation processing unit12described with reference toFIG.6. The Gaussian filter processing unit36applies a Gaussian filter to the image output from the interpolation processing unit35and performs smoothing. The interpolation processing unit35shown inFIG.8is an example of a “second interpolation processing unit” in the present disclosure. The Gaussian filter processing unit36is an example of a “smoothing processing unit” in the present disclosure.

It is preferable that the low resolution image LQ1output from the up-sampling processing unit34be data having the same number of pixels as the real high resolution image RH1. Here, the low resolution image LQ1and the real high resolution image RH1have the same size. The low resolution image LQ1is an image having a low quality (that is, low resolution) as compared with the real high resolution image RH1. The pair in which the low resolution image LQ1generated in this way is associated with the real high resolution image RH1which is the generation source thereof is generated is stored in the learning data storage unit38.

The original real high resolution image ORH1is an example of an “original image” in the present disclosure. The real high resolution image RH1is an example of a “second learning image” in the present disclosure. The low resolution image LQ1is an example of a “first learning image” in the present disclosure. The image information of the low resolution image LQ1is an example of “first resolution information” in the present disclosure. The image information of the real high resolution image RH1is an example of “second resolution information” in the present disclosure.

The learning data generating unit30changes the cutout position of the fixed-size region from one original real high resolution image ORH1, cuts out a plurality of real high resolution images RH, and generates the low resolution images LQ corresponding to the real high resolution images RH to generate a plurality of image pairs.

Further, the learning data generating unit30can generate the low resolution images with various slice conditions by changing the combination of the slice interpolation magnification in the up-sampling processing unit34and the conditions of the Gaussian filter applied to the up-sampling processing unit34. The slice interpolation magnification corresponds to the down-sampling condition in the down-sampling processing unit32.

It is preferable that the data of various slice conditions be provided in a case of learning. In the present embodiment, learning is performed by using the low resolution images corresponding to various slice conditions as shown inFIG.9.FIG.9is a table showing an example of a combination of the Gaussian filter conditions corresponding to the slice interval applied in a case in which the learning data is generated and an assumed slice thickness.

In this example, the slice intervals of the low resolution image LQ are set to two ways of 4 mm and 8 mm. That is, the slice interpolation magnification at the time of learning has two patterns of 4 times or 8 times. The slice thickness is in a range of 0 mm to 8 mm corresponding to the slice interval. By randomly providing the standard deviation a of the Gaussian filter within the numerical range shown inFIG.9, the low resolution image in which pseudo various slice thicknesses are assumed can be generated.

By using a plurality of types of the original real high resolution images, a large number of various learning data can be prepared.

Example of Procedure for Processing of Generating Learning Data

FIG.10is a flowchart showing an example of a procedure for processing of generating the learning data. Each step of the flowchart shown inFIG.10is executed by a computer including a processor that functions as the learning data generating unit30. The computer comprises a central processing unit (CPU) and a memory. The computer may include a graphics processing unit (GPU).

As shown inFIG.10, a generation method of the learning data includes an original image acquiring step (step S1), a fixed-size region cutout step (step S2), a down-sampling step (step S3), an up-sampling step (step S4), and a learning data storing step (step S5).

In step S1, the learning data generating unit30acquires the original real high resolution image ORH from the image storage unit24. Here, the real high resolution image ORH equalized to the slice interval of 1 mm and the slice thickness of 1 mm is acquired.

In step S2, the fixed-size region cutout unit31performs processing of cutting out the fixed-size region from the input original real high resolution image ORH to generate the real high resolution image RH1of the fixed-size region.

In step S3, the down-sampling processing unit32performs down-sampling of the real high resolution image RH1to generate the thick slice image LK1. Here, as described with reference toFIG.9, the thick slice image LK1having the slice interval of 4 mm or 8 mm is generated.

In step S4, the up-sampling processing unit34performs up-sampling of the thick slice image LK1obtained by the down-sampling to generate the low resolution image LQ1corresponding to the low quality thin slice image. Here, as described with reference toFIG.9, the interpolation processing and the Gaussian filter processing are performed by applying the slice interpolation magnification corresponding to the slice interval and the Gaussian filter conditions.

In step S5, the learning data generating unit30associates the low resolution image LQ1generated in step S4with the real high resolution image RH, which is the generation source data thereof, as an image pair, and stores these data in the learning data storage unit38as learning data.

After step S5, the learning data generating unit30completes the flowchart ofFIG.8.

In a case in which a plurality of pieces of the learning data are generated from the same original real high resolution image ORH by changing the location of the cutout region, after step S5, the process is returned to step S2, and the processes of steps S2to S5are repeated.

Further, in a case in which the low resolution images having different slice conditions or different assumed slice thicknesses are generated from the real high resolution image RH of the same fixed-size region, after step S5, the process is returned to step S3or step S4, the processing condition is changed, and the process from step S3or step S4is repeated.

The learning data generating unit30can generate a large number of learning data by repeatedly executing the processes of steps S1to S5on a plurality of original real high resolution images stored in the image storage unit24.

Learning Architecture

As described above, the generator14mounted on the super resolution image generating device10according to the present embodiment is the generative model acquired by performing learning by the GAN. The configuration and the learning method of the learning unit40will be described below in detail.

FIG.11is a conceptual diagram of processing in the learning unit40to which the GAN is applied.FIG.11shows an example in which a pair of the low resolution image LQ1and the real high resolution image RH1is input to the learning unit40as the data for learning.

The low resolution image LQ1is the input to the generator42G. The generator42G generates a virtual high resolution image VH1from the input low resolution image LQ1and outputs the generated virtual high resolution image VH1. The virtual high resolution image VH1corresponds to the virtual thin slice image (VTS image). A pair of the virtual high resolution image VH1generated by the generator42G and the low resolution image LQ1which is the generation source of the virtual high resolution image VH1, or a pair of the real high resolution image RH1which is the learning data and the low resolution image LQ1is provided to be input to the discriminator44D.

The discriminator44D identifies whether the input image pair is a real pair including the real high resolution image RH1(whether it is the learning data) or a fake pair including the virtual high resolution image VH1derived from the output of the generator42G, and outputs the identification result.

The error calculating unit50evaluates an error between the output of the discriminator44D and the correct answer using the loss function. The optimizer52performs processing of automatically adjusting the network parameters based on the calculation result of the error calculating unit50. The network parameters include the weight of connection between nodes and node bias. The optimizer52performs parameter calculation processing of calculating the update amount of the parameters of each network of the generator42G and the discriminator44D from the calculation result of the error calculating unit50, and parameter update processing of updating the parameters of each network of the generator42G and the discriminator44D depending on the calculation result of the parameter calculation processing. The optimizer52updates the parameters based on the algorithm such as the gradient descent method. The technique disclosed in Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio “Generative Adversarial Nets”, arXiv: 1406.2661 or the like may be adopted for the part of the basic mechanism of learning regarding the error evaluation and the parameter update.

The generator42G is learned to generate a more elaborate virtual high resolution image such that the discriminator44D is deceived, and the discriminator44D is learned to more accurately identify real or fake.

Finally, the portion of the generator42G is used as the generator14which is the image generation module in the super resolution image generating device10.

The self-attention mechanism is implemented in the network applied to the discriminator44D according to the present embodiment. The self-attention mechanism is a method for improving calculation efficiency by considering a wide part in the image.

Explanation of Discriminator44D Including Self-Attention Mechanism

The content of the self-attention mechanism is disclosed in Han Zhang, Ian Goodfellow, Dimitris Metaxas, Augustus Odena “Self-Attention Generative Adversarial Networks”, arXiv: 1805.08318. However, the present embodiment is different from the method disclosed in Han Zhang, Ian Goodfellow, Dimitris Metaxas, Augustus Odena “Self-Attention Generative Adversarial Networks”, arXiv: 1805.08318 in that the self-attention mechanism is added to both networks of the generator and discriminator in Han Zhang, Ian Goodfellow, Dimitris Metaxas, Augustus Odena “Self-Attention Generative Adversarial Networks”, arXiv: 1805.08318, whereas the self-attention mechanism is not implemented in the generator42G, and the self-attention mechanism is implemented in only the discriminator44D in the present embodiment.

The self-attention mechanism will be briefly outlined with reference to the contents of Han Zhang, Ian Goodfellow, Dimitris Metaxas, Augustus Odena “Self-Attention Generative Adversarial Networks”, arXiv: 1805.08318. The self-attention mechanism generates a query f(x) and a key g(x) from the convolutional feature map CFM(x) output from the hidden layer of the previous layer, and uses these query and key to calculate a value (similarity), which indicate which other pixel is similar, for each pixel. The map of similarity calculated as described above, which corresponds to all pixels of the feature map CFM(x) is called an “attention map”.

The attention map serves to find and emphasize regions having similar features in the image. In the convolutional calculation of the convolutional layer that configures the identification network, local information is superimposed, but by introducing the attention map, it is possible to consider the information of the wide (overall) part.

This attention map is multiplied by a weight h(x) to obtain a self-attention feature map SAFM(o). Then, the self-attention feature map SAFM(o) is multiplied by a scale parameter γ, added to the convolutional feature map CFM(x) which is the original input feature map, and passed to the next layer. That is, a final output y to be passed to the next layer is obtained by the following equation.
y=γ·o+x

In the network of the discriminator44D which includes such a self-attention mechanism, the parameters of f(x), g(x), and h(x) of the self-attention mechanism are also learned.

Example of Identification Network

FIG.12is a conceptual diagram showing an example of the identification network applied to the discriminator44D. The network of the discriminator44D is a multilayered neural network classified as a deep neural network, and includes a plurality of convolutional layers. The network of the discriminator44D is configured by a convolutional neural network (CNN).

InFIG.12, the white arrows indicated by the reference numerals C01, C02. . . C05represent the “convolutional layers”. The rectangles shown on the input side and/or output side of each layer represent a set of feature maps. The length of the rectangle in the vertical direction represents the size (number of pixels) of the feature map, and the width of the rectangle in the horizontal direction represents the number of channels. In the discriminator44D of this example, a pooling layer is not present, and for example, by convolving the filter having a size of 4×4×4 with stride=2, the image size of the feature map is reduced. For example, in a case in which the CNN processing is performed, the minimum image size after convolution can be 1/16 of the size of the input data.

In the example shown inFIG.12, the self-attention mechanism is introduced from the convolutional layer C02to each of the subsequent layers. For example, the self-attention feature map is generated for each of CNN feature map of 128 channel output from the convolutional layer C02.FIG.12shows that the self-attention feature maps of the 128 channel corresponding to the CNN feature maps of the 128 channel are added. Each of the CNN feature map and the self-attention feature map is added for each channel, and the output is input to the next convolutional layer. The same applies to the convolutional layers C03and C04.

The CNN feature map to be input to the self-attention mechanism is calculated by converting the entire image to be input into the one-dimensional array. The CNN feature map in which the number of input channels is C and the total number of pixels is N is input to the self-attention mechanism as a vector in which C×N elements of each pixel are arranged one-dimensionally.

The actual CT image data is the three-dimensional data, and the multidimensional data can be calculated as the one-dimensional array in the same manner as described above. The same processing algorithm can be applied to both the two-dimensional image data and the three-dimensional image data by calculating the data as the one-dimensional array.

Example of Generation Network

FIG.13is a conceptual diagram showing an example of the generation network applied to the generator42G. The network of the generator42G is also configured by a convolutional neural network. The generator42G preferably has a configuration with an encoder and decoder structure in which an encoder portion and a decoder portion are combined. An example of a U-shaped network called a U-Net structure is shown inFIG.13. “Net” in the notation of “U-Net” is a simple notation of “network”.

InFIG.13, the arrows indicated by the reference numerals C1, C2. . . C10each represent the “convolutional layers”. The arrows, which are indicated by the reference numerals U1, U2, U3, and U4, represent the convolutional layers that perform “convolution and up-sampling”. Similar to the discriminator44D described with reference toFIG.12, in the generator42G shown inFIG.13, a pooling layer is not present, and by convolving the filter with stride=2, the image size of the feature map is reduced in the encoder portion.

The generator42G estimates a high-frequency component image VHFC as the high resolution information required for the increase in a resolution from the input low resolution image LQ and outputs the estimated high-frequency component image VHFC. As shown inFIG.14, a virtual high resolution image VH is obtained by adding the low resolution image LQ which is the input data to the generator42G and the high-frequency component image VHFC generated by the generator42G. The slice interval and the slice thickness of the virtual high resolution image VH1are the same as the slice interval and the slice thickness of the low resolution image LQ1, but the virtual high resolution image VH1is an image sharper in the Z direction as compared with the low resolution image LQ1.

The learning unit40comprises an addition unit46that adds the input of the generator42G and the output of the generator42G, and has a configuration in which the output of the addition unit46is input to the discriminator44D. The addition unit46is an example of a “second addition unit” in the present disclosure. The addition unit46is not shown inFIGS.7and11. The virtual high resolution image VH1input to the discriminator44D is an example of a “virtual second image” in the present disclosure.

The low resolution image LQ provided as the input to the generator42G is an example of a “first image” in the present disclosure. The high-frequency component image output from the generator42G is an example of a “second image” in the present disclosure.

InFIGS.13and14, an example in which the output of the generator42G is the high-frequency component image VHFC is described, a form in which the output of the generator42G is the virtual high resolution image VH can be adopted. In this case, the addition unit46is unnecessary. Such an embodiment will be described below as a second embodiment.

Identification Operation of Discriminator44D at Time of Learning

FIG.15is a diagram for explaining an operation of identification by the discriminator44D at the time of learning. The addition unit46is not shown inFIG.15.

An operation state70P shown on the left side ofFIG.15indicates a case in which a positive sample (positive example) is input to the discriminator44D, an operation state70N shown on the right side ofFIG.15indicates a case in which a negative sample (negative example) is input to the discriminator44D.

This is an example in which the real high resolution image RH1and the low resolution image LQ1corresponding to the real high resolution image RH1, which are an image pair of learning data, are input. In this case, in a case in which the discriminator44D identifies the input high resolution image as the real high resolution image RH1, the output (identification result) of the discriminator44D is correct, and in a case in which the discriminator44D identifies the input high resolution image as the virtual high resolution image VH1, identification result is incorrect.

On the other hand, in the case of the operation state70N shown on the right side ofFIG.15, the image pair of the virtual high resolution image VH1derived from the generator42G and the low resolution image LQ1which is the generation source data thereof is input to the discriminator44D. In this case, in a case in which the discriminator44D identifies the input high resolution image as the real high resolution image RH1, the identification result is incorrect, and in a case in which the discriminator44D identifies the input high resolution image as the virtual high resolution image VH1, the identification result is correct.

The discriminator44D is learned to make the identification correct as to whether the input high resolution image is the real CT image captured by the CT device (not shown) or the virtual CT image generated by the generator42G. On the other hand, the generator42G is learned to generate the virtual CT image resembling the real CT image captured by the CT device (not shown) and to make the identification of the discriminator44D incorrect.

As the learning progresses, the discriminator44D and the generator42G increase each other's accuracy, and the generator42G can generate the virtual high resolution image VH close to the real CT image, which is not identified as a fake (virtual high resolution image) by the discriminator44D.

The learned generator42G acquired by such learning is applied as the generator14of the super resolution image generating device10described with reference toFIG.6.

Learning Method Using Learning System20

FIG.16is a flowchart showing an example of a procedure for processing in the learning unit40. Each step of the flowchart shown inFIG.16is executed by a computer including a processor that functions as the learning unit40.

The learning unit40acquires the learning data in step S11. The learning unit40reads the learning data from the learning data generating unit30described with reference toFIG.8. The learning unit40can acquire the learning data in a unit of mini batch including a plurality of pieces of the learning data.

The learning unit40inputs the low resolution image of the learning data to the generator42G in step S12.

The generator42G generates the virtual high resolution image from the input low resolution image in step S13. The output from the generator42G may be the high-frequency component image VHFC required to produce the virtual high resolution image. In this case, as described with reference toFIG.14, the high-frequency component image VHFC and the low resolution image are added to generate the virtual high resolution image VH.

The learning unit40inputs the data to the discriminator44D in step S14. As the input to the discriminator44D, any of a pair (real pair) of the learning data including the real high resolution image as the correct image or a fake pair including the virtual high resolution image derived from the generator42G is selectively provided.

The discriminator44D identifies the data in step S15.

The error calculating unit50calculates the error of the identification result and sends the calculation result to the optimizer52in step S16.

The optimizer52calculates the update amount of the network parameters based on the calculated error in step S17.

In step S18, the optimizer52performs parameter update processing depending on the update amount of the parameters calculated in step S17. The parameter update processing is performed in a unit of mini batch.

The learning unit40determines whether or not to complete the learning in step S19. A learning completing condition may be determined based on the error value or may be determined based on the number of updates of parameter. As the method based on the error value, for example, the learning completing condition may be that the error converges within a specified range. As the method based on the number of updates, for example, the learning completing condition may be that the number of updates reaches the specified number of times.

In a case in which the determination result in step S19is No, the learning unit40returns the process to step S11and repeats the learning processing until the learning completing condition is satisfied.

In a case in which the determination result in step S19is Yes, the learning unit completes the flowchart ofFIG.16.

The portion of the learned generator42G obtained as described above is applied as the generator14of the super resolution image generating device10.

Effect of First Embodiment

FIG.17is an example of an image, which shows the effect of the present embodiment.FIG.17shows an example of an image, which shows the effect of introducing the self-attention mechanism. An image VHA1shown in the upper center ofFIG.17is an example of the virtual high resolution image generated by using the learned model (generator14) to which the learning method according to the present embodiment is applied. An image LI1shown on the upper left ofFIG.17is an example of the low resolution image used for input to the generator14. An image GT1shown on the upper right ofFIG.17is the correct image (ground truth) corresponding to the input image LI1.

An image VHN2shown in the lower center ofFIG.17is an example of the virtual high resolution image generated by using the learned model according to Comparative Example. The learned model according to Comparative Example is learned by using the discriminator that does not include the attention mechanism. An image LI2shown on the lower left ofFIG.17is an example of the low resolution image used for input to the learned model according to Comparative Example. An image GT2shown on the lower right ofFIG.17is the correct image (ground truth) corresponding to the input image LI2.

The image VHA1shown in the upper center ofFIG.17is the image very close to the correct image GT1. Further, it can be seen that the image VHA1reduces the local noise as compared with the image VHN2in Comparative Example on the lower side ofFIG.17.

That is, according to the present embodiment, the noise locally generated in Comparative Example can be reduced by the effect of introducing the self-attention mechanism into the discriminator44D.

FIG.18is a diagram for explaining other effects of the present embodiment.FIG.18shows an image example showing that the learned model (generator14) to which the learning method according to the present embodiment is applied can execute image generation processing (super resolution processing) regardless of the input size.

An image LI3shown on the far left ofFIG.18is an example of an image input to the learned model (generator14) to which the learning method according to the present embodiment is applied. An image VHA3which is second from the left inFIG.18is an example of the virtual high resolution image generated by using the generator14from the input of the image LI3.

An image LI4which is third from the left inFIG.18is another example of the image input to the generator14. This image LI4has a smaller image size than the image LI3shown on the far left ofFIG.18. An image VHA4shown on the far right ofFIG.18is an example of the virtual high resolution image generated by using the generator14from the input of the image LI4.

As shown inFIG.18, the generator14can also perform the estimation processing on the input of the image having a size different from the image size used at the time of learning. The generator14can generate the image with high accuracy for the input data of any image size. That is, according to the present embodiment, the image generation processing can be performed on the image of any size without being restricted by the image size, which is the fixed size, used as the learning data at the time of learning. According to the present embodiment, the processing can be performed by dividing the input data into memories of any size.

Since the discriminator44D is only used in a case of learning and does not need to be mounted on the super resolution image generating device10, learning is performed with the fixed size even in a case in which the attention mechanism is added to the discriminator44D, and there is no problem.

Modification Example

In the first embodiment described above, the pair of the real high resolution image RH and the low resolution image LQ1or the pair of the virtual high resolution image VH derived from the generator42G and the low resolution image LQ1is provided as the input to the discriminator44D, but it is not essential to input the low resolution image LQ1to the discriminator44D. At least the real high resolution image RH or the virtual high resolution image VH need only be input to the discriminator44D.

Second Embodiment

In the first embodiment, as shown inFIG.14, the virtual high resolution image VH is obtained by outputting the (virtual) high-frequency component image VHFC from the generator42G and adding the low resolution image LQ and the high-frequency component image VHFC. On the other hand, in the second embodiment, the generator42G outputs the virtual high resolution image VH.

FIG.19is a functional block diagram schematically showing a flow of the processing by the learning system20according to the second embodiment. InFIG.19, the portions common to or similar to the configurations shown inFIGS.7,8, and11to14are designated by the same reference numerals, and detailed description thereof will be omitted.

InFIG.19, the content of the learning data generating unit30is the same as that inFIG.8. The generator42G of the learning unit40according to the second embodiment generates the virtual high resolution image VH1from the low resolution image LQ1.

Also, the low resolution image LQ1is input to the discriminator44D in a pair with the real high resolution image RH1or the virtual high resolution image VH1. The discriminator44D identifies whether the input image is the real high resolution image RH1or the virtual high resolution image VH1. The low resolution image LQ1may not be input to the discriminator44D.

According to the second embodiment, it is possible to obtain the generative model (generator42G) that generates the virtual high resolution image VH1from the low resolution image LQ1. In a case in which the generator42G generated in the second embodiment is incorporated in the super resolution image generating device10, the addition unit16shown inFIG.6can be omitted.

Third Embodiment

FIG.20is a functional block diagram schematically showing a flow of processing by the learning system20according to a third embodiment. InFIG.20, the portions common to or similar to the configurations shown inFIG.19are designated by the same reference numerals, and detailed description thereof will be omitted. In the third embodiment shown inFIG.20, the input to the generator42G is a thick slice image LK, and the output from the generator42G is the virtual high resolution image VH. In this case, the pair of the thick slice image LK and the real high resolution image RH is the learning data. InFIG.20, the thick slice image LK is an example of the “first image” and the “first learning image” in the present disclosure.

According to the third embodiment, the generator42G is learned to generate the virtual high resolution image VH1from the thick slice image LK. Therefore, by performing the learning of the third embodiment, it is possible to obtain the generative model (generator42G) that generates the virtual high resolution image VH1from the thick slice image LK.

Fourth Embodiment

FIG.21is a functional block diagram schematically showing a flow of processing by the learning system20according to a fourth embodiment. InFIG.21, the portions common to or similar to the configurations shown inFIG.19are designated by the same reference numerals, and detailed description thereof will be omitted. The learning system20according to the fourth embodiment shown inFIG.21inputs the thick slice image LK to the generator42G, outputs the high-frequency component image VHFC from the generator42G, inputs the high-frequency component image to the discriminator44D, and performs the identification. The learning data generating unit30comprises a high-frequency component extracting unit33in order to produce the high-frequency component image for learning used for input of the discriminator44D.

The high-frequency component extracting unit33extracts the high-frequency components from the real high resolution image RH and generates a real high-frequency component image RHFC. Extraction of the high-frequency components is performed by using a high-pass filter. Similar to the real high resolution image RH, the real high-frequency component image RHFC has the slice interval of 1 mm and the slice thickness of 1 mm.

In the fourth embodiment, a pair of the thick slice image LK and the real high-frequency component image RHFC is the learning data. InFIG.21, the real high-frequency component image RHFC is an example of the “second learning image” in the present disclosure.

The real high-frequency component image RHFC generated by the high-frequency component extracting unit33is input to the discriminator44D of the learning unit40.

The generator42G of the learning unit40generates a virtual high-frequency component image VHFC having the same resolution as the real high-frequency component image RHFC from the input thick slice image LK. Here, the generator42G generates the virtual high-frequency component image VHFC having the slice interval of 1 mm and the slice thickness of 1 mm.

The pair of the real high-frequency component image RHFC and the thick slice image LK or the pair of a virtual high resolution image VHFC derived from the output of the generator42G and the thick slice image LK is input to the discriminator44D.

The discriminator44D identifies whether the input high-frequency component image is the real high-frequency component image RHFC or the virtual high-frequency component image VHFC.

According to the fourth embodiment, the generator42G is learned to generate the high-frequency component image from the thick slice image LK which is the low resolution image. As described with reference toFIG.14, the high resolution image can be obtained by adding processing of the high-frequency component image generated by the generator42G and the thick slice image LK which is the input of the generator42G.

Example of Hardware Configuration of Computer

FIG.22is a block diagram showing an example of a computer hardware configuration, which is used in the learning system20. A computer500may be a personal computer, may be a workstation, or may be a server computer. The computer500can be used as any one of the super resolution image generating device10, the image storage unit24, the learning data generating unit30, and the learning unit40, or a device having a plurality of functions thereof.

The computer500comprises a communicating unit512, a storage514, an operating unit516, a central processing unit (CPU)518, a graphics processing unit (GPU)519, a random access memory (RAM)520, a read only memory (ROM)522, and a display unit524. The graphics processing unit (GPU)519may be omitted.

The communicating unit512is an interface that performs communication processing with an external device by wire or wirelessly and exchanges information with the external device.

The storage514is configured to include, for example, a hard disk apparatus, an optical disk, a magneto-optical disk, or a semiconductor memory, or a storage device configured by using an appropriate combination thereof. The storage514stores various programs, data, and the like required for the image processing such as the learning processing and/or the image generation processing. The program stored in the storage514is loaded into the RAM520, and the CPU518executes the program, so that the computer functions as means for performing various processing specified by the program.

The operating unit516is an input interface that receives various operation inputs with respect to the computer500. The operating unit516may be, for example, a keyboard, a mouse, a touch panel, an operation button, a voice input device, or an appropriate combination thereof.

The CPU518reads out various programs stored in the ROM522, the storage514, and the like, and executes various processing. The RAM520is used as a work region of the CPU518. Further, the RAM520is used as a storage unit that temporarily stores the read program and various data.

The display unit524is an output interface on which various types of information are displayed. The display unit524may be, for example, a liquid crystal display, an organic electro-luminescence (OEL) display, a projector, or an appropriate combination thereof.

About Program Causing Computer to Operate

The program that causes the computer to realize a part or all of at least one processing functions among the learning data generation function, the learning function, or the image generation function described in each of the embodiments can be recorded on the computer-readable medium which is the tangible non-temporary information storage medium such as the optical disk, the magnetic disk, the semiconductor memory, or other objects, and the program can be provided through the information storage medium.

Further, instead of the aspect in which the program is stored in such a tangible non-temporary information storage medium and provided, the program signal can be provided as a download service by using a telecommunication line such as the Internet.

Further, a part or all of at least one processing function among the learning data generation function, the learning function, or the image generation function described in each of the embodiments are provided as an application server, and services that provide the processing function can be performed through the telecommunication line.

The computer that functions as the learning data generating unit30is understood as a learning data generating device. The computer that functions as the learning unit40is understood as a learning device.

About Hardware Configuration Of Each Processing Unit

The hardware structures of the processing units that execute various processing, such as the interpolation processing unit12, the generator14, and the addition unit16inFIG.6, the image storage unit24, the learning data generating unit30, the learning unit40, the GAN41, the generator42G, the discriminator44D, the error calculating unit50, and the optimizer52inFIG.7, the fixed-size region cutout unit31, the down-sampling processing unit32, the up-sampling processing unit34, the interpolation processing unit35, and the Gaussian filter processing unit36inFIG.8, and the high-frequency component extracting unit33inFIG.21, are various processors as follows.

The various processors include the CPU that is a general-purpose processor executing the program and functioning as the various processing units, the GPU that is a processor specialized in the image processing, a programmable logic device (PLD) that is a processor whose circuit configuration can be changed after manufacture, such as a field programmable gate array (FPGA), and a dedicated electric circuit that is a processor having a circuit configuration that is designed for exclusive use in order to execute specific processing, such as an application specific integrated circuit (ASIC).

One processing unit may be configured by one of these various processors or may be configured by two or more processors of the same type or different types. For example, one processing unit may be configured by a plurality of FPGAs, a combination of the CPU and the FPGA, or a combination of the CPU and the GPU. Alternatively, a plurality of processing units may be configured by one processor. As an example of configuring a plurality of processing units with one processor, first, as represented by a computer such as a client or a server, there is a form in which one processor is configured by a combination of one or more CPUs and the software and the processor functions as a plurality of processing units. Second, as represented by a system on chip (SoC), there is a form in which a processor is used in which the function of the entire system including a plurality of processing units is realized by a single integrated circuit (IC) chip. As described above, the various processing units are configured by one or more of the above various processors as a hardware structure.

Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit (circuitry) in which the circuit elements such as semiconductor elements are combined.

Others

Here, the learning method of the super resolution generative model of the CT image has been described, but the learning method of the generative model according to the present disclosure is not applied only to the CT image, and can also be applied to various three-dimensional tomographic images. For example, the learning method may be applied to a magnetic resonance (MR) image acquired by a magnetic resonance imaging (MRI) device, a positron emission tomography (PET) image acquired by a PET device, an optical coherence tomography (OCT) image acquired by an OCT device, a three-dimensional ultrasound image acquired by a three-dimensional ultrasound imaging device, and the like.

Further, the learning method of the generative model according to the present disclosure is not applied only to the three-dimensional tomographic image, and can also be applied to various a two-dimensional image. For example, the learning method may be applied to an X-ray image. Further, the learning method is not applied only to the medical image, and can be applied to a normal camera image.

The technical scope of the present invention is not limited to the scope of the embodiments described above. The configurations and the like in the embodiments can be appropriately combined between the respective embodiments without departing from the spirit of the present invention.

EXPLANATION OF REFERENCES

10: super resolution image generating device12: interpolation processing unit14: generator16: addition unit20: learning system24: image storage unit30: learning data generating unit31: fixed-size region cutout unit32: down-sampling processing unit33: high-frequency component extracting unit34: up-sampling processing unit35: interpolation processing unit36: Gaussian filter processing unit38: learning data storage unit40: learning unit41: generative adversarial network (GAN)42G: generator44D: discriminator46: addition unit50: error calculating unit52: optimizer70N: operation state70P: operation state500: computer512: communicating unit514: storage516: operating unit518: CPU519: GPU520: RAM522: ROM524: display unitC01to C05: convolutional layerC1to C10: convolutional layerGT1: imageGT2: imageIM1to IM4: CT imageIPT: interpolation imageLI1to LI4: imageLK, LK1: thick slice imageLQ, LQ1: low resolution imageORH, ORH1: real high resolution imageRH, RH1: real high resolution imageRHFC: real high-frequency component imageSAFM: self-attention feature mapSD: slice intervalST: slice thicknessTCK: thick slice imageVH, VH1: virtual high resolution imageVHA1: imageVHN2: imageVHA3: imageVHA4: imageVHFC: high-frequency component imageVT: virtual thin slice imageS1to S5: step of learning data generating processingS11to S19: step of learning processing