Patent ID: 12260960

DETAILED DESCRIPTION

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.FIG.1is a hardware configuration diagram illustrating an outline of a diagnosis support system to which a medical image processing apparatus according to a first embodiment of the present disclosure is applied. As illustrated inFIG.1, in the diagnosis support system, a medical image processing apparatus1according to the first embodiment, a three-dimensional image capturing device2, and an image storage server3are connected via a network4in a communicable state.

The three-dimensional image capturing device2is a device that images a part as a diagnosis target of a subject to generate a three-dimensional image representing the part, and specifically, is a CT device, an MRI device, a positron emission tomography (PET) device, or the like. The three-dimensional image generated by the three-dimensional image capturing device2is transmitted to the image storage server3to be saved. In the present embodiment, the three-dimensional image capturing device2is a CT device, and a CT image of a head including the brain of the subject as a structure is generated as a three-dimensional brain image. The brain image includes a plurality of tomographic images. Further, the brain image and the tomographic image correspond to the medical image of the present disclosure.

The image storage server3is a computer that saves and manages various kinds of data, and comprises a large-capacity external storage device and software for database management. The image storage server3performs communication with other devices via the network4in a wired or wireless manner, and transmits and receives image data and the like. Specifically, the various kinds of data including image data of the brain image generated by the three-dimensional image capturing device2and image data of a standard division image representing the divided regions for the standard ASPECTS described below are acquired via the network, and are saved and managed in a recording medium such as a large-capacity external storage device. The image data storage format and the communication between the devices via the network4are based on a protocol such as Digital Imaging and Communication in Medicine (DICOM).

The medical image processing apparatus1is obtained by installing a medical image processing program of the first embodiment in one computer. The computer may be a workstation or a personal computer that a doctor performing a diagnosis operates directly, or a server computer connected to the workstation or personal computer via a network. The medical image processing program may be stored in a storage device of the server computer connected to the network or in a network storage in a state of being accessible from the outside, and may be downloaded and installed in a computer in response to a request. The medical image processing program is distributed by being recorded on a recording medium such as a digital versatile disc (DVD) or a compact disc read only memory (CD-ROM), and is installed to a computer from the recording medium.

FIG.2is a diagram illustrating a schematic configuration of the medical image processing apparatus according to the first embodiment, which is realized by installing the medical image processing program in a computer. As illustrated inFIG.2, the medical image processing apparatus1comprises a central processing unit (CPU)11, a memory12, and a storage13as a standard workstation configuration. In addition, a display14such as a liquid crystal display, and an input device15such as a keyboard and a mouse are connected to the medical image processing apparatus1.

The storage13consists of a hard disk drive or the like, and various kinds of information including the brain images of the subject and information required for the process which are acquired from the image storage server3via the network4are stored.

In the memory12, the medical image processing program is stored. The medical image processing program defines, as the process executed by the CPU11, an image acquisition process of acquiring the medical image, a division process of dividing a structure in the medical image including an axisymmetric structure into a plurality of predetermined regions, a reference line derivation process of deriving a reference line of the structure on the basis of the plurality of divided regions, a normalization process of generating a normalized medical image by normalizing a position of the brain included in the medical image on the basis of the reference line, an inversion process of generating an inverted image obtained by inverting the normalized medical image using the reference line as a reference, a discrimination process of discriminating an abnormality of the structure using the normalized medical image and the inverted image, and a display control process of causing the display14to display the discrimination result. In the present embodiment, the medical image is the brain image, and the structure is the brain.

With the CPU11executing those processes according to the program, the computer functions as an image acquisition unit21, a division unit22, a reference line derivation unit23, a normalization unit24, an inversion unit25, a discrimination unit26, and a display control unit27.

The image acquisition unit21acquires a brain image B0of the subject from the image storage server3. In a case where the brain image B0is already stored in the storage13, the image acquisition unit21may acquire the brain image B0from the storage13. In the present embodiment, the ASPECTS described later is derived. Therefore, in the present embodiment, only two tomographic images for deriving the ASPECTS may be acquired among three-dimensional brain images B0. In the present embodiment, the standard division image representing the divided regions for the standard ASPECTS described later is also acquired from the image storage server3.

FIG.3is a diagram illustrating two tomographic images for deriving the ASPECTS. Two tomographic images S1and S2illustrated inFIG.3respectively represent a tomographic plane at a basal ganglia level and a tomographic plane at a corona radiata level in a middle cerebral artery region of the brain. The head of a human body included in the two tomographic images S1and S2are not at the center thereof, and the midline that divides the brain into the left brain and the right brain is inclined with respect to the perpendicular line of the tomographic image. This is because patients with the cerebral infarction are often unconscious and often more urgent, so that the imaging is performed in a hurry while the patient is on a stretcher. Further, in a case where there is a disability in the cervical spine, it is better not to move the cervical spine, but the unconscious patient is unable to answer an inquiry about the disability in the cervical spine. In such a case, since the imaging is performed without moving the head, the midline is inclined with respect to the perpendicular line of the tomographic image. In the present embodiment, the midline of the brain is used as the reference line. The tomographic images S1and S2are images of the tomographic plane seen from the lower side of the human body to the parietal side, and the face is located on the upper side. Therefore, in the brain included in the tomographic images S1and S2, the left brain is on the right side, and the right brain is on the left side.

The division unit22divides the structure in the medical image including the brain as the axisymmetric structure, into a plurality of predetermined regions. In the present embodiment, the medical images are two tomographic images for deriving the ASPECTS included in the brain image B0, the axisymmetric structure is the brain, and the brain is divided into a plurality of regions for deriving the ASPECTS.

FIG.4is a diagram illustrating the standard division image representing the divided regions of the ASPECTS. The ASPECTS is an abbreviation for the Alberta Stroke Program Early CT Score, and is a scoring method that quantifies the early CT sign of simple CT for cerebral infarction in the middle cerebral artery region. Specifically, the ASPECTS is a method in which, in a case where the medical image is the CT image, the middle cerebral artery region is divided into 10 regions in two typical sections (the basal ganglia level and the corona radiata level), the presence or absence of early ischemic changes is evaluated for each region, and a positive part is scored by a point deduction method. In a standard division image D1, each of the left and right middle cerebral artery regions in the tomographic plane at the basal ganglia level of the brain is divided into seven regions of C, I, L, IC, and M1to M3. In a standard division image D2, each of the left and right middle cerebral artery regions in the tomographic plane at the corona radiata level is divided into three regions of M4to M6. InFIG.4, for the simplicity of the description, the reference numeral is illustrated only in the regions of the left brain.

In the present embodiment, the division unit22divides the tomographic image S1at the basal ganglia level of the brain into seven regions of C, I, L, IC, and M1to M3of each of the left and right middle cerebral artery regions, and divides the tomographic image S2at the corona radiata level into three regions of M4to M6of each of the left and right middle cerebral artery regions. For this purpose, the division unit22aligns the tomographic image S1with the standard division image D1illustrated inFIG.4, and aligns the tomographic image S2with the standard division image D2illustrated inFIG.4. Then, the divided regions in the aligned standard division images D1and D2are applied to the tomographic images S1and S2, and the tomographic images S1and S2are divided into a plurality of regions.FIG.5is a diagram illustrating the tomographic images S1and S2divided into a plurality of regions.

The reference line derivation unit23derives a reference line of the brain on the basis of the plurality of divided regions in the tomographic images S1and S2. In the present embodiment, the midline of the brain is the reference line. In order to derive the reference line, the reference line derivation unit23derives the centroid of each of the left brain and the right brain in the tomographic images S1and S2. Since the process of deriving the reference line is the same for each of the tomographic images S1and S2, only the derivation of the centroid for the tomographic image S2will be described, and the derivation of the centroid for the tomographic image S1will be omitted.

FIG.6is a diagram for describing the derivation of the centroid. As illustrated inFIG.6, the reference line derivation unit23derives centroids GL4to GL6of the regions M4to M6of the left brain and centroids GR4to GR6of the regions M4to M6of the right brain of the tomographic image S2. Further, the reference line derivation unit23derives a centroid GL of the centroids GL4to GL6of the left brain, and a centroid GR of the centroids GR4to GR6of the right brain. The reference line derivation unit23may derive the centroid of the regions M4to M6in each of the left brain and the right brain as the centroid GL of the left brain and the centroid GR of the right brain without deriving the centroids GL4to GL6and the centroids GR4to GR6.

As illustrated inFIG.7, the reference line derivation unit23derives a perpendicular bisector of the centroids GL and GR as a reference line BL. The reference line derivation unit23also derives a midpoint C0of the centroids GL and GR. For the tomographic image S1, the reference line derivation unit23derives the centroid GL and GR of the left brain and the right brain, and derives a perpendicular bisector of the centroids GL and GR as the reference line BL.

The normalization unit24normalizes the position of the brain included in the tomographic images S1and S2. For this purpose, the normalization unit24translates the midpoint C0of the centroids GL and GR of the brain so that the midpoint C0coincides with the center of the tomographic images S1and S2.FIG.8is a diagram illustrating the tomographic image S2of which the midpoint C0coincides with the center. In this state, the reference line BL is inclined by θ degrees clockwise with respect to a perpendicular line XL passing through the center of the tomographic image S2. Therefore, the normalization unit24rotates the brain included in the tomographic image S2counterclockwise by θ degrees using the midpoint C0as the center to cause the reference line BL to coincide with the perpendicular line XL of the tomographic image S2. In this manner, as illustrated inFIG.9, the position of the brain in the tomographic image S2is normalized. The normalization unit24normalizes the position of the brain of the tomographic image S1in the same manner as the tomographic image S2. In the following description, the tomographic images S1and S2that are normalized are referred to as normalized tomographic images Ss1and Ss2.

The inversion unit25generates inverted tomographic images obtained by inverting the normalized tomographic images Ss1and Ss2horizontally using the reference line BL as the reference.FIG.10is a diagram illustrating inverted tomographic images Sc1and Sc2.

The discrimination unit26discriminates a disease region of the brain using each of the normalized tomographic images Ss1and Ss2and the inverted tomographic images Sc1and Sc2. In the present embodiment, an infarction region is discriminated as the disease region of the brain. For this purpose, it is assumed that the discrimination unit26has a discrimination model consisting of a convolutional neural network (hereinafter, referred to as CNN) which is one of multi-layer neural networks in which a plurality of processing layers are hierarchically connected to each other and deep learning is performed.

FIG.11is a conceptual diagram illustrating a process performed by a discrimination model of the discrimination unit26together with a configuration of the discrimination model in the first embodiment.FIG.11illustrates only the normalized tomographic image Ss1and the inverted tomographic image Sc1, but the same applies to the normalized tomographic image Ss2and the inverted tomographic image Sc2. A discrimination model30illustrated inFIG.11consists of a CNN having an encoder30A and a decoder30B. The normalized tomographic image Ss1and the inverted tomographic image Sc1are input to the encoder30A.

The encoder30A has a plurality of processing layers including at least one of a convolutional layer or a pooling layer. In the present embodiment, the processing layers of the encoder30A have both the convolutional layer and the pooling layer. The convolutional layer performs a convolution process using various kernels on the two input images (that is, the normalized tomographic image Ss1and the inverted tomographic image Sc1, and the normalized tomographic image Ss2and the inverted tomographic image Sc2) so as to detect the infarction region on the basis of the difference in pixel values of corresponding pixel positions, and outputs at least one feature map consisting of feature data obtained in the convolution process. The kernel has an n×n pixel size (for example, n=3), and a weight is set in each element. Specifically, a weight such as a differential filter for emphasizing the edge of the input image is set. The convolutional layer applies the kernel to the input image or the entire feature map output from the processing layer at the former stage while shifting the attention pixel of the kernel. Further, the convolutional layer applies an activation function such as a sigmoid function to a convolved value to output the feature map. Here, by using the difference in pixel values of the corresponding pixel positions of the two input images, the infarction region is detected using the symmetry using the reference line in the brain as the reference.

The pooling layer reduces the feature map by pooling the feature map output by the convolutional layer, and outputs the reduced feature map.

Then, the encoder30A specifies the infarction region in the feature map by repeating the convolution and pooling.

The decoder30B has a plurality of convolutional layers and upsampling layers. The convolutional layer performs the same process as the convolutional layer of the encoder30A. The upsampling layer performs upsampling of the feature map to output an enlarged feature map. Then, the decoder30B performs a process of classifying each pixel in the normalized tomographic images Ss1and Ss2into a pixel in the infarction region and a pixel that is not in the infarction region while increasing the resolution of the feature map output by the encoder30A such that the feature map has a resolution of the normalized tomographic images Ss1and Ss2. In this manner, a discrimination result of the infarction region in the normalized tomographic images Ss1and Ss2is output from the final layer of the decoder30B which is the final layer of the discrimination model30.

In a case where the normalized tomographic images Ss1and Ss2and the inverted tomographic images Sc1and Sc2are input, the discrimination model30performs learning so as to discriminate the infarction region in the normalized tomographic images Ss1and Ss2. For the learning, as illustrated inFIG.12, a large number of combinations of learning images40and ground truth data41in which the infarction regions in the learning images40are labeled are used.

In a case of the learning, an inverted image of the learning image40(referred to as a learning inverted image) is generated. Then, the learning image40and the learning inverted image are input to the CNN constituting the discrimination model30, and the discrimination result of the infarction region is output from the CNN. The discrimination result of the infarction region is compared with the ground truth data41, and the difference with the ground truth data is derived as a loss. Further, the learning of the CNN constituting the discrimination model30is performed using a large number of learning images40and the ground truth data41such that the loss is equal to or less than a predetermined threshold value. Specifically, the learning of the CNN is performed by repeatedly deriving the number of convolutional layers, the number of pooling layers, the kernel coefficient and the kernel size in the convolutional layer, and the like which constitute the CNN each time a loss is derived, such that the loss is equal to or less than the predetermined threshold value. In this manner, the discrimination model30is constructed which discriminates an infarction region32in the normalized tomographic images Ss1and Ss2in a case where the normalized tomographic images Ss1and Ss2and the inverted tomographic images Sc1and Sc2are input to the discrimination model30.

The display control unit27causes the display14to display the discrimination result of the infarction region.FIG.13is a diagram illustrating a discrimination result display screen. As illustrated inFIG.13, on a discrimination result display screen50, the normalized tomographic images Ss1and Ss2are displayed. Further, labels51and52are assigned to the infarction regions discriminated in the normalized tomographic images Ss1and Ss2. In the present embodiment, the ASPECTS55is displayed on the discrimination result display screen50. The ASPECTS55includes a table in which a check mark is to be assigned to each of the 10 regions C, I, L, IC, and M1to M6, for which the ASPECTS55is determined, in the normalized tomographic images Ss1and Ss2. An operator determines the position of the infarction region, and assigns the check mark in the ASPECTS55on the discrimination result display screen50. On the discrimination result display screen50illustrated inFIG.13, the labels51and52specifying the infarction region are respectively assigned to the region M2of the left brain of the normalized tomographic image Ss1and the region M5of the left brain of the normalized tomographic image Ss2. Therefore, the operator assigns the check mark to each of the regions M2and M5of the left brain. Since the infarction region is not included in the right brain, there is no check mark assigned in the table for the right brain. As a result, the ASPECTS of the right brain is 10, and the ASPECTS of the left brain is 8.

Next, the process performed in the first embodiment will be described.FIG.14is a flowchart illustrating the process performed in the first embodiment. First, the image acquisition unit21acquires the tomographic images S1and S2included in the brain image B0(Step ST1). Next, the division unit22divides the brain included in the tomographic images S1and S2into a plurality of predetermined regions (Step ST2). The reference line derivation unit23derives the reference line BL of the brain on the basis of the plurality of divided regions in the tomographic images S1and S2(Step ST3). Further, the normalization unit24normalizes the position of the brain included in the tomographic images S1and S2(Step ST4). In this manner, the normalized tomographic images Ss1and Ss2are generated. Next, the inversion unit25inverts the normalized tomographic images Ss1and Ss2horizontally using the reference line BL as the reference (Step ST5). In this manner, the inverted tomographic images Sc1and Sc2are generated.

The discrimination unit26discriminates the disease region of the brain using the normalized tomographic images Ss1and Ss2and the inverted tomographic images Sc1and Sc2(Step ST6). Then, the display control unit27causes the display14to display the discrimination result (Step ST7), and the process is ended.

In this manner, in the first embodiment, the structure in the medical image including the axisymmetric structure is divided into the plurality of predetermined regions, and the reference line of the structure is derived on the basis of the plurality of divided regions. Specifically, each of the left brain and the right brain included in the tomographic images S1and S2is divided into 10 regions based on the ASPECTS, and the reference line BL is derived on the basis of the plurality of divided regions. In this manner, in the present embodiment, since the reference line BL is derived on the basis of the plurality of regions in the brain, the reference line can be derived more reliably and accurately as compared with the method of deriving the midline using only the eyeballs as in the method disclosed in JP2019-500110A. Further, the burden on the operator can also be reduced as compared with the method disclosed in JP2011-167333A.

Since the position of the brain included in the tomographic images S1and S2is normalized on the basis of the derived reference line BL, the normalized tomographic images Ss1and Ss2in which the position of the brain is more accurately normalized can be generated.

Since the normalized tomographic images Ss1and Ss2which are normalized on the basis of the derived reference line BL are inverted, the inverted tomographic images Sc1and Sc2that have been more accurately inverted horizontally can be generated.

Hereinafter, a second embodiment of the present disclosure will be described.FIG.15is a diagram illustrating a schematic configuration of a medical image processing apparatus according to the second embodiment of the present disclosure. InFIG.15, the same reference numerals are given to the same configurations as those inFIG.2, and the detailed description thereof will be omitted. A medical image processing apparatus1A according to the second embodiment is different from the first embodiment in that the medical image processing apparatus1A comprises a discrimination unit29that discriminates an abnormality of the brain using the normalized tomographic images Ss1and Ss2, instead of the inversion unit25and the discrimination unit26of the medical image processing apparatus1in the first embodiment.

FIG.16is a schematic block diagram illustrating a configuration of a discrimination model of the discrimination unit29in the second embodiment. A discrimination model60of the discrimination unit29illustrated inFIG.16has an encoder60A and a decoder60B. The discrimination model60in the second embodiment performs learning in the same manner as the discrimination model30, by using a large number of learning images and ground truth data such that the discrimination result of the infarction region in the normalized tomographic images Ss1and Ss2is output in a case where the normalized tomographic images Ss1and Ss2are input, but is different from the discrimination model30in that the inverted tomographic images Sc1and Sc2are internally generated. For this purpose, the encoder60A has a first discriminator61, a second discriminator62, and a third discriminator63.

The first discriminator61consists of a convolutional neural network having a plurality of processing layers including at least one of the convolutional layer or the pooling layer, and performs at least one of the convolution process or the pooling process in each processing layer to output a feature map F1.FIG.17is a diagram illustrating an example of the feature map F1output from the first discriminator61. InFIG.17, for the simplicity of the description, the resolution of the feature map F1is set to 5×5 pixels, but the disclosure is not limited thereto. Here, in a case where the normalized tomographic image Ss1includes the infarction region at the same position as the normalized tomographic image Ss1illustrated inFIG.13, the resolution of the feature map F1is 5×5 pixels, and a feature A1is included at a position corresponding to the infarction region of the normalized tomographic image Ss1as illustrated inFIG.17.

The second discriminator62generates an inverted feature map F2by inverting the feature map F1, which is output by the first discriminator61, using the axis of symmetry thereof as the reference. The axis of symmetry corresponds to the reference line BL output by the reference line derivation unit23. For this purpose, the processing layer of the second discriminator62performs the convolution process of inverting the feature map F1using the axis of symmetry as the reference.FIG.18is a diagram illustrating the inverted feature map. As illustrated inFIG.18, the inverted feature map F2is generated by inverting the feature map F1illustrated inFIG.17horizontally using an axis of symmetry X0as the reference. Therefore, a feature A1of the feature map F1is present as the feature A2of the inverted feature map F2. The second discriminator62may have only one processing layer or may have a plurality of processing layers as long as the inverted feature map F2can be generated from the feature map F1.

The third discriminator63consists of a convolutional neural network having a plurality of processing layers including at least one of the convolutional layer or the pooling layer, and superimposes the feature map F1output by the first discriminator61and the inverted feature map F2output by the second discriminator62on each other to generate a superimposition map in the first processing layer. InFIG.16, for the description of superimposition, the first processing layer of the third discriminator63is indicated by a positive sign separately from the third discriminator63.FIG.19is a diagram illustrating the superimposition map. The third discriminator63discriminates the infarction region in the normalized tomographic images Ss1and Ss2on the basis of a superimposition map F3. Specifically, a process of specifying the infarction region is performed on the basis of the superimposition map F3.

The decoder60B performs a process of classifying each pixel in the normalized tomographic images Ss1and Ss2into a pixel in the infarction region and a pixel that is not in the infarction region while increasing the resolution of the feature map, in which the infarction region is specified, such that the feature map has a resolution of the normalized tomographic images Ss1and Ss2. In this manner, a discrimination result of the infarction region in the normalized tomographic images Ss1and Ss2is output from the final layer of the decoder60B which is the final layer of the discrimination model60.

Next, the process performed in the second embodiment will be described.FIG.20is a flowchart illustrating the process performed in the second embodiment. First, the image acquisition unit21acquires the tomographic images S1and S2included in the brain image B0(Step ST11). Next, the division unit22divides the brain included in the tomographic images S1and S2into a plurality of predetermined regions (Step ST12). The reference line derivation unit23derives the reference line BL of the brain on the basis of the plurality of divided regions in the tomographic images S1and S2(Step ST13). Further, the normalization unit24normalizes the position of the brain included in the tomographic images S1and S2(Step ST14). In this manner, the normalized tomographic images Ss1and Ss2are generated.

The discrimination unit29discriminates the disease region of the brain using the normalized tomographic images Ss1and Ss2(Step ST15). Then, the display control unit27causes the display14to display the discrimination result (Step ST16), and the process is ended.

Hereinafter, a third embodiment of the present disclosure will be described. Since a configuration of a medical image processing apparatus according to the third embodiment of the present disclosure is the same as that of the medical image processing apparatus1according to the first embodiment illustrated inFIG.2, except that a configuration of the discrimination model of the discrimination unit26is different, the detailed description for the configuration is omitted.

FIG.21is a conceptual diagram illustrating a process performed by the discrimination model of the discrimination unit26together with a configuration of the discrimination model in the third embodiment.FIG.21illustrates only the normalized tomographic image Ss1and the inverted tomographic image Sc1, but the same process is performed on the normalized tomographic image Ss2and the inverted tomographic image Sc2. A discrimination model70illustrated inFIG.21consists of a CNN having an encoder70A and a decoder70B. The encoder70A has a first discriminator71, a second discriminator72, and a third discriminator73.

Similar to the first discriminator61in the second embodiment, the first discriminator71consists of a convolutional neural network having a plurality of processing layers including at least one of the convolutional layer or the pooling layer. The first discriminator71performs at least one of the convolution process or the pooling process in each processing layer to output a feature map F11for the normalized tomographic image Ss1.

The second discriminator72consists of a convolutional neural network having a plurality of processing layers including at least one of the convolutional layer or the pooling layer. Parameters such as the weight of the kernel in each processing layer of the second discriminator72are common to those of the first discriminator71. Accordingly, the first discriminator71and the second discriminator72are substantially the same discriminator. The second discriminator72performs at least one of the convolution process or the pooling process in each processing layer to output a feature map F12for the inverted tomographic image Sc1.

The third discriminator73consists of a convolutional neural network having a plurality of processing layers including at least one of the convolutional layer or the pooling layer. The third discriminator73superimposes the feature map F11output by the first discriminator71and the inverted feature map F12output by the second discriminator72on each other to generate a superimposition map F13in the first processing layer. InFIG.21, for the description of superimposition, the first processing layer of the third discriminator73is indicated by a positive sign separately from the third discriminator73. The third discriminator73discriminates the infarction region in the normalized tomographic images Ss1and Ss2on the basis of a superimposition map F13. Specifically, a process of specifying the infarction region is performed on the basis of the superimposition map F13.

The decoder70B performs a process of classifying each pixel in the normalized tomographic images Ss1and Ss2into a pixel in the infarction region and a pixel that is not in the infarction region while increasing the resolution of the feature map, in which the infarction region is specified, such that the feature map has a resolution of the normalized tomographic images Ss1and Ss2. In this manner, a discrimination result of an infarction region75in the normalized tomographic images Ss1and Ss2is output from the final layer of the decoder70B which is the final layer of the discrimination model70.

Next, the process performed in the third embodiment will be described.FIG.22is a flowchart illustrating the process performed in the third embodiment. First, the image acquisition unit21acquires the tomographic images S1and S2included in the brain image B0(Step ST21). Next, the division unit22divides the brain included in the tomographic images51and S2into a plurality of predetermined regions (Step ST22). The reference line derivation unit23derives the reference line BL of the brain on the basis of the plurality of divided regions in the tomographic images S1and S2(Step ST23). Further, the normalization unit24normalizes the position of the brain included in the tomographic images S1and S2(Step ST24). In this manner, the normalized tomographic images Ss1and Ss2are generated. Next, the inversion unit25inverts the normalized tomographic images Ss1and Ss2horizontally using the reference line BL as the reference (Step ST25). In this manner, the inverted tomographic images Sc1and Sc2are generated.

The discrimination unit26discriminates the disease region of the brain using the normalized tomographic images Ss1and Ss2and the inverted tomographic images Sc1and Sc2(Step ST26). Then, the display control unit27causes the display14to display the discrimination result (Step ST27), and the process is ended.

In the third embodiment, the third discriminator73may have the function of the decoder70B. In this case, at the former stage of the third discriminator73, a process is performed which discriminates the infarction region in the normalized tomographic images Ss1and Ss2on the basis of the superimposition map F13. Then, at the latter stage of the third discriminator73, a process is performed which classifies each pixel in the normalized tomographic images Ss1and Ss2into a pixel in the infarction region and a pixel that is not in the infarction region while increasing the resolution of the feature map, in which the infarction region is specified, such that the feature map has a resolution of the normalized tomographic images Ss1and Ss2.

In the third embodiment, the feature map F11and the inverted feature map F12are superimposed on each other in the first processing layer of the third discriminator73, but the disclosure is not limited thereto. A difference map representing the difference between the feature map F11and the inverted feature map F12may be generated. In this case, the third discriminator73discriminates the infarction region in the normalized tomographic images Ss1and Ss2on the basis of the difference map.

Hereinafter, a fourth embodiment of the present disclosure will be described. Since a configuration of a medical image processing apparatus according to the fourth embodiment of the present disclosure is the same as that of the medical image processing apparatus1according to the first embodiment illustrated inFIG.2, except that the process to be performed is different, the detailed description for the configuration is omitted.

FIG.23is a conceptual diagram illustrating a process to be performed together with a configuration of the discrimination model in the fourth embodiment.FIG.23illustrates only the normalized tomographic image Ss1, but the same process is performed on the normalized tomographic image Ss2. As illustrated inFIG.23, in the fourth embodiment, the inversion unit25generates divided normalized tomographic images Ssh1and Ssh2by dividing the normalized tomographic image Ss1into left and right using the reference line BL as the reference. The divided normalized tomographic images Ssh1and Ssh2have half the size of the normalized tomographic images Ss1and Ss2. The divided normalized tomographic image Ssh1indicates the right brain side, and the divided normalized tomographic image Ssh2indicates the left brain side. Then, the inversion unit25generates a divided inverted tomographic image Sch1by inverting any one of the divided normalized tomographic image Ssh1or Ssh2horizontally using the reference line BL as the reference. InFIG.23, the divided inverted tomographic image Sch1is generated by inverting the divided normalized tomographic image Ssh2horizontally.

In the fourth embodiment, the discrimination unit26has a discrimination model80that discriminates the disease region (that is, the infarction region) of the brain using the divided normalized tomographic image and the divided inverted tomographic image. The discrimination model80illustrated inFIG.23has an encoder80A and a decoder80B. Similar to the encoder30A in the first embodiment, the encoder80A has a plurality of processing layers including at least one of the convolutional layer or the pooling layer. The encoder80A performs the convolution process using various kernels on the basis of the difference in pixel values of the corresponding pixel positions of the divided normalized tomographic image Ssh1and the divided inverted tomographic image Sch1such that the infarction region can be detected so as to generate a feature map from the feature data obtained in the convolution process. Then, the encoder80A specifies the infarction region in the feature map. Here, by using the difference in pixel values of the corresponding pixel positions of the two input images, the infarction region is detected using the symmetry using the reference line BL in the brain as the reference.

In the fourth embodiment, since the divided normalized tomographic image Ssh1and the divided inverted tomographic image Sch1are used, different labels are assigned to the infarction region detected in the divided normalized tomographic image Ssh1and the infarction region detected in the divided inverted tomographic image Sch1. For example, a label of “1” is assigned to the infarction region detected in the divided normalized tomographic image Ssh1, that is, on the right brain side. Further, a label of “2” is assigned to the infarction region detected in the divided inverted tomographic image Sch1, that is, in the left brain side. A label of “0” is assigned to the region other than the infarction region.

Similar to the decoder30B in the first embodiment, the decoder80B has a plurality of convolutional layers and upsampling layers. The decoder80B performs a process of classifying each pixel in the normalized tomographic images Ss1and Ss2into a pixel in the infarction region and a pixel that is not in the infarction region while increasing the resolution of the feature map output by the encoder80A such that the feature map has a resolution of the normalized tomographic images Ss1and Ss2. In this manner, a discrimination result of the infarction region in the normalized tomographic images Ss1and Ss2is output from the final layer of the decoder80B which is the final layer of the discrimination model80.

Here, in the encoder80A of the fourth embodiment, the infarction region is specified by using the feature map for the image having half the size of the normalized tomographic images Ss1and Ss2. Therefore, the feature map in which the infarction region is specified is upsampled in the decoder80B, and it is necessary for the feature map to have the same size as the normalized tomographic image Ss1in a case where the infarction region is finally specified in the normalized tomographic images Ss1and Ss2. Accordingly, the decoder80B upsamples the feature map to half the resolution of the normalized tomographic images Ss1and Ss2in the processing layer at the former stage. Then, the size of the feature map is made the same as the normalized tomographic images Ss1and Ss2by interpolating the region according to the label of the detected infarction region in the upsampled feature map. Further, the decoder80B generates the feature map in which the region is interpolated and specifies the infarction region in the normalized tomographic images Ss1and Ss2, in the processing layer at the latter stage.

FIGS.24to27are diagrams for describing the interpolation of the region for the feature map. InFIGS.24to27, feature maps F21, F23, F25, and F27that are upsampled to the same resolution as the divided normalized tomographic images Ssh1and Ssh2are illustrated. As illustrated inFIG.24, in a case where the label of the infarction region detected in the feature map F21is “1”, the infarction region is detected on the right brain side. Therefore, as illustrated inFIG.24, the decoder80B generates the feature map F22having the same size as the normalized tomographic images Ss1and Ss2by interpolating a region F21A which has the same size as the feature map F21and in which a label of “0” is assigned to the entire region, in a region on the right side of the feature map F21.

As illustrated inFIG.25, in a case where the label of the infarction region detected in the feature map F23is “2”, the infarction region is detected on the left brain side. Therefore, as illustrated inFIG.25, the decoder80B generates a feature map F23A by inverting the infarction region having a label of “2” included in the feature map F23horizontally using the right side of the feature map F23as the reference. Then, a feature map F24having the same size as the normalized tomographic images Ss1and Ss2is generated by interpolating a region F23B which has the same size as the feature map F23A and in which a label of “0” is assigned to the entire region, in a region on the left side of the inverted feature map F23A.

As illustrated inFIG.26, in a case where two infarction regions are detected in the feature map F25and the labels of the two detected infarction regions are respectively “1” and “2”, the infarction region is detected in both the left and right brains. Therefore, the decoder80B interpolates a region F25A which has the same size as the feature map F25and in which a label of “0” is assigned to the entire region, in a region on the right side of the feature map F25. Further, the decoder80B inverts the infarction region having a label of “2” included in the feature map F25horizontally using the right side of the feature map F25as the reference, and assigns the label to the interpolated region F25A. In this manner, as illustrated inFIG.26, the decoder80B generates a feature map F26having the same size as the normalized tomographic images Ss1and Ss2.

In a case where there is no detected infarction region in the feature map F27as illustrated inFIG.27, the decoder80B generates a feature map F28having the same size as the normalized tomographic images Ss1and Ss2by interpolating a region F27A which has the same size as the feature map F27and in which a label of “0” is assigned to the entire region, in a region on the left side of the feature map F27, as illustrated inFIG.27.

In a case where the infarction region is detected in both the left and right brains, the region having a label of “1” and the region having a label of “2” may overlap each other in a feature map F29as illustrated inFIG.28. In this case, in the encoder80A, a label of “3” is assigned to a region where the region having a label of “1” and the region having a label of “2” overlap each other. Then, the decoder80B interpolates a region F29A which has the same size as the feature map F29and in which a label of “0” is assigned to the entire region, in a region on the right side of the feature map F29. Further, the decoder80B inverts the infarction region having labels of “2” and “3” included in the feature map F29horizontally using the right side of the feature map F29as the reference while deleting the region having a label of “2” included in the feature map F29, and assigns the labels to the interpolated region F29A. In this manner, as illustrated inFIG.28, the decoder80B generates a feature map F30which consists of a feature map F29B obtained by deleting the region having a label of “2” in the feature map F29, and a feature map F29A, and has the same size as the normalized tomographic images Ss1and Ss2.

Also in the third embodiment, similar to the fourth embodiment, the infarction region of the brain may be detected using the divided normalized tomographic image and the divided inverted tomographic image instead of the normalized tomographic images Ss1and Ss2and the inverted tomographic images Sc1and Sc2in the first embodiment.

The same method as in the fourth embodiment can be applied to the second embodiment. In a case where the same method as in the fourth embodiment is applied to the second embodiment, as illustrated inFIG.29, the feature map F1output by the first discriminator61is divided using the axis of symmetry corresponding to the reference line BL as the reference so that divided feature maps Fh31and Fh32are generated. It is assumed that the divided feature map Fh31indicates the right brain side, and the divided feature map Fh32indicates the left brain side. The second discriminator62generates a divided inverted feature map Fh33by inverting any one of the divided feature maps (the divided feature map Fh32inFIG.29) horizontally. Then, the third discriminator63generates a superimposition map or a difference map from the divided feature map Fh31and the divided inverted feature map Fh33, and detects the infarction region in both the right brain side and the left brain side using the superimposition map or the difference map. The decoder60B may generate a feature map having the same size as the normalized tomographic images Ss1and Ss2by interpolating a region in the same manner as in the decoder80B in the fourth embodiment, and may output a discrimination result of the infarction region in the normalized tomographic images Ss1and Ss2.

In the fourth embodiment, the divided inverted tomographic image Sch1is generated from the divided normalized tomographic image Ssh2on the left brain side, but the divided inverted tomographic image may be generated from the divided normalized tomographic image Ssh1on the right brain side. In this case, the divided inverted tomographic image generated from the divided normalized tomographic image Ssh1on the right brain side and the divided normalized tomographic image Ssh2are input to the discrimination model80, and a discrimination result of the infarction region in the normalized tomographic images Ss1and Ss2is output.

In each embodiment described above, in the normalization unit24, the normalized tomographic images Ss1and Ss2are generated by normalizing the position of the brain included in the tomographic images S1and S2on the basis of the reference line BL derived by the reference line derivation unit23, but the disclosure is not limited thereto. The method is not limited to the method using the reference line BL as long as the normalized medical image can be generated by normalizing the position of the structure included in the medical image such as the tomographic images S1and S2. That is, the normalization unit24may generate the normalized medical image by normalizing the position of the structure (brain) included in the medical image (tomographic images S1and S2) without being on the basis of the reference line derived by the reference line derivation unit23. For example, the tomographic images S1and S2are displayed on the display14, and the normalized tomographic images Ss1and Ss2may be generated on the basis of an operation of the operator to normalize the position of the brain included in the tomographic images S1and S2using the input device15. The normalization unit24may generate the normalized tomographic images Ss1and Ss2in which the position of the brain included in the tomographic images S1and S2is normalized by aligning the standard image of the brain with the defined reference line, with the tomographic images S1and S2.

In each embodiment described above, the operator assigns a check mark to the ASPECTS on the discrimination result display screen, but the disclosure is not limited thereto. For example, as in a medical image processing apparatus1B illustrated inFIG.30, a derivation unit90that derives the ASPECTS may be provided to the medical image processing apparatus1according to the first embodiment. The derivation unit90determines which region for deriving the ASPECTS in the normalized tomographic images Ss1and Ss2includes the detected infarction region. That is, the derivation unit90determines which region among the regions of C, I, L, IC, and M1to M3of each of the left and right brains in the normalized tomographic image Ss1and the regions of M4to M6of each of the left and right brains in the normalized tomographic image Ss2includes the infarction region. Specifically, in a case where the infarction region is included in a region for deriving the ASPECTS in a predetermined ratio or more, it is determined that the region is the infarction region. Then, the derivation unit90assigns the check mark to the ASPECTS55illustrated inFIG.13for the region determined to be the infarction region.

Here, the predetermined ratio can be appropriately set according to the degree of strictness of the determination. For example, the predetermined ratio can be 10%, but may be 20% or 30%. Further, in a case where the infarction region is included in a region for deriving the ASPECTS even a little, the region may be determined as the infarction region.

The derivation unit90may be provided not only in the first embodiment but also in any of the second embodiment to the fourth embodiment.

Further, in each embodiment described above, the CNN is used as the discrimination model, but the disclosure is not limited thereto. As long as the neural network includes a plurality of processing layers, a deep neural network (DNN), a recurrent neural network (RNN), U-Net or the like can be used. Further, as the neural network, a neural network using Mask Regions with CNN features (R-CNN) (“Mask R-CNN”, Kaiming He et al., arXiv, 2018) may be used. Hereinafter, the Mask R-CNN will be described.

FIG.31is a schematic configuration diagram of the Mask R-CNN.FIG.31illustrates an example in which the Mask R-CNN is applied as the discrimination model30in the first embodiment. As illustrated inFIG.31, a Mask R-CNN100includes a convolutional layer101that generates a feature map F40by extracting a feature quantity from the input image; a Region Proposal Network (RPN)102that specifies candidate regions for the infarction region in the feature map F40; a classification network103that cuts out the feature map F40on the basis of the candidate regions for the infarction region, and outputs a class of the candidate region using the cut-out feature map and coordinate information of the candidate region in the normalized tomographic images Ss1and Ss2; and a segmentation104that specifies the infarction region in the normalized tomographic images Ss1and Ss2using a pixel level.

Similar to the encoder in each embodiment described above, the convolutional layer101performs the convolution process using various kernels on the input normalized tomographic images Ss1and Ss2and the input inverted tomographic images Sc1and Sc2, and outputs the feature map F40consisting of feature data obtained by the convolution process.

In the RPN102, a rectangular region called an anchor having a plurality of types of aspect ratios and sizes is defined in advance. In the RPN102, the plurality of types of anchors are applied to each pixel position of the feature map F40, and an anchor with the highest overlap rate with an object candidate included in the normalized tomographic images Ss1and Ss2is selected. In the RPN102, a process of regressing (that is, deforming and moving) the anchor so as to coincide with a rectangle (ground truth box) surrounding the object candidate using the selected anchor is performed on all the pixels of the feature map F40, and the position and size of the anchor regressed to coincide with the ground truth box are output from the RPN102as a candidate region A10of the infarction region in the input normalized tomographic images Ss1and Ss2. The candidate region A10is a rectangular region surrounding the infarction region.

The classification network103consists of fully connected layers, and performs classification of the candidate region A10in the normalized tomographic images Ss1and Ss2and derivation of the coordinate information of the candidate region A10in the normalized tomographic images Ss1and Ss2on the basis of the candidate region A10and the feature map F40.

The segmentation104consists of a fully convolutional network (FCN), segments the infarction region in the normalized tomographic images Ss1and Ss2by specifying the pixel which is the infarction region in the candidate region A10on the basis of the candidate region A10and the feature map F40.FIG.31illustrates a segmented state of an infarction region105in the normalized tomographic image Ss1.

As described above, the infarction region in the normalized tomographic images Ss1and Ss2can be specified by using the Mask R-CNN100as the discrimination model30in the first embodiment.

The Mask R-CNN can be used as the discrimination model60in the second embodiment. In this case, in the Mask R-CNN, only the normalized tomographic images Ss1and Ss2are input, and the feature map of the normalized tomographic images Ss1and Ss2and the inverted feature map thereof are generated in the convolutional layer101. Further, in the convolutional layer101, the superimposition map of the feature map and the inverted feature map thereof is generated and output. In the convolutional layer101, convolution and pooling are further performed on the superimposition map, and the superimposition map to which the convolution and pooling have been performed may be output. In this case, in the RPN102, the map output by the convolutional layer101is input, and the candidate region A10of the infarction region in the normalized tomographic images Ss1and Ss2is output. Further, in the classification network103and the segmentation104, the map output by the convolutional layer101and the candidate region A10output by the RPN102are input, and the infarction region105in the normalized tomographic images Ss1and Ss2is specified.

The Mask R-CNN can be used as the discrimination model70in the third embodiment. In this case, in the Mask R-CNN100, each of the normalized tomographic images Ss1and Ss2and the inverted tomographic images Sc1and Sc2is input, the feature map for the normalized tomographic images Ss1and Ss2and the feature map for the inverted tomographic images Sc1and Sc2(hereinafter, referred to as inverted feature map) are generated in the convolutional layer101, and a superimposition map or a difference map of the feature map and the inverted feature map thereof is generated and output. In the convolutional layer101, convolution and pooling are further performed on the superimposition map or the difference map, and the superimposition map of the difference map to which the convolution and pooling have been performed may be output. In this case, in the RPN102, the map output by the convolutional layer101is input, and the candidate region A10of the infarction region in the normalized tomographic images Ss1and Ss2is output. Further, in the classification network103and the segmentation104, the map output by the convolutional layer101and the candidate region A10output by the RPN102are input, and the infarction region105in the normalized tomographic images Ss1and Ss2is specified.

The Mask R-CNN can be used as the discrimination model80in the fourth embodiment. In this case, in the Mask R-CNN100, each of the divided normalized tomographic images Ssh1and Ssh2and the divided inverted tomographic images Sch1and Sch2is input, and a feature map focusing on the difference in pixel values of the corresponding pixel positions of the divided normalized tomographic images Ssh1and Ssh2and the divided inverted tomographic images Sch1and Sch2is output from the convolutional layer101. In this case, in the RPN102, the feature map output by the convolutional layer101is input, and the candidate region A10of the infarction region in the normalized tomographic images Ss1and Ss2is output. Further, in the classification network103and the segmentation104, the feature map output by the convolutional layer101and the candidate region A10output by the RPN102are input, and the infarction region105in the normalized tomographic images Ss1and Ss2is specified. In this case, the feature map is interpolated to have the same size as the normalized tomographic images Ss1and Ss2in the processing layer before the final layer of the classification network103and the segmentation104.

As described above, the same method as that in the fourth embodiment can be applied even in the second embodiment, and as the discrimination model used in such a case, the Mask R-CNN can be used. As described above, the same method as that in the fourth embodiment can be applied even in the third embodiment, and as the discrimination model used in such a case, the Mask R-CNN can be used.

In each embodiment described above, the tomographic images S1and S2are divided into the plurality of regions for determining the ASPECTS, but the disclosure is not limited thereto. For example, the tomographic images S1and S2may be divided into the plurality of regions by a method of dividing the brain into functional regions, such as Brodmann's brain map.

In the embodiment described above, the reference line BL of the brain included in the two-dimensional tomographic images S1and S2is derived, but the disclosure is not limited thereto. The three-dimensional brain image B0may be divided into a plurality of regions, and a reference plane corresponding to a median plane of the brain included in the three-dimensional brain image B0may be derived on the basis of the plurality of regions.

In each embodiment described above, the discrimination result of the normalized tomographic images Ss1and Ss2is displayed on the discrimination result display screen50, but the disclosure is not limited thereto. The tomographic images S1and S2before normalization may be displayed on the discrimination result display screen50. In this case, a mask for specifying the infarction region may be displayed on the tomographic images S1and S2by aligning the normalized tomographic images Ss1and Ss2including the discrimination result with the tomographic images S1and S2.

In each embodiment described above, in the reference line derivation unit23and the normalization unit24, a new normalized tomographic image may be generated by performing again the derivation of the centroids of the left brain and the right brain, the derivation of the reference line, and the normalization on the normalized tomographic images Ss1and Ss2generated by performing the derivation of the centroids of the left brain and the right brain, the derivation of the reference line, and the normalization. In this case, the derivation of the centroids of the left brain and the right brain, the derivation of the reference line, and the normalization may be further repeatedly performed on the new normalized tomographic image. In this manner, since the accuracy of the normalization can be improved, it is possible to more accurately discriminate the infarction region.

In the second embodiment, the second discriminator62of the discrimination model60generates the inverted feature map F2, and the third discriminator63of the discrimination model60generates the superimposition map of the feature map F1and the inverted feature map F2to discriminate the infarction region, but the disclosure is not limited thereto. The third discriminator63may generate the difference map by deriving the difference in corresponding pixels of the feature map F1and the inverted feature map F2, and discriminate the infarction region on the basis of the feature map F1and the difference map. Even in a case where the Mask R-CNN is used as the discrimination model60in the second embodiment, the difference map of the feature map and the inverted feature map thereof may be generated in the convolutional layer101. In this case, in the RPN102, the difference map or a map obtained by further performing the convolution and the pooling on the difference map is input.

The first discriminator61, the second discriminator62, and the third discriminator63included in the discrimination model60in the second embodiment may not be the same type of neural network. For example, the first discriminator61and the second discriminator62may be the convolutional neural network, and the third discriminator63may be the recurrent neural network instead of the CNN.

The first discriminator71, the second discriminator72, and the third discriminator73included in the discrimination model70in the third embodiment may not be the same type of neural network. For example, the first discriminator71and the second discriminator72may be the convolutional neural network, and the third discriminator73may be the recurrent neural network instead of the CNN.

In each embodiment described above, the infarction region of the brain is discriminated, but the disclosure is not limited thereto, and a bleeding region of the brain may be discriminated. In this case, the discrimination model is trained to discriminate the bleeding region of the brain.

Further, in each embodiment described above, the CT image is used as the brain image B0and the tomographic images S1and S2, but the disclosure is not limited thereto, and other medical images such as the MM image and the PET image may be used.

Further, in each embodiment described above, the brain image is used as the medical image, but the disclosure is not limited thereto. For example, the technique of the present disclosure can be applied even in a case of discriminating the disease region in the medical image including a pair or a plurality of pairs of structures present in an axisymmetric manner such as lungs, kidneys, eyeballs, and ears.

In each embodiment described above, the following various processors can be used as the hardware structure of processing units executing various processes such as the image acquisition unit21, the division unit22, the reference line derivation unit23, the normalization unit24, the inversion unit25, the discrimination unit26, the display control unit27, the discrimination unit29, and the derivation unit90. The various processors include, for example, a programmable logic device (PLD) that is a processor of which the circuit configuration can be changed after manufacture, such as a field-programmable gate array (FPGA), and a dedicated electric circuit that is a processor having a dedicated circuit configuration designed to execute a specific process, such as an application specific integrated circuit (ASIC), in addition to the CPU that is a general-purpose processor which executes software (programs) to function as various processing units as described above.

One processing unit may be configured by one of the various processors or a combination of the same or different kinds of two or more processors (for example, a combination of a plurality of FPGAs or a combination of a CPU and an FPGA). In addition, a plurality of processing units may be configured by one processor.

As an example where a plurality of processing units are configured by one processor, first, there is a form where one processor is configured by a combination of one or more CPUs and software as typified by a computer, such as a client and a server, and this processor functions as a plurality of processing units. Second, there is a form where a processor fulfilling the functions of the entire system including a plurality of processing units by means of one integrated circuit (IC) chip as typified by a system on chip (SoC) or the like is used. In this manner, various processing units are configured by using one or more of the above-described various processors as hardware structures.

In addition, specifically, an electric circuit (circuitry) obtained by combining circuit elements, such as semiconductor elements, can be used as the hardware structure of the various processors.