Medical image processing device and medical image processing program

A controller of a medical image processing device acquires a medical image of a subject. The controller causes a display to display a pre-modification image in which at least the position or range of a lesion to be modified on the medical image is displayed. The controller receives an instruction to designate at least the position or range of the lesion to be modified in a state in which the pre-modification image is displayed on the display. When at least the position or range of the lesion is designated, the controller acquires a predicted disease image in which the lesion is modified according to the designated information on the basis of the medical image. The controller causes the display to display the predicted disease image and the pre-modification image simultaneously or in a switching manner.

TECHNICAL FIELD

The present disclosure relates to a medical image processing device that processes a medical image that is an image of a tissue of a subject and a medical image processing program executed by the medical image processing device.

BACKGROUND ART

In recent years, various techniques for processing medical images to obtain useful information have been proposed. For example, an image processing device described in Patent Literature 1 acquires a diagnosis result of an eye to be examined for the purpose of obtaining an image analysis result suitable for a disease and processes an image of the eye to be examined using a processing method according to the acquired diagnosis result.

CITATION LIST

Patent Literature

Patent Literature 1: JP 2018-121885 A

SUMMARY OF INVENTION

It is useful for both a doctor and a patient if the doctor can explain, to the patient, the state of a tissue in a case in which a disease occurs, the state of the tissue in a case in which the disease progresses, the state of the tissue in a case in which the disease is cured, and the like while illustrating the subject himself/herself the image of the tissue. However, conventionally, an image of another subject in a state equivalent to the predicted patient state can only be illustrated to the subject at the time of explanation.

A typical object of the present disclosure is to provide a medical image processing device and a medical image processing program that can appropriately present an image useful in explanation from a doctor to a subject.

A medical image processing device provided by a typical embodiment of the present disclosure is a medical image processing device that processes a medical image that is an image of a tissue of a subject. A controller of the medical image processing device executes an image acquisition step of acquiring a medical image of a subject, a pre-modification image display step of displaying, on a display, an image prepared for modification in which at least one of a position and a range of a lesion to be modified on the medical image is displayed, a lesion reception step of receiving an instruction to designate at least one of a position and a range of the lesion to be modified in a state in which the pre-modification image is displayed on the display, a predicted disease image acquisition step of, in a case in which at least one of a position and a range of the lesion is designated, acquiring a predicted disease image in which the lesion is modified according to designated information on the basis of the medical image acquired in the image acquisition step, and a predicted disease image display step of displaying the acquired predicted disease image and the pre-modification image on the display simultaneously or in a switching manner.

A medical image processing program provided by a typical embodiment of the present disclosure is a medical image processing program executed by a medical image processing device that processes a medical image that is an image of a tissue of a subject. The medical image processing program is executed by a controller of the medical image processing device to execute an image acquisition step of acquiring a medical image of a subject, a pre-modification image display step of displaying, on a display, an image prepared for modification in which at least one of a position and a range of a lesion to be modified on the medical image is displayed, a lesion reception step of receiving an instruction to designate at least one of a position and a range of the lesion to be modified in a state in which the pre-modification image is displayed on the display, a predicted disease image acquisition step of, in a case in which at least one of a position and a range of the lesion is designated, acquiring a predicted disease image in which the lesion is modified according to designated information on the basis of the medical image acquired in the image acquisition step, and a predicted disease image display step of displaying the acquired predicted disease image and the pre-modification image on the display simultaneously or in a switching manner.

According to the medical image processing device and the medical image processing program according to the present disclosure, an image useful for explanation from a doctor to a subject is appropriately presented.

The controller of the medical image processing device exemplified in the present disclosure executes an image acquisition step, a pre-modification image display step, a lesion reception step, a predicted disease image acquisition step, and a predicted disease image display step. In the image acquisition step, the controller acquires a medical image of the subject. In the pre-modification image display step, the controller causes the display to display an image prepared for modification in which at least the position or range of the lesion to be modified on the medical image is displayed. In the lesion reception step, the controller receives an instruction to designate at least the position or range of the lesion to be modified in a state in which the pre-modification image is displayed on the display. In the predicted lesion image generation step, when at least the position or range of the lesion is designated, the controller acquires a predicted disease image in which the lesion is modified according to the designated information on the basis of the medical image acquired in the image acquisition step. In the predicted disease image display step, the controller causes the display to display the acquired predicted disease image and the pre-modification image simultaneously or in a switching manner.

According to the medical image processing device exemplified in the present disclosure, when the user inputs an instruction to designate at least the position or range of the lesion to be modified in a state in which the pre-modification image is displayed, the predicted disease image in which the lesion is modified according to the designated information is acquired on the basis of the medical image of the subject. The acquired predicted disease image is then displayed on the display in a state in which the pre-modification image is displayed or in a state of being switched to the pre-modification image. That is, when an instruction to designate a lesion is input, a predicted disease image in which the lesion has been modified on the medical image of the subject is acquired and displayed according to the designated information. Therefore, instead of an image of a subject different from the subject to be explained by the doctor, a predicted disease image in which a medical image of the subject himself who provides explanations is appropriately modified is presented to the subject. Accordingly, an image useful for explanation from the doctor to the subject is appropriately presented to the subject.

In the predicted disease image acquisition step, the controller acquires a predicted disease image based on the medical image of the subject actually photographed. That is, the controller acquires the predicted disease image on the basis of at least the pixel value, the structure of the tissue, the brightness, the contrast, or the like in the actually captured medical image. Therefore, it is easy for the subject to feel the predicted disease state as compared with the case in which the predicted disease image is acquired without being based on the subject's own medical image (for example, a case in which a predicted disease image is generated by normal image rendering software).

Note that the “modification of a medical image with respect to a lesion” in the present disclosure includes at least “addition” of a lesion to a region where no lesion exists, “enlargement” of a lesion, “reduction” of a lesion, “deletion” of a lesion, or the like. That is, the medical image, on which the modification concerning the lesion is to be performed, may be a medical image in which the lesion does not exist or a medical image in which the lesion already exists. In addition, the instruction to designate at least the position or range of the lesion (hereinafter simply referred to as “position/range” in some cases) may be an instruction to designate “addition” of the position/range or an instruction to designate “change” of the position/range.

It is possible to appropriately select a device that executes an image acquisition step, a pre-modification image display step, a lesion reception step, a predicted disease image acquisition step, and a predicted disease image display step. For example, the controller of a personal computer (hereinafter referred to as “PC”) may execute all the steps. That is, the controller of the PC may acquire a medical image from the medical image photographing device and execute acquisition and display of a predicted disease image based on the acquired medical image and the like. Furthermore, the controllers of a plurality of devices (for example, at least a medical image photographing device, a PC, a mobile terminal, or a server) may cooperate to execute each step.

In addition, various images can be adopted as pre-modification images. For example, a pre-modification image may be an image indicating the position and range of a lesion on a plain background. Furthermore, a pre-modification image may be an image of a blood vessel of a tissue appearing in a medical image (for example, in a case in which a medical image is a fundus image, an image of a fundus blood vascular is obtained). In this case, an image of the blood vessel may be acquired by extracting the blood vessel from the medical image or may be photographed by an angiography device different from the photographing device that photographs the medical image. In addition, the medical image itself acquired in the image acquisition step may be used as a pre-modification image. Furthermore, in a case in which the medical image is a fundus image, the pre-modification image may be a map image that two-dimensionally shows a distribution (for example, the distribution of the thicknesses of at least one of the layers) related to the structure of the fundus. In this case, the user may designate the position/region of the lesion by designating a region (for example, a region where the thickness of the layer is within a specific range) of a desired structure illustrated in the map image. Furthermore, in a case in which some detection result is acquired by a mathematical model trained by a machine learning algorithm, an attention map may be adopted as a pre-modification image. As will be described in detail later, the attention map indicates the distribution of influence degrees (attention degrees) on the respective positions of influences when a mathematical model obtains detection results in an image region.

In addition, the pre-modification image and the predicted disease image may be the same. That is, by specifying the position and range of a lesion on a medical image or a predicted disease image, the predicted disease image according to specified information may be generated in real time, and the displayed medical image or the predicted disease image may be switched to the newly generated predicted disease image.

The controller may further execute the lesion position acquisition step. In the lesion position acquisition step, when the lesion already exists in the medical image acquired in the image acquisition step, the controller acquires the detection result of the position of the lesion in the medical image. The controller may execute the processing of receiving an instruction in the lesion reception step and the processing of acquiring a predicted disease image in the predicted disease image acquisition step, with the position of the lesion acquired in the lesion position acquisition step as the position of the already existing lesion.

In this case, the user can input various instructions (for example, at least an instruction to enlarge or reduce the range of an already existing lesion, an instruction to change the position of an already existing lesion, an instruction to add a new lesion in addition to an already existing lesion, or the like) according to the position of the already existing lesion. Therefore, an appropriate predicted disease image corresponding to the position of the already existing lesion is acquired.

The controller may further execute the lesion range acquisition step. In the lesion range acquisition step, when the lesion already exists in the medical image acquired in the image acquisition step, the controller acquires the detection result of the range (for example, the shape and size of the lesion without position information) of the lesion in the medical image. The controller may execute the processing of receiving an instruction in the lesion reception step and the processing of acquiring a predicted disease image in the predicted disease image acquisition step, with the range of the lesion acquired in the lesion range acquisition step as the range of the already existing lesion.

In this case, the user can input various instructions (for example, at least an instruction to enlarge or reduce the range of an already existing lesion, an instruction to add a new lesion in addition to an already existing lesion, or the like) according to the range of the already existing lesion. Therefore, an appropriate predicted disease image corresponding to the range of the already existing lesion is acquired.

Note that, when causing the display to display an image prepared for modification in the pre-modification image display step, the controller may display at least the position or range (hereinafter simply referred to as “position/range” in some cases) of the lesion acquired on the basis of the medical image on the pre-modification image as the position/range of the already existing lesion. In this case, the user can input various instructions after appropriately grasping the position/range of the already existing lesion.

A specific method for acquiring the position/range of the already existing lesion in the medical image can be appropriately selected. For example, the controller may acquire the position/range of the lesion output by the mathematical model by inputting the medical image acquired in the image acquisition step to the mathematical model trained by the machine learning algorithm. In this case, the mathematical model may be trained in advance by a training data set in which the data of the medical image is used as the input training data and information indicating the position/range of the lesion included in the input training data is used as the output training data. By using the machine learning algorithm, the position/range of the lesion in the medical image are more appropriately acquired. Furthermore, the controller may acquire the position/range of the lesion in the medical image by executing known image processing on the medical image acquired in the image acquisition step.

However, the technology exemplified in the present disclosure is also useful in a case in which the position/range of the lesion is not detected from a medical image, a case in which the position/range of the lesion cannot be detected from a medical image, and the like. For example, the controller may execute the lesion reception step while the medical image acquired in the image acquisition step is displayed on the display. In this case, even if the position/range of the lesion is not automatically detected, the user can determine or estimate the position/range of the already existing lesion on the basis of the medical image displayed on the display and input various instructions on the basis of the determination or estimation result. In addition, in a case in which there is no lesion in the acquired medical image, a predicted disease image to which a lesion has been added according to the specified information may be acquired on the basis of the medical image.

In the lesion reception step, the controller may receive an instruction to designate the type (that is, the type of disease) of a lesion to be modified. In a case in which the type of lesion is designated, the controller may acquire a predicted disease image with the lesion of the designated type being modified on the basis of the medical image. In this case, for example, in a case in which a lesion is newly added to the medical image, the user can appropriately select the type of lesion to be added. Furthermore, the user can change the type of lesion in the medical image. Therefore, the predicted disease image is more appropriately acquired.

Furthermore, in the lesion reception step, the controller may receive an instruction to designate the degree of disease (progress status) of the lesion to be modified. When the degree of disease is designated, the controller may acquire the predicted disease image obtained by modifying the color of the lesion to a color (including color density, color distribution roughness, and the like) corresponding to the designated degree of disease on the basis of the medical image. In this case, the lesion whose color is changed according to the degree of disease is also appropriately displayed on the predicted disease image.

The controller may further execute the lesion type acquisition step. In the lesion type acquisition step, when the lesion already exists in the medical image acquired in the image acquisition step, the controller acquires the detection result of the type of already existing lesion. The controller may execute the processing of receiving an instruction in the lesion reception step and the processing of acquiring a predicted disease image in the predicted disease image acquisition step with the type of lesion acquired as the type of already existing lesion when the type of lesion is acquired in the lesion type acquisition step.

In this case, the user can input various instructions (for example, at least an instruction to enlarge or reduce the range of the already existing lesion, an instruction to add the same type of lesion as the already existing lesion to another position, or the like) according to the type of already existing lesion. Therefore, an appropriate predicted disease image corresponding to the type of the already existing lesion is acquired.

Note that the controller may cause the display to display the type of lesion acquired in the lesion type acquisition step. In this case, the user can input various instructions after appropriately grasping the type of already existing lesion.

A specific method for acquiring the type of already existing lesion in the medical image can be appropriately selected. For example, the controller may acquire the type of lesion output by the mathematical model by inputting the medical image acquired in the image acquisition step to the mathematical model trained by the machine learning algorithm. In this case, the mathematical model may be trained in advance by a training data set in which the data of the medical image is used as the input training data and information indicating the type of lesion included in the input training data is used as the output training data. By using the machine learning algorithm, the type of lesion in the medical image are more appropriately acquired.

In a case in which the range of the lesion is designated, the controller may acquire a predicted disease image in which the range of the lesion is between the designated range and the range of the lesion in the medical image acquired in the image acquisition step together with the predicted disease image in which the lesion in the designated range is modified. In this case, a transition in which the range of the lesion is enlarged or reduced is more appropriately grasped from a plurality of predicted disease images acquired based on the medical image of the subject himself/herself.

It is possible to appropriately select a method of displaying a plurality of predicted disease images having different lesion ranges on the display. For example, the controller may sequentially switch (for example, in a moving image) and display each of the plurality of predicted disease images in chronological order. In this case, the medical image (that is, a raw image in which a lesion is not modified) acquired in the image acquisition step may be displayed before the plurality of predicted disease images. In addition, the controller may display a plurality of predicted disease images side by side at the same time. In this case, a raw image may also be displayed in addition to the plurality of predicted disease images.

In the predicted disease image acquisition step, the controller may acquire a predicted disease image including an image of a lesion predicted from information of the lesion on the basis of information of the lesion including at least the position or range and an image of a tissue of a subject. The information of the lesion is set according to the instruction received in the lesion reception step. In this case, the image of the lesion predicted from the information designated by the user is appropriately acquired on the basis of the image of the tissue of the subject. Therefore, it is easy for the subject to realize the predicted disease state.

In the predicted disease image acquisition step, the controller may acquire a predicted disease image by inputting an image of the tissue of the subject and the information of the lesion to the mathematical model trained by the machine learning algorithm. In this case, using a mathematical model trained by a plurality of training data including an actual medical image makes it easy to acquire a predicted disease image approximate to the actual medical image as compared with a case of using a function or the like that does not use a machine learning algorithm.

When a machine learning algorithm is utilized, the mathematical model may have been trained by a plurality of training data sets. The training data set may include training data for input and training data for output (e.g., correct data). The training data for output may be a medical image of a tissue including a lesion photographed by the medical image photographing device. The training data for input may include a lesion removed image obtained by removing information of a lesion from the training data for output and information of a lesion in the training data for output (for example, information indicating at least a position, a range, or a type of lesion). In this case, the mathematical model for outputting a predicted disease image is appropriately trained based on the actually photographed medical image.

A medical image may be a fundus image (for example, a fundus front image obtained by photographing the fundus from the line-of-sight direction, a fundus tomographic image including information in the depth direction of the fundus, or the like) obtained by photographing the fundus of the eye to be examined. For example, in a case in which a medical image is a fundus front image, training data for output may be an image including a lesion photographed by a fundus photographing device (for example, a fundus camera, a scanning laser ophthalmoscope (SLO), or an OCT device) that photographs a fundus front image. In addition, a lesion removed image used as training data for input may be an image of a fundus vascular image. An image of the fundus blood vessel may be acquired by extracting the fundus blood vessel from a fundus front image or may be photographed by a device different from a fundus photographing device that photographs a fundus front image. The information of a lesion used as training data for input may be generated, for example, by specifying the position and range of the lesion in an image by the operator who looks at training data for output (fundus front image). In addition, the information of a lesion may be acquired by inputting training data for output to a mathematical model trained in advance to input a fundus front image and output the information of the lesion. In addition, information such as the position and range of a lesion may be acquired by performing known image processing on the fundus front image.

However, the controller can also acquire a predicted disease image without using any machine learning algorithm. For example, the controller may generate a predicted disease image from a medical image by converting at least part of the medical image according to designated information (information of the designated lesion).

Furthermore, the controller of the medical image processing device may output a medical image and designated information to another device (for example, a server) storing a program for implementing a mathematical model. A device at the output destination of a medical image may acquire a predicted disease image by inputting an input medical image and specified information to a mathematical model and output the acquired predicted disease image to the medical image processing device.

DESCRIPTION OF EMBODIMENTS

One embodiment of the present disclosure will be described below with reference to the accompanying drawings. As illustrated inFIG.1, the present embodiment uses a mathematical model construction device1, a medical image processing device21, and medical image photographing devices11A and11B. The mathematical model construction device1constructs a mathematical model by training the mathematical model by a machine learning algorithm. The constructed mathematical model outputs information according to the training contents based on the input information (details of the mathematical model will be described later with reference toFIG.2). The medical image processing device21executes various types of processing for acquiring a predicted disease image60(seeFIGS.2and3) using the mathematical model. The predicted disease image60is an image indicating a tissue state in a case in which the disease is predicted to occur, a tissue state in a case in which the disease is predicted to progress, a tissue state in a case in which the disease is predicted to cure, or the like. The predicted disease image60is acquired based on the actually photographed image of the tissue of the subject. The medical image photographing devices11A and11B photographs a medical image30(seeFIGS.2and3) which is an image of a tissue of a subject.

Note that the present embodiment will exemplify a case in which two-dimensional fundus front images obtained by photographing the fundus tissue of the eye to be examined from the line-of-sight direction of the eye to be examined are used as the medical image30and the predicted disease image60. However, at least part of the technology exemplified by the present disclosure can be applied to even a case in which the medical image30and the predicted disease image60are images other than fundus front images. For example, the medical image30and the predicted disease image60may be two-dimensional tomographic images or three-dimensional tomographic images including information in the depth direction of the fundus of the subject eye. The medical image30and the predicted disease image60may be ophthalmic images of tissues other than the fundus of the eye to be examined (anterior segment images). Furthermore, the medical image30and the predicted disease image60may be medical images of a living tissue other than the eye to be examined (for example, images of internal organs).

For example, a personal computer (hereinafter referred to as a “PC”) is used as the mathematical model construction device1according to the present embodiment. Although details will be described later, the mathematical model construction device1constructs a mathematical model by training the mathematical model using the medical image acquired from the medical image photographing device11A. However, the device that can function as the mathematical model construction device1is not limited to the PC. For example, the medical image photographing device11A may function as the mathematical model construction device1. In addition, controllers (for example, the CPU of the PC and the CPU13A of the medical image photographing device11A) of a plurality of devices may cooperate to construct a mathematical model.

In addition, a PC is used as the medical image processing device21according to the present embodiment. However, the device that can function as the medical image processing device21is not limited to the PC. For example, the medical image photographing device11B, a server, or the like may function as the medical image processing device21. In addition, a portable terminal such as a tablet terminal or a smartphone may function as the medical image processing device21. The controllers (for example, the CPU of the PC and the CPU13B of the medical image photographing device11B) of the plurality of devices may perform various types of processing in cooperation.

In addition, the present embodiment will exemplify a case in which a CPU is used as an example of a controller that performs various processes. Obviously, however, a controller other than the CPU may be used for at least some of the various devices. For example, by adopting a GPU as the controller, the processing speed may be increased.

The mathematical model construction device1will be described. The mathematical model construction device1is provided for, for example, a manufacturer that provides the medical image processing device21and a medical image processing program to a user. The mathematical model construction device1includes a control unit2that performs various control processes and a communication I/F5. The control unit2includes a CPU3that is a controller configured to perform control and a storage device4that can store programs, data, and the like. The storage device4stores a mathematical model construction program for constructing a mathematical model. In addition, the communication I/F5connects the mathematical model construction device1to another device (for example, the medical image photographing device11A, the medical image processing device21, or the like).

The mathematical model construction device1is connected to an operation part7and a display device8. The operation part7is operated by the user in order for the user to input various instructions to the mathematical model construction device1. As the operation part7, for example, at least one of a keyboard, a mouse, and a touch panel can be used. Note that a microphone or the like for inputting various instructions may be used together with the operation part7or instead of the operation part7. The display device8displays various images. Various devices (for example, at least one of a monitor, a display, and a projector) that can display an image can be used as the display device8. Note that the “image” in the present disclosure includes both a still image and a moving image.

The mathematical model construction device1can acquire information (to be sometimes simply referred to as a “medical image”) of the medical image30from the medical image photographing device11A. The mathematical model construction device1may acquire the information of the medical image30from the medical image photographing device11A via, for example, at least one of wired communication, wireless communication, and a removable storage medium (for example, a USB memory).

The medical image processing device21will be described. The medical image processing device21is installed, for example, in a facility (for example, a hospital or a health examination facility) that performs diagnosis, examination, or the like of a subject. The medical image processing device21includes a control unit22that performs various control processes and a communication I/F25. The control unit22includes a CPU23that is a controller configured to perform control and a storage device24that can store programs, data, and the like. The storage device24stores a medical image processing program for executing medical image processing (seeFIG.4) to be described later. The medical image processing program includes a program for implementing the mathematical model constructed by the mathematical model construction device1. The communication I/F25connects the medical image processing device21to another device (for example, the medical image photographing device11B or the mathematical model construction device1).

The medical image processing device21is connected to an operation part27and a display device28. Various devices can be used as the operation part27and the display device28similarly to the operation part7and the display device8described above.

The medical image processing device21can acquire the medical image30from the medical image photographing device11B. The medical image processing device21may acquire the medical image30from the medical image photographing device11B via, for example, at least one of wired communication, wireless communication, and a removable storage medium (for example, a USB memory). Furthermore, the medical image processing device21may acquire a program or the like for implementing the mathematical model constructed by the mathematical model construction device1via communication or the like.

The medical image photographing devices11A and11B will be described. The following is an example in which the present embodiment uses the medical image photographing device11A that provides the medical image30to the mathematical model construction device1and the medical image photographing device11B that provides the medical image30to the medical image processing device21. However, the number of medical image photographing devices used is not limited to two. For example, the mathematical model construction device1and the medical image processing device21may acquire the medical images30from a plurality of medical image photographing devices. In addition, the mathematical model construction device1and the medical image processing device21may acquire the medical image30from one common medical image photographing device. Note that the two medical image photographing devices11A and11B exemplified in the present embodiment have the same configuration. Accordingly, the two medical image photographing devices11A and11B will be collectively described below.

A medical image photographing device11(11A,11B) includes a control unit12(12A,12B) that performs various control processes and a medical image photographing part16(16A,16B). The control unit12includes a CPU13(13A,13B), which is a controller that performs control, and a storage device14(14A,14B) that can store programs, data, and the like.

The medical image photographing part16has various configurations necessary for photographing the medical image (fundus front image in the present embodiment)30of the tissue of the subject. For example, in a case in which the medical image photographing device11is a fundus camera, the medical image photographing part16includes an illumination optical system, a light receiving optical system, a photographing element, and the like for photographing a front image of the fundus of the subject. Note that the device that can be used as the medical image photographing device11is not limited to the fundus camera. For example, a scanning laser ophthalmoscope (SLO), an OCT device, a corneal endothelial cell photographing device (CEM), a computed tomography (CT) device, or the like may be used as the medical image photographing device11.

(Model Structure of Mathematical Model)

A model structure of a mathematical model according to the present embodiment for the medical image processing device21to acquire the predicted disease image60will be described with reference toFIG.2. In the present embodiment, a lesion removal model71, a lesion identifying model72, and a predicted disease image generation model74are used as mathematical models for acquiring the predicted disease image60.

By inputting the medical image30, the lesion removal model71outputs a lesion removed image40obtained by removing the information of a lesion31from the medical image30. As the lesion removed image40, various images that do not include the information of the lesion31and include the information of the structure of the tissue can be adopted. In the present embodiment, an image of a blood vessel (fundus vascular image) which is one of the structures of the tissue is used as the lesion removed image40.

Upon receiving the medical image30, the lesion identifying model72outputs a detection result (lesion information73) of the lesion31if the lesion31exists in the medical image30. In the present embodiment, the lesion information73output by the lesion identifying model72includes information of the position of the lesion31, information of the range of the lesion31, and information of the type (that is, the type of disease) of the lesion31. Although details will be described later, when the predicted disease image60is acquired, at least addition or change (including deletion) of the lesion information73is executed according to an instruction input from the user.

As an example, the lesion identifying model72according to the present embodiment outputs an attention map80(seeFIG.3) when obtaining a detection result of the type of the lesion31included in the medical image30. Information of the position and range of the lesion31is obtained from the attention map80. The attention map80indicates the distribution of influence degrees (attention degrees) at the respective positions of influences imposed when the lesion identifying model72obtains the detection result of the type of the lesion31in the image region. A region having a high influence degree strongly affects the detection result of the type of the lesion31as compared with a region having a low influence degree. An example of the attention map80is described in, for example, the following paper or the like. “Ramprasaath R. Selvaraju, et al. “Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization” Proceedings of the IEEE International Conference on Computer Vision, 2017-October, pp. 618-626”

However, it is also possible to change the method of acquiring information of the position and range of the lesion31. For example, information of the position and range of the lesion31may be acquired by executing known image processing on the medical image30and segmenting the region of the lesion31.

The predicted disease image generation model74outputs the predicted disease image60by inputting the lesion removed image40and the lesion information73for which at least addition, change, or deletion has been executed. As described above, the lesion removed image40does not include information of the lesion31. Accordingly, using the lesion removed image40and the lesion information73according to the instruction input from the user makes the predicted disease image generation model74properly output the predicted disease image60desired by the user.

In the present embodiment, the predicted disease image60is acquired on the basis of at least one (in the present embodiment, the structure of the tissue indicated by the lesion removed image40acquired from the medical image30) of the pixel value, the structure of the tissue, the brightness, the contrast, and the like of the actually photographed medical image30. Accordingly, it is easy for the subject to feel the predicted disease state as compared with the case in which the predicted disease image is acquired without being based on the medical image30of the subject himself/herself.

Furthermore, in the present embodiment, the predicted disease image60including the image of the lesion31predicted from the lesion information73is acquired on the basis of the actually photographed medical image30and the lesion information73including the information of the position and the range. Therefore, the image of the lesion31predicted from the information designated by the user is appropriately acquired on the basis of the image of the tissue of the subject.

An example of the mathematical model construction processing executed by the mathematical model construction device1will be described. The CPU3of the mathematical model construction device1constructs a mathematical model by executing training of a mathematical model using a training data set by a machine learning algorithm. A program for implementing the constructed mathematical model is stored in the storage device24of the medical image processing device21. As the machine learning algorithm, for example, a neural network, a random forest, boosting, a support vector machine (SVM), and the like are generally known.

A neural network is a technique that mimics the behavior of a biological neuronal network. Neural networks include, for example, a feedforward (forward propagation type) neural network, a RBF network (radiation basis function), a spiking neural network, a convolutional neural network, a recursive neural network (recurrent neural net, feedback neural net, or the like), and a probabilistic neural network (Boltzmann machine, Bayesian network, or the like).

Random forest is a method of performing learning based on randomly sampled training data to generate a large number of decision trees. In the case of using random forest, branches of a plurality of decision trees learned in advance as identifiers are followed, and an average (or majority decision) of results obtained from the respective decision trees is taken.

Boosting is a method of generating a strong identifier by combining a plurality of weak identifiers. A strong identifier is constructed by sequentially learning a simple weak identifier.

The SVM is a method of configuring a two-class pattern identifier using a linear input element. The SVM learns the parameters of the linear input element based on, for example, a criterion (hyperplane separation theorem) of obtaining a margin maximizing hyperplane having the maximum distance from the training data to each data point.

In the present embodiment, each mathematical model is constructed by a generative adversarial network (GAN) that uses two competing neural networks, a convolutional neural network (CNN) that is a type of multilayer neural network, and the like.

A mathematical model refers, for example, to a data structure for predicting a relationship between input data and output data. The mathematical model is constructed by being trained using a plurality of training data sets. The training data set is a set of training data for input and training data for output. The mathematical model is trained such that when certain training data for input is input, corresponding training data for output is output. For example, training updates the correlation data (for example, a weight) between each input and each output.

An example of a method of constructing the lesion removal model71will be described. In the present embodiment, when constructing the lesion removal model71, the CPU3of the mathematical model construction device1trains the mathematical model using the medical image30of the subject photographed by the medical image photographing device11A as training data for input and using the lesion removed image (fundus vascular image in the present embodiment)40of the same subject as training data for output. For example, the lesion removed image40may be photographed by the medical image photographing device11A by a method different from the method of photographing the medical image30, or may be photographed by a device different from the medical image photographing device11A. In addition, the lesion removed image40may be extracted from the medical image30by known image processing. Furthermore, the lesion removed image40may be extracted from the medical image30according to an instruction input from the user.

An example of a method of constructing the lesion identifying model72will be described. In the present embodiment, when constructing the lesion identifying model72, the CPU3trains the mathematical model using the medical image30of the subject photographed by the medical image photographing device11A as training data for input and using information (in the present embodiment, information of the position of the lesion31, information of the range of the lesion31, and information of the type of the lesion31) of the lesion31in the training data for input (medical image30) as training data for output. For example, the information of the lesion31may be generated according to an instruction input from the user who has confirmed the training data for input (medical image30). Furthermore, at least part of the information of the lesion31(for example, position and range information) may be generated by executing known image processing on the training data for input.

An example of a method of constructing the predicted disease image generation model74will be described. In the present embodiment, when constructing the predicted disease image generation model74, the CPU3uses the medical image30of the subject photographed by the medical image photographing device11A as training data for output. In addition, the CPU3uses, as training data for input, the lesion removed image40obtained by removing the information of the lesion31from the training data for output (the medical image30) and the information of the lesion31related to the training data for output (in the present embodiment, information of the position of the lesion31, information of the range of the lesion31, and information of the type of the lesion31). As a method by which the CPU3acquires the lesion removed image40and the information of the lesion31, a method similar to the above-described method can be adopted.

(Predicted Disease Image Acquisition Screen)

An example of a predicted disease image acquisition screen9displayed on the display device28will be described with reference toFIG.3. The predicted disease image acquisition screen9is displayed on the display device28when the user causes the medical image processing device21to generate the predicted disease image60of the subject.

On the predicted disease image acquisition screen9according to the present embodiment, the medical image30, the lesion removed image40, the attention map80, a pre-modification image50, the predicted disease image60, an addition/deletion selector91, and a lesion type display92are displayed.

The medical image30is an image of the tissue of the subject actually photographed by the medical image photographing device11B (seeFIG.1). Including the medical image30in the predicted disease image acquisition screen9allows the user to easily grasp the state of the tissue appearing in the actually photographed medical image30and the state of the tissue appearing in the predicted disease image60.

As described above, the lesion removed image40is an image (fundus vascular image in the present embodiment) obtained by removing the information of the lesion31from the medical image30. Including the lesion removed image40in the predicted disease image acquisition screen9allows the user to grasp the structure of the tissue of the subject and then input an appropriate instruction to generate the predicted disease image60to the medical image processing device21.

As described above, the attention map80indicates the distribution of influence degrees (attention degrees) at the respective positions of influences imposed when the lesion identifying model72obtains the detection result of the type of the lesion31. Therefore, the attention map80shows the position and range of the lesion31appearing in the medical image30. Accordingly, including the attention map80in the predicted disease image acquisition screen9allows the user to input an appropriate instruction to the medical image processing device21after grasping the position and range of the lesion31actually existing in the tissue of the subject.

At least the position or range of the lesion31(seeFIG.2) to be modified on the medical image30is displayed on the pre-modification image50. The pre-modification image50according to the present embodiment is an image illustrating at least the position or range (in the present embodiment, both the position and range) of the lesion31on a plain background. Note that, in a state before an instruction to add or change (including deletion) the position or range of the lesion31is input, the position and range of the lesion31appearing in the attention map80are indicated in the pre-modification image50. When an instruction to add or change the position or range is input, the position/range of the lesion31in the pre-modification image50is changed according to the input instruction. The pre-modification image50exemplified inFIG.3is an image after an instruction to change the position/range of the lesion31is input.

The predicted disease image60is an image indicating a predicted tissue state. When an instruction to designate the lesion31is input by the user, the predicted disease image60generated according to the designated information is displayed on the display device28in real time.

The addition/deletion selector91is displayed to make the user input an instruction to switch between a case of newly adding the lesion31and a case of deleting the lesion31. In the present embodiment, when the user inputs an instruction to select “Insert”, the lesion31can be newly added. When the user inputs an instruction to select “Delete”, the lesion31can be deleted.

The lesion type display92displays the type of the lesion31already existing in the photographed medical image30. Furthermore, switching the type of the lesion31displayed on the lesion type display92allows the user to input, to the medical image processing device21, an instruction to designate the type of the lesion31to be newly added and an instruction to change the type of the lesion31set at that time.

Note that the predicted disease image acquisition screen9illustrated inFIG.3is merely an example. That is, the configuration of the predicted disease image acquisition screen9can be appropriately changed. First of all, the configuration of the pre-modification image50can be changed. In a pre-modification image, only the position of the lesion31(for example, the position of the center of the lesion31) may be displayed. In this case, the range of the lesion31may be displayed with a parameter (for example, a numerical value indicating the breadth of the range) other than the image. In addition, the position and range of the lesion31may be displayed in different pre-modification images. When, for example, the position of the lesion31is determined in advance, only the range (shape and size) of the lesion31may be displayed in the pre-modification image.

In addition, various images can be used as pre-modification images. For example, the lesion removed image40may be used as a pre-modification image, or the medical image30actually photographed by the medical image photographing device11B (seeFIG.1) may be used as a pre-modification image. In addition, an image photographed by a device different from the medical image photographing device11B that has photographed the medical image30may be used as a pre-modification image. In addition, the pre-modification image and the predicted disease image60may be the same. That is, by specifying the position and range of the lesion31on the medical image30or the predicted disease image60, the predicted disease image60according to the specified information may be generated in real time, and the displayed medical image30or the predicted disease image60may be switched to the newly generated predicted disease image60. In addition, the pre-modification image and the predicted disease image60may be switched and displayed. The lesion removed image40may not be displayed on the display device28during the execution of medical image processing (seeFIG.4).

In the present embodiment, the pre-modification image50and the predicted disease image60are displayed on the same display device28. However, the pre-modification image50and the predicted disease image60may be displayed on different display devices simultaneously or in a switched manner. For example, the pre-modification image50may be displayed on a display device visually recognized by a doctor or the like, and the predicted disease image60may be displayed on a display device visually recognized by a subject or the like.

Medical image processing executed by the medical image processing device21according to the present embodiment will be described with reference toFIGS.3and4. The medical image processing is executed by the CPU23according to the medical image processing program stored in the storage device24. When the user inputs an instruction to start generation and display of the predicted disease image60, the CPU23starts the medical image processing illustrated inFIG.4.

As illustrated inFIG.4, upon starting the medical image processing, the CPU23acquires the medical image30(seeFIG.2) of the subject (S1). As described above, the medical image30according to the present embodiment is an image of the tissue of the subject actually photographed by the medical image photographing device11B. The CPU23causes the display device28to display the medical image30acquired in S1(more specifically, in the predicted disease image acquisition screen9in the present embodiment)

The CPU23acquires the lesion removed image40of the same tissue as the photographing target of the medical image30and causes the display device28to display the image (S2). As described above, the CPU23according to the present embodiment acquires the lesion removed image40by inputting the medical image30acquired in S1to the lesion removal model71(seeFIG.2). Therefore, the lesion removed image40is more appropriately acquired as compared with a case of using a method not using a machine learning algorithm (for example, known image processing). However, it is also possible to change the method of acquiring the lesion removed image40. For example, the lesion removed image40may be photographed by the medical image photographing device11A by a method different from the method of photographing the medical image30, or may be photographed by a device different from the medical image photographing device11A. In addition, the lesion removed image40may be extracted from the medical image30by known image processing. Furthermore, the lesion removed image40may be extracted from the medical image30according to an instruction input from the user.

The CPU23then displays the pre-modification image50(seeFIG.3) on the display device28(S3). As described above, the pre-modification image50may be the lesion removed image40, the medical image30, or the like

The CPU23then inputs the medical image30acquired in S1to the lesion identifying model72(S4). When the lesion31already exists in the input medical image30, the lesion identifying model72according to the present embodiment outputs the lesion information73(detection result of at least the range or type of the lesion31). On the other hand, when the lesion30does not exist in the medical image30, the lesion information73is not output. If the lesion31does not exist in the medical image30acquired in S1(S6: NO), the process directly proceeds to S11.

When the lesion31is present in the medical image30acquired in S1(S6: YES), the CPU23acquires the lesion information73output by the lesion identifying model72and stores the same in the storage device24(S7). As a result, the processing in S11to S21described later is executed with the position, range, and type of the lesion31acquired in S7as the position, range, and type of the already existing lesion31. Therefore, the user can appropriately input various instructions to the medical image processing device in accordance with the position, range, and type of the already existing lesion31. In addition, the CPU23displays the position and range of the already existing lesion31in the medical image30on the pre-modification image50based on the lesion information73acquired in S7(S8). Furthermore, in the present embodiment, the CPU23causes the display device28to display the attention map80(seeFIG.3) obtained by the lesion identifying model72. The CPU23causes the lesion type display92(seeFIG.3) to display the type of the lesion31already existing in the medical image30on the basis of the lesion information73acquired in S7(S9). Therefore, the user can appropriately grasp the position, range, and type of the already existing lesion31and then input various instructions to the medical image processing device21.

The CPU23then determines whether or not an instruction to designate the type (for example, the type of the lesion31to be newly added to the predicted disease image60) of the lesion31to be modified has been input (S11). Upon receiving the instruction to designate the type of the lesion31(S11: YES), the CPU23stores the designated type of the lesion31in the storage device23(S12). The process proceeds to S13.

The CPU23then determines whether an instruction to newly add the lesion31to the predicted disease image60has been input (S13). For example, in the present embodiment, in a state in which the type of the lesion31is designated, the user can input an instruction to add the lesion31to a desired position by moving the cursor to the desired position on the pre-modification image50and operating the operation part27. Upon receiving an instruction to add the lesion31(S13: YES), the CPU23adds information of the position and type of the designated lesion31to the lesion information73(seeFIG.2) (S14). In addition, the CPU23displays the position of the lesion31added in S14on the pre-modification image50(seeFIG.3). Thereafter, the process proceeds to S20(described later).

If an instruction to add the lesion31is not input (S13: NO), the CPU23determines whether an instruction to move the position of the lesion31is input (S15). For example, in the present embodiment, the user can input an instruction to designate the movement position of the lesion31by dragging the position of the lesion31on the pre-modification image50to a desired position. Upon receiving the movement instruction (S15: YES), the CPU23changes the information of the position of the selected lesion31in the lesion information73(seeFIG.2) to the position designated in S15(S16). In addition, the CPU23moves the position of the lesion31displayed on the pre-modification image50(seeFIG.3) on the basis of the information of the position of the lesion31changed in S16. Thereafter, the process proceeds to S20(described later).

When an instruction to move the lesion31has not been input (S15: NO), the CPU23determines whether an instruction to change (enlarge, reduce, or delete) the range of the lesion31has been input (S17). For example, in the present embodiment, the user can input an instruction to designate enlargement or reduction of the range of the lesion31by dragging the frame portion of the lesion31on the pre-modification image50in a desired direction. In addition, the user can input an instruction to designate deletion of the selected lesion31by operating the deletion button in a state in which the lesion31on the pre-modification image50is selected. If an instruction to change the range of the lesion31is input (S17: YES), the CPU23changes (enlarge, reduce, or delete) the information of the range of the selected lesion31in the lesion information73(seeFIG.2) to the range designated in S17(S18). In addition, the CPU23changes the range of the lesion31displayed on the pre-modification image50(seeFIG.3) on the basis of the information of the range of the lesion31changed in S18. Thereafter, the process proceeds to S20.

When the instruction to designate the lesion31is received in S13, S15, or S17or the lesion information73is added or changed in S14, S16, or S18, the CPU23acquires the predicted disease image60in which the lesion31is modified according to the designated information based on the medical image30acquired in S1(S20, S21). More specifically, in S20in the present embodiment, the CPU23inputs the lesion removed image40acquired in S2on the basis of the medical image30and the added or changed lesion information73to the predicted disease image generation model74(seeFIG.2). In S21, the CPU23acquires the predicted disease image60output by the predicted disease image generation model74, stores the predicted disease image in the storage device24, and displays the predicted disease image on the display device28(the predicted disease image acquisition screen9according to the present embodiment). That is, the CPU23causes the display device28to display the newly acquired predicted disease image60in real time while causing the display device28to display the pre-modification image50.

Note that, in S20and S21in the present embodiment, the CPU23can acquire the predicted disease image60in which the lesion31at the designated position/range is modified and the predicted disease image60in the middle of transition of the state of the lesion31(hereinafter referred to as an “image during transition”). The image during transition is an image in which the position/range of the lesion31is between the position/range designated in S13, S15, or S17and the position/range of the lesion31in the medical image30acquired in S1. In this case, the user can appropriately grasp the process in which the state of the lesion31changes. When acquiring the image during transition in S20and S21, the CPU23may sequentially switch the plurality of predicted disease images60along the time series (for example, in a moving image) and display the plurality of predicted disease images on the display device28. In this case, the medical image30acquired in S1may be displayed before the plurality of predicted disease images60. In addition, the CPU23may cause the display device28to simultaneously display a plurality of predicted disease images60side by side.

When an instruction to end the medical image processing is not input (S23: NO), the process returns to S11, and the processing in S11to S23is repeated. When an end instruction is input (S23: YES), the medical image processing ends.

The technique disclosed in the above embodiment is merely an example. Accordingly, it is also possible to change the technique exemplified in the above embodiment. First of all, it is also possible to execute only some of the plurality of techniques exemplified in the above embodiments. For example, at least one of the processes in S11, S13, S15, and S17may be omitted.

In S20and S21in the above embodiment, the predicted disease image60is acquired on the basis of the lesion removed image40generated from the medical image30acquired in S1and the lesion information73added or changed according to the designated information from the user. That is, in the above embodiment, the predicted disease image60is acquired not on the basis of the medical image30itself but on the basis of the lesion removed image40acquired from the medical image30. Therefore, in S1, the lesion removed image40(fundus vascular image in the present embodiment) may be acquired as a medical image of the subject used to acquire the predicted disease image60. In this case, the processing in S2may be omitted.

In the above embodiment, the predicted disease image generation model74(seeFIG.2) outputs the predicted disease image74by inputting the lesion removed image40acquired from the medical image30and the lesion information73. However, it is also possible to change a specific method for acquiring the predicted disease image60on the basis of the medical image30and the lesion information73. For example, the CPU23may extract an image of a region of the lesion31from the medical image30. The CPU23may convert at least the position, range, type, color, or the like of the image of the region of the lesion31according to the lesion information73designated by the user. The CPU23may generate the predicted disease image60by combining the converted image with the medical image30. The above processing may be executed by using a machine learning algorithm or may be executed by known image processing. In this case, the process of acquiring the lesion removed image40may be omitted.

Furthermore, in a case in which the medical image30is a fundus front image, the pre-modification image may be a map image that two-dimensionally shows a distribution (for example, the distribution of the thicknesses of at least one of the layers in the eye fundus) related to the structure of the fundus. In this case, the user may designate at least the position or the region of the lesion31by designating a region (for example, a region where the thickness of the layer is within a specific range) of a desired structure illustrated in the map image.

In addition, the CPU23may receive an instruction to designate the degree of disease (progress status or cure status) of the lesion31to be modified. When the degree of disease is designated, the CPU23may acquire the predicted disease image60obtained by modifying the color of the lesion31to a color (including color density, color distribution roughness, and the like) corresponding to the designated degree of disease on the basis of the medical image30. In this case, the predicted disease image generation model74may be trained using the lesion information73including information of the degree of disease as training data for input. In this case, the lesion31whose color is converted according to the degree of disease is also appropriately displayed on the predicted disease image60.

Note that the process of acquiring the medical image30in S1inFIG.4is an example of the “image acquisition step”. The process of displaying the pre-modification image50in S3inFIG.4is an example of the “pre-modification image display step”. The process of receiving the designation instruction related to the lesion31in S11, S13, S15, and S17inFIG.4is an example of the “lesion reception step”. The process of acquiring the predicted disease image60in S20and S21inFIG.4is an example of the “predicted disease image acquisition step”. The process of displaying the predicted disease image in S21inFIG.4is an example of the “predicted disease image display step”. The process of acquiring the detection result regarding the lesion31in the medical image30in S4to S7inFIG.4is an example of the “lesion position acquisition step”, the “lesion range acquisition step”, and the “lesion type acquisition step”.

LIST OF REFERENCE SIGNS