Patent Description:
Diabetic retinopathy is an important disease that is the second leading cause of blindness in Japan, but subjective symptoms do not appear until the stage of the disease has progressed considerably, and thus, early detection and early treatment at medical examinations, or the like, is important. To solve this problem, a fundus image analysis system that emphasizes a capillary blood vessel aneurysm that shows an initial change in diabetic retinopathy (see Patent Document <NUM>) and an image analysis system that screens diabetic retinopathy from fundus images (see Patent Document <NUM>) have been proposed.

<CIT> (hereinafter Patent Document <NUM>) and <CIT> (hereinafter Patent Document <NUM>) form part of the state of the art relative to the present disclosure. Also forming part of the state of the art are "Non-rigid registration of Fluorescein Angiography and Optical Coherence Tomography via scanning laser ophthalmoscope imaging" DOI: <NUM>/EMBC. <NUM> and "ReLayNet: Retinal Layer and Fluid Segmentation of Macular Optical Coherence Tomography using Fully Convolutional Network", DOI: <NUM>.

The techniques proposed in Patent Document <NUM> and Patent Document <NUM> are aimed at discovery and screening of diseases as described above. On the other hand, advancing one step from the disease screening of the diabetic retinopathy, in the case of the stage of medical treatment or cure of diabetic retinopathy patients, it has become clear that the diabetic retinopathy is caused by an ischemic state due to vascular disorder associated with hyperglycemia so that the diabetic retinopathy is progressing, and thus, it is most important to understand retinal circulatory dynamics. In order to understand the retinal circulatory dynamics, angiography of retina such as fundus fluorescein angiography is essential. However, due to its risk, invasiveness, and geographical restrictions limited to large hospitals, the angiography of retina is a burdensome examination for both patients and medical staffs, which causes hesitancy to take pictures of patients with good eyesight and patients with reduced physical functions, to perform repeated imaging, or the like. For this reason, there is a problem that in many cases, detection of symptom appearances and disease states is delayed, and the treatment is delayed. The same problem appears in all ocular ischemic diseases represented by diabetic retinopathy. OCT angiography is a non-invasive device for understanding retinal circulation abnormality, but unlike the contrast examination, the OCT angiography cannot detect the blood flow dynamics and cannot detect the whole image of the retina due to a narrow angle of view while being effective for local diseases such as a macular disease. Accordingly, it is not suitable for detecting the retinal circulatory dynamics of ocular ischemic diseases such as diabetic retinopathy.

The present invention has been made in view of such circumstances, and an object of the present invention is to quickly and easily recognize abnormal circulatory findings from a fundus image without performing fundus fluorescein angiography.

According to a second aspect of the present invention, there is provided a device for supporting diagnosis as recited in claim <NUM> below.

According to a third aspect of the present invention, there is provided a computer program as recited in claim <NUM> below.

The dependent claims define particular embodiments and implementations of the claimed invention.

The following detailed description and accompanying drawings neither define not limit the scope of the claimed invention, but rather are to illustrate implementations and embodiments of the claimed invention and to show how the same may be put into effect.

The scope of protection is therefore defined according to the appended claims.

In the present invention, the "fundus image" means an image obtained by imaging the fundus of a patient with a fundus camera in order to make a diagnosis regarding a disease of an ophthalmological system, and the "fluorescein angiography image" means an image obtained by continuously taking a photograph of the fundus using an excitation light and a filter corresponding to fluorescence dye while injecting the fluorescence dye through a vein of the patient's arm and taking a contrast image of a circulating state of the fundus.

According to the present invention, it is possible to predict abnormal circulatory findings from a fundus image easily, without performing fundus fluorescein angiography.

<FIG> is a configuration diagram illustrating an information processing system including a server <NUM> that is an embodiment of an information processing device of the present invention.

The information processing system shown in <FIG> is configured to include the server <NUM>, an ophthalmologist terminal <NUM>, and an examination device <NUM>.

The server <NUM>, the ophthalmologist terminal <NUM>, and the examination device <NUM> are connected to each other through a network N such as the Internet.

The server <NUM> is a server that manages the information processing system shown in <FIG>, and executes various processes such as an NPA/NV existence probability map generating process, an accompanying finding existence probability map generating process, and an estimated NPA/NV recognizing process. The content of specific processes executed by the server <NUM> will be described later with reference to <FIG>.

The "NPA/NV existence probability map generating process" is a series of processes executed by the server <NUM> from generation of NPA/NV training information to generation of an NPA/NV existence probability map.

The "NPA/NV training information" refers to training information for calculating a retinal non-perfusion area (NPA/non-perfusion area) (hereinafter, referred to as "a retinal non-perfusion area" or "NPA") existence probability (hereinafter, referred to as an "NPA existence probability") and a neovascularization (NV/neovascularization) existence probability (hereinafter, referred to as an "NV existence probability"), in fundus image information of a patient. Specifically, the "NPA/NV training information" is generated on the basis of fluorescein angiography image information and NPA/NV annotation information attached to the fluorescein angiography image information.

The "retinal non-perfusion area" refers to a poor circulatory area of the retina that results from retinal vessel occlusion in ocular ischemic diseases.

The "NPA/NV existence probability map" is image information in which the NPA existence probability and the NV existence probability are identifiably displayed in fundus image information.

The "fundus image information" refers to image information based on a fundus image.

The "fluorescein angiography image information" refers to information based on a fluorescein angiography image.

The "NPA/NV annotation information" refers to a diagnostic note of an ophthalmologist D, attached to fluorescein angiography image information, regarding at least one of a retinal non-perfusion area (NPA) or a neovascularization (NV).

The "neovascularization" means that poor retinal circulation and ischemia have further progressed in the retinal non-perfusion area. In a case where bleeding or retinal detachment through proliferative membrane formation occurs from the neovascularization, it eventually leads to blindness, and thus, it is very important to specify a poor circulatory area of the retina such as a retinal non-perfusion area or a neovascularization.

In addition, a specific processing flow of the NPA/NV existence probability map generating process will be described later with reference to a flowchart of <FIG>.

The "accompanying finding existence probability map generating process" refers to a series of processes executed by the server <NUM> from generation of the accompanying finding training information to generation of the accompanying finding existence probability map.

The "accompanying finding training information" refers to training information in calculating an accompanying finding existence probability in fundus image information of a patient. Specifically, the "accompanying finding training information" refers to training information generated on the basis of the fluorescein angiography image information and the fundus image information, and accompanying finding annotation information attached to these image information.

The "accompanying finding annotation information" means a diagnostic note other than the diagnostic note regarding a retinal non-perfusion area (NPA) or a neovascularization (NV), among diagnostic notes indicating that the ophthalmologist D incidentally makes a judgment "not normal" with respect to the fluorescein angiography image information and the fundus image information. For example, information such as a capillary aneurysm, fundus bleeding, vitiligo, soft vitiligo, venous abnormality, intra-retinal microvascular abnormality, vitreous hemorrhage, a proliferative membrane, and retinal detachment is an example of the "accompanying finding annotation information".

The "accompanying finding existence probability map" is image information (not shown) in which the existence probability of the accompanying finding is identifiably displayed in the fundus image information.

A specific processing flow of the accompanying finding existence probability map generating process will be described later with reference to the flowchart of <FIG>.

The "estimated NPA/NV recognizing process" refers to a series of processes from a process of recognizing an area estimated to correspond to the retinal non-perfusion area (NPA) in the fundus image information as an estimated NPA, on the basis of the NPA existence probability, and a process of recognizing an area estimated to correspond to the neovascularization (NV) in the fundus image information as an estimated NV, on the basis of the NV existence probability, in the processes executed by the server <NUM>.

Note that a specific processing flow of the estimated NPA/NV recognizing process will be described later with reference to the flowchart of <FIG>.

The ophthalmologist terminal <NUM> is an information processing device operated by the ophthalmologist D, and is configured by, for example, a personal computer or the like. The ophthalmologist terminal <NUM> transmits NPA/NV annotation information and accompanying finding annotation information to the server <NUM>, and acquires information on estimated NPA/NV identified by the server <NUM>, for example. The various types of information acquired by the ophthalmologist terminal <NUM> is output from the ophthalmologist terminal <NUM>, and is used for examination by the ophthalmologist D.

The examination device <NUM> is configured by various devices used in eye examination of a patient. The examination device <NUM> transmits each of fundus image information obtained by imaging in fundus examination and fluorescein angiography image information obtained by imaging in fundus fluorescein angiography to the server <NUM>.

<FIG> is a block diagram illustrating a hardware configuration of the server <NUM> in the information processing system of <FIG>.

The server <NUM> includes a central processing unit (CPU) <NUM>, a read only memory (ROM) <NUM>, a random access memory (RAM) <NUM>, a bus <NUM>, an input/output interface <NUM>, an output unit <NUM>, an input unit <NUM>, a storage unit <NUM>, a communication unit <NUM>, and a drive <NUM>.

The CPU <NUM> executes various processes according to a program recorded on the ROM <NUM> or a program loaded from the storage unit <NUM> into the RAM <NUM>.

The RAM <NUM> also appropriately stores data and the like necessary in a case where the CPU <NUM> executes various processes.

The CPU <NUM>, ROM <NUM>, and RAM <NUM> are connected to each other through the bus <NUM>. The input/output interface <NUM> is also connected to the bus <NUM>. The output unit <NUM>, the input unit <NUM>, the storage unit <NUM>, the communication unit <NUM>, and the drive <NUM> are connected to the input/output interface <NUM>.

The output unit <NUM> is configured by various liquid crystal displays or the like, and outputs various types of information.

The input unit <NUM> is configured by various types of hardware, through which various types of information are input.

The storage unit <NUM> is configured by a dynamic random access memory (DRAM) or the like, and stores various types of data.

The communication unit <NUM> controls communication with other devices through the network N including the Internet.

A drive <NUM> is provided as necessary. A removable medium <NUM> including a magnetic disc, an optical disc, a magneto-optical disc, a semiconductor memory, or the like is appropriately mounted on the drive <NUM>. A program read from the removable medium <NUM> by the drive <NUM> is installed in the storage unit <NUM> as necessary. In addition, the removable medium <NUM> may also store various types of data stored in the storage unit <NUM>, similarly to the storage unit <NUM>.

Next, a functional configuration of the server <NUM> having such a hardware configuration will be described with reference to <FIG>.

<FIG> is a functional block diagram illustrating an example of a functional configuration for realizing the NPA/NV existence probability map generating process, the accompanying finding existence probability map generating process, the estimated NPA/NV recognizing process, and the estimated NPA/NV display process in the functional configuration of the server <NUM> of <FIG>, in the information processing system of <FIG>.

As shown in <FIG>, in the CPU <NUM> (<FIG>) of the server <NUM>, in a case where the NPA/NV existence probability map generating process is executed, an image acquisition unit <NUM>, an annotation acquisition unit <NUM>, a training information generation unit <NUM>, and an arithmetic unit <NUM> perform their functions.

In a case where the accompanying finding existence probability map generating process is executed, the image acquisition unit <NUM>, the annotation acquisition unit <NUM>, the training information generation unit <NUM>, and the arithmetic unit <NUM> perform their functions.

In a case where the estimated NPA/NV recognizing process is executed, an estimated NPA/NV identification unit <NUM> performs its function.

In a case where the estimated NPA/NV display process is executed, an estimated NPA/NV display control unit <NUM> performs its function.

An image DB <NUM>, an annotation DB <NUM>, and a training DB <NUM> are provided in one area of the storage unit <NUM> (<FIG>). The storage unit <NUM> may be disposed in the ophthalmologist terminal <NUM> instead of the server <NUM>.

The image acquisition unit <NUM> acquires fluorescein angiography image information of a patient and fundus image information of the patient. The fluorescein angiography image information and the fundus image information acquired by the image acquisition unit <NUM> are respectively stored in the image DB <NUM> for management. Specifically, in fundus fluorescein angiography of a patient, in a case where a fluorescein angiography image of the patient is captured by the examination device <NUM>, fluorescein angiography image information based on the fluorescein angiography image is transmitted to the server <NUM>. Further, in the fundus examination of the patient, in a case where a fundus image of the patient is captured by the examination device <NUM>, fundus image information based on the fundus image is transmitted to the server <NUM>. The image acquisition unit <NUM> of the server <NUM> acquires the fluorescein angiography image information and the fundus image information transmitted from the examination device <NUM> to the server <NUM>, and stores the image information in the image DB <NUM>.

Thus, the server <NUM> can accurately manage the fluorescein angiography image information and the fundus image information of the patient without omission.

The annotation acquisition unit <NUM> acquires a diagnostic note of the ophthalmologist D, which is attached to the fluorescein angiography image information of the patient, regarding at least one of a retinal non-perfusion area (NPA) or a neovascularization (NV) as NPA/NV annotation information. Specifically, in the fundus fluorescein angiography, when the diagnostic note of the ophthalmologist D regarding the retinal non-perfusion area (NPA) and the neovascularization (NV) is attached to the fluorescein angiography image information, the ophthalmologist terminal <NUM> transmits the diagnostic note to the server <NUM> as the NPA/NV annotation information based on the operation of the ophthalmologist D. The annotation acquisition unit <NUM> of the server <NUM> acquires the NPA/NV annotation information transmitted from the ophthalmologist terminal <NUM>, and stores the information in the annotation DB <NUM>. The fluorescein angiography image information and the NPA/NV annotation information attached to the image information are managed in association with each other.

As a result, the server <NUM> can manage, without omission, the diagnostic note of the ophthalmologist D regarding at least one of the retinal non-perfusion area (NPA) or the neovascularization (NV) attached to the fluorescein angiography image information, as the NPA/NV annotation information.

In addition, the annotation acquisition unit <NUM> acquires the diagnostic note of the ophthalmologist D regarding an accompanying finding attached to the fluorescein angiography image and the fundus image, as accompanying finding annotation information. Specifically, in a case where the diagnostic note of the ophthalmologist D regarding the accompanying finding is attached to the fluorescein angiography image information and the fundus image information, the ophthalmologist terminal <NUM> transmits the diagnostic note to the server <NUM> as the accompanying finding annotation information on the basis of the operation of the ophthalmologist D. The annotation acquisition unit <NUM> of the server <NUM> acquires the accompanying finding annotation information transmitted from the ophthalmologist terminal <NUM>, and stores this information in the annotation DB <NUM>. The fluorescein angiography image information and the fundus image information, and the accompanying finding annotation information attached to these image information are managed in association with each other.

As a result, the server <NUM> can manage the diagnostic note of the ophthalmologist D regarding the accompanying finding attached to the fluorescein angiography image information and the fundus image information as the accompanying finding annotation information, without omission.

The training information generation unit <NUM> generates NPA/NV training information that serves as training information for calculating an NPA existence probability and an NV existence probability on the basis of the fluorescein angiography image information and the NPA/NV annotation information corresponding to the fluorescein angiography image information.

That is, the image DB <NUM> stores fluorescein angiography image information obtained from a plurality of patients, and the annotation DB <NUM> stores NPA/NV annotation information. The training information generation unit <NUM> generates NPA/NV training information which serves as training information in calculating the NPA existence probability and the NV existence probability in the fundus image information on the basis of the information stored in these databases.

Thus, the server <NUM> can generate and store the training information for calculating the NPA existence probability and the NV existence probability in the fundus image information of the patients.

In addition, the training information generation unit <NUM> generates the accompanying finding training information on the basis of the fluorescein angiography image information and the fundus image information, and the accompanying finding annotation information corresponding to these image information.

That is, the image DB <NUM> stores the fundus image information and the fluorescein angiography image information obtained from a plurality of patients, and the annotation DB <NUM> stores the accompanying finding annotation information. The training information generation unit <NUM> generates the accompanying finding training information that serves as training information in calculating the existence probability of the accompanying findings in the fundus image information, on the basis of the information stored in these databases.

Thus, the server <NUM> can store the training information for calculating the existence probability of the accompanying findings in the fundus image information of the patients.

The arithmetic unit <NUM> calculates the NPA existence probability and the NV existence probability on the basis of at least the NPA/NV training information. In addition, in a case where an accompanying finding existence probability map (which will be described later) is generated, the arithmetic unit <NUM> calculates the NPA existence probability and the NV existence probability on the basis of the accompanying finding existence probability map and the NPA/NV training information. A specific method for calculating the NPA existence probability and the NV existence probability is not particularly limited. For example, a method for calculating the NPA existence probability and the NV existence probability by extracting a feature common in the fundus images having the NPA/NV from the NPA/NV training information and normalizing the matching degree withthe feature with respect to whether or not the fundus image information of the patient has the feature may be used. The NPA existence probability and the NV existence probability may be calculated using a deep learning technique.

As a result, it is possible to set a standard for recognizing an area of the fundus image information that is estimated to correspond to the retinal non-perfusion area (NPA) and an area that is estimated to correspond to the neovascularization (NV).

Further, the arithmetic unit <NUM> calculates the existence probability of the accompanying finding in the fundus image information on the basis of the accompanying finding training information. A specific method of calculating the existence probability of the accompanying findings is not particularly limited. For example, a method for calculating the accompanying finding existence probability by extracting a feature common in the fundus images having the accompanying finding from the accompanying finding training information and normalizing the matching degree with the feature with respect to whether or not the fundus image information of the patient has the feature may be used. The accompanying finding existence probability may be calculated using a deep learning technique.

Accordingly, it is possible to set a standard for recognizing an area of the fundus image information that is estimated to correspond to the retinal non-perfusion area (NPA) and an area that is estimated to correspond to the neovascularization (NV).

The map generation unit <NUM> generates an NPA/NV existence probability map as image information in which the NPA existence probability and the NV existence probability are identifiably displayed in the fundus image information. Specifically, image information such as an NPA/NV existence probability map E illustrated in <FIG> is generated.

Thus, it is possible to generate information that serves as a basis for estimating the existence of the retinal non-perfusion area (NPA) and the neovascularization (NV) in the fundus image information.

Note that a method of identifiably displaying the NPA existence probability and the NV existence probability in the fundus image information is not particularly limited. For example, the NPA existence probability and the NV existence probability may be identifiably displayed according to a color difference, or the NPA existence probability and the NV existence probability may be identifiably displayed according to color shades.

The map generation unit <NUM> also generates an accompanying finding existence probability map (not shown) as image information in which the accompanying finding existence probability is identifiably displayed the fundus image information.

A method for identifiably displaying the accompanying finding existence probability in the fundus image information is not particularly limited. For example, the accompanying finding existence probability may be identifiably displayed according to a color difference, or the accompanying finding existence probability may be identifiably displayed according to color shades.

The estimated NPA/NV identification unit <NUM> identifies an area of the fundus image information that is estimated to correspond to the retinal non-perfusion area (NPA), as an estimated NPA, on the basis of the NPA existence probability and the NV existence probability, and identifies an area that is estimated to correspond to the neovascularization (NV), as an estimated NV. Specifically, in the fundus image information of the patient, an area in which the NPA existence probability and the NV existence probability exceed a predetermined threshold value is identified as an area that is estimated to correspond to the retinal non-perfusion area (NPA) (estimated NPA), or an area that is estimated to correspond to the neovascularization (estimated NV). Note that the threshold value may be discretionally changed according to judgment of the ophthalmologist D.

As a result, it is possible to early and easily identify an area where the existence of the retinal non-perfusion area (NPA) is estimated (estimated NPA) and an area where the existence of the neovascularization (NV) is estimated (estimated NV), in the image information based on the fundus image.

The estimated NPA/NV display control unit <NUM> executes a control for displaying the estimated NPA area and the estimated NV area on the fundus image information.

Thus, the area where the existence of the retinal non-perfusion is estimated (estimated NPA) and the area where the existence of the neovascularization (NV) is estimated (estimated NV) are displayed to be overlaid on the image information based on the fundus image.

Next, a flow of a series of processes executed by the server <NUM> having the functional configuration of <FIG> will be described with reference to <FIG>.

<FIG> is a flowchart illustrating a flow of a series of processes executed by the server <NUM> of <FIG>.

As shown in <FIG>, the server <NUM> executes the following series of processes.

In step S1, the image acquisition unit <NUM> determines whether or not fluorescein angiography image information has been transmitted from the examination device <NUM>.

In a case where the fluorescein angiography image information has been transmitted, the determination in step S1 is affirmative, and the procedure proceeds to step S2. On the other hand, in a case where the fluorescein angiography image information has not been transmitted, the determination in step S1 is negative, and the procedure returns to step S1. That is, the determination process of step S1 is repeated until the fluorescein angiography image information is transmitted. Thereafter, in a case where the fluorescein angiography image information is transmitted, the determination in step S1 becomes affirmative, and the procedure proceeds to step S2.

In step S2, the image acquisition unit <NUM> acquires the transmitted fluorescein angiography image information.

In step S3, the annotation acquisition unit <NUM> determines whether or not NPA/NV annotation information has been transmitted from the ophthalmologist terminal <NUM>.

In a case where the NPA/NV annotation information has been transmitted, the determination in step S3 is affirmative, and the procedure proceeds to step S4. On the other hand, in a case where the NPA/NV annotation information has not been transmitted, the determination in step S3 is negative, and the procedure returns to step S3. That is, the determination process of step S3 is repeated until the NPA/NV annotation information is transmitted. Thereafter, in a case where the NPA/NV annotation information is transmitted, the determination in step S3 becomes affirmative, and the procedure proceeds to step S4.

In step S4, the annotation acquisition unit <NUM> acquires the transmitted NPA/NV annotation information.

In step S5, the training information generation unit <NUM> generates NPA/NV training information on the basis of the fluorescein angiography image information and the NPA/NV annotation information corresponding to the fluorescein angiography image information.

In step S6, the image acquisition unit <NUM> determines whether or not fundus image information has been transmitted from the examination device <NUM>.

In a case where the fundus image information has been transmitted, the determination in step S6 is affirmative, and the procedure proceeds to step S7. On the other hand, in a case where the fundus image information has not been transmitted, the determination in step S6 is negative, and the procedure returns to step S6. That is, the determination process of step S6 is repeated until the fundus image information is transmitted. Thereafter, in a case where the fundus image information is transmitted, the determination in step S6 becomes affirmative, and the procedure proceeds to step S7.

In step S7, the image acquisition unit <NUM> acquires the transmitted fundus image information.

In step S8, the annotation acquisition unit <NUM> determines whether or not accompanying finding annotation information has been transmitted from the ophthalmologist terminal <NUM>.

In a case where the accompanying finding annotation information has been transmitted, the determination in step S8 is affirmative, and the procedure proceeds to Step S9. On the other hand, in a case where the accompanying finding annotation information has not been transmitted, the determination in step S8 is negative, and the procedure skips steps S9 to S11 and proceeds to step S12.

In step S9, the annotation acquisition unit <NUM> acquires the accompanying finding annotation information.

In step S10, the training information generation unit <NUM> generates the accompanying finding training information on the basis of the fluorescein angiography image information and the NPA/NV annotation information corresponding to the fluorescein angiography image information.

In step S11, the arithmetic unit <NUM> calculates an existence probability of an accompanying finding in the fundus image information on the basis of the accompanying finding training information.

In step S12, the map generation unit <NUM> generates an accompanying finding existence probability map as image information in which the accompanying finding existence probability is identifiably displayed in the fundus image information.

In step S13, the arithmetic unit <NUM> calculates the NPA existence probability and the NV existence probability on the basis of at least the NPA/NV training information. Further, in a case where the accompanying finding existence probability map is generated, the arithmetic unit <NUM> calculates the NPA existence probability and the NV existence probability on the basis of the NPA/NV training information and the accompanying finding existence probability map.

In step S14, the map generation unit <NUM> generates an NPA/NV existence probability map as image information in which the NPA existence probability and the NV existence probability are identifiably displayed in the fundus image information.

In step S15, the estimated NPA/NV identification unit <NUM> identifies an area that is estimated to correspond to the retinal non-perfusion area (NPA) in the fundus image information, as an estimated NPA, and identifies an area that is estimated to correspond to the neovascularization (NV), as an estimated NV, on the basis of the NPA existence probability and the NV existence probability.

In step S16, the estimated NPA/NV display control unit <NUM> executes a control for displaying the estimated NPA and the NV area on the fundus image information.

In step S17, the server <NUM> determines whether or not there is a process ending command.

In a case where there is no process ending command, the determination in step S17 is negative, and the procedure returns to step S1. On the other hand, in a case where there is the process ending command, the determination in step S17 is affirmative, and the procedure ends.

In a case where the server <NUM> executes the series of processes described above, the estimated NPA area and the estimated NV area are displayed in the fundus image information.

Next, a flow of various types of information used in various types of processes executed by the server <NUM> will be described with reference to <FIG>.

<FIG> is a diagram illustrating a flow of various types of information in the processes executed by the server <NUM>.

As shown in <FIG>, in a case where a fluorescein angiography image of a patient is captured in fundus fluorescein angiography, the server <NUM> acquires fluorescein angiography image information based on the fluorescein angiography image. A diagnostic note of the ophthalmologist D regarding at least one of the retinal non-perfusion area (NPA) or the neovascularization (NV) is attached to this fluorescein angiography image information. The diagnostic note forms the NPA/NV training information together with the fluorescein angiography image information as the NPA/NV annotation information.

In a case where a fundus image of a patient is captured in a fundus examination, fundus image information based on the fundus image is acquired by the server <NUM>. The fundus image information and the NPA/NV training information are used for a calculation process based on a NPA/NV annotation program of the arithmetic unit <NUM>. As a result of the calculation process based on the NPA/NV annotation program, an NPA/NV existence probability map is generated. An estimated NPA and an estimated NV in the fundus image information are identified on the basis of the NPA/NV existence probability map.

A diagnostic note of the ophthalmologist D regarding an accompanying finding may be attached to the fluorescein angiography image information and the fundus image information. The diagnostic note in this case forms the accompanying finding training information together with the fluorescein angiography image information and the fundus image information as the accompanying finding annotation information.

The fundus image information and the NPA/NV training information are used for a calculation process based on an accompanying finding determination program of the arithmetic unit <NUM>. As a result of the calculation process based on the accompanying finding determination program, an accompanying finding existence probability map is generated. In this manner, in a case where the accompanying finding existence probability map is generated, the estimated NPA and the estimated NV in the fundus image information are identified on the basis of the accompanying finding existence probability map and the NPA/NV training information.

<FIG> is a diagram illustrating an example of fundus image information acquired in the process executed by the server <NUM>.

<FIG> is a diagram illustrating an example of fluorescein angiography image information acquired in the process executed by the server <NUM>.

In examination of the fundus of a patient, imaging of the fundus of the patient is performed by the examination device <NUM>, and accordingly, fundus image information as shown in <FIG> is obtained. The ophthalmologist D performs diagnosis with reference to the fundus image information as shown in <FIG>. However, as shown in <FIG>, the fundus image information only shows findings resulting from abnormal perfusion such as bleeding and vitiligo, and it is difficult to read circulatory dynamics and an abnormal circulatory site. Therefore, fundus fluorescein angiography is performed, and accordingly, fluorescein angiography image information as shown in <FIG> is obtained. As shown in <FIG>, retinal circulatory dynamics can be clearly discriminated from the fluorescein angiography image information. However, the fundus fluorescein angiography is a burdensome examination for both patients and medical staffs due to the risk of the examination, its invasiveness, and geographical restrictions such as limitation to large hospitals, which causes hesitancy to take pictures of patients with good eyesight, patients with reduced physical functions, to perform repeated imaging, or the like, and thus, there are many cases in that detection of symptom appearances or disease states is delayed and treatment is delayed.

Accordingly, the server <NUM> performs an NPA/NV existence probability map generating process using the stored fluorescein angiography image information obtained from a plurality of patients and the NPA/NV training information generated on the basis of the NPA/NV annotation information attached to each piece of fluorescein angiography image information. Thus, it is possible to easily recognize abnormal circulatory findings from fundus images that can be easily captured at a nearby clinic or medical examination without performing the burdensome fundus fluorescein angiography.

<FIG> is a diagram illustrating an example of the NPA/NV existence probability map output in the NPA/NV existence probability map generating process executed by the server <NUM>.

An NPA/NV existence probability map E shown in <FIG> is image information in which the NPA existence probability and the NV existence probability are identifiably displayed in the fundus image information. In the NPA/NV existence probability map E, the NPA existence probability and the NV existence probability may be identifiably displayed according to color differences or color shades. For example, the identification display may be performed in such a manner that an area indicated by a warm color has a high NPA existence probability and a high NV existence probability and an area indicated by a cold color has a low NPA existence probability and a low NV existence probability. In addition, even in an area indicated by a cold color having a low NPA existence probability and a low NV existence probability, the identification display may be performed in such a manner that a dark color area has a lower NPA existence probability and a lower NV existence probability than a light color area.

<FIG> is a diagram illustrating an example of an estimated NPA and an estimated NV identified in the estimated NPA/NV recognizing process executed by the server <NUM>.

On the basis of the content of the NPA/NV existence probability map, the server <NUM> identifies an area of the fundus image information that is estimated to correspond to the retinal non-perfusion area (NPA), as an estimated NPA, and identifies an area that is estimated to correspond to the neovascularization (NV), as an estimated NV. For example, as shown in <FIG>, an area A indicated by a broken line may be set as the estimated NPA, and an area B may be set as the estimated NV. The estimated NPA and the estimated NV may be displayed to be superimposed on the fundus image information.

Thus, it is possible to quickly and easily identify an area in which the existence of a retinal non-perfusion is estimated and an area where the existence of a neovascularization is estimated, from a fundus image obtained by imaging the fundus of a patient, without performing the fundus fluorescein angiography that requires a special fundus camera or a diagnosis device.

One embodiment of the present invention has been described above, but the present invention is not limited to the above-described embodiment, and modifications, improvements, or the like within a range in which the object of the present invention can be achieved are included in the present invention.

For example, in the above-described embodiment, a configuration in which the image acquisition unit <NUM> acquires, in a case where various types of image information are transmitted from the examination device <NUM>, the various types of image information is shown, but a configuration in which the image acquisition unit <NUM> voluntarily goes to obtain various types of image information in a case where imaging is performed by the examination device <NUM> may be used. Similarly, a configuration in which the annotation acquisition unit <NUM> acquires, in a case where various types of annotation information are transmitted from the ophthalmologist terminal <NUM>, the various types of annotation information is shown, but a configuration in which in a case where the various types of annotation information is input to the ophthalmologist terminal <NUM>, the annotation acquisition unit <NUM> voluntarily goes to obtain the various types of annotation information may be used.

The hardware configurations shown in <FIG> are merely examples for achieving the object of the present invention, and are not particularly limited.

The functional block diagram shown in <FIG> is merely an example, and is not particularly limited. That is, it is sufficient if the information processing system has functions capable of executing the above-described series of processes as a whole, and what kind of functional block is used to realize the function is not particularly limited to the example of <FIG>.

Further, existence locations of the functional blocks are not limited to locations shown in <FIG>, and may be appropriately modified. For example, at least a part of the functional blocks of the server <NUM> may be provided in the ophthalmologist terminal <NUM> or the examination device <NUM>.

Then, one functional block may be configured by a hardware unit, or may be configured by a combination with a software unit.

In a case where the processing of each functional block is executed by software, a program that forms the software is installed in a computer or the like from a network or a recording medium.

The computer may be a computer embedded in dedicated hardware. In addition, the computer may be a computer capable of executing various functions by installation of various programs, for example, a general-purpose smartphone or a personal computer as well as a server.

The recording medium that stores such a program is not only configured by a removable medium that is distributed separately from a device body in order to provide the program to each user, but is also configured by a recording medium or the like provided to each user in a state of being incorporated in the device body in advance.

In the present specification, steps of describing the program recorded on the recording medium are not only processes sequentially performed in a time series manner, but may also include processes performed in parallel or individually although the processes are not necessarily performed in a time series manner.

For example, in step S6 of <FIG>, the image acquisition unit <NUM> determines whether or not the fundus image information has been transmitted from the examination device <NUM>, and acquires the fundus image information in step S7. However, in a case where the accompanying finding existence probability map generating process is executed, the fundus image information has only to be appropriately managed at a time point when the accompanying finding training information is generated. Further, in a case where the accompanying finding existence probability map generating process is not executed, the fundus image information has only to be appropriately managed at a time point when the NPA existence probability and the NV existence probability are calculated. Accordingly, in a case where the accompanying finding training information is generated, the fundus image information may be stored for management in the image DB <NUM> from any time point before the accompanying finding training information is generated in step S10. Further, in a case where the accompanying finding training information is not generated, the fundus image information may be stored for management in the image DB <NUM> from any time point before the NPA existence probability and the NV existence probability are calculated in step S13.

In summary, a program to which the present invention is applied may have the following configuration, and may have various embodiments.

That is, the program to which the present invention is applied causes a computer that controls the information processing device to execute: a fluorescein angiography image acquisition step (for example, step S2 in <FIG>) of acquiring fluorescein angiography image information (for example, fluorescein angiography image information C in <FIG>); an NPA/NV annotation acquisition step (for example, step S4 in <FIG>) of acquiring a diagnostic note of an ophthalmologist regarding at least one of a retinal non-perfusion area (NPA) or a neovascularization (NV), attached to the fluorescein angiography image information, as NPA/NV annotation information; an NPA/NV training information generation step (for example, step S5 in <FIG>) of generating NPA/NV training information that is training information for calculating an existence probability of the retinal non-perfusion area (NPA) and an existence probability of the neovascularization (NV), on the basis of the fluorescein angiography image information and the NPA/NV annotation information corresponding to the fluorescein angiography image information; a fundus image acquisition step (for example, step S7 in <FIG>) of acquiring fundus image information (for example, fundus image information F in <FIG>); an NPA/NV existence probability calculation step (for example, step S13 in <FIG>) of calculating the existence probability of the retinal non-perfusion area (NPA) and the existence probability of the neovascularization (NV) in the fundus image information on the basis of the NPA/NV training information; and an estimated NPA/NV identification step (for example, step S15 in <FIG>) of recognizing an area of the fundus image information that is estimated to correspond to the retinal non-perfusion area (NPA), as an estimated NPA, and recognizing an area that is estimated to correspond to the neovascularization (NV), as an estimated NV, on the basis of the existence probability of the retinal non-perfusion area and the existence probability of the neovascularization.

Thus, it is possible to quickly and easily identify the area in which the existence of retinal non-perfusion is estimated from the fundus image obtained by imaging the fundus of a patient, without performing fundus fluorescein angiography that requires a special fundus camera or a diagnosis device.

In addition, the program may cause the computer to execute the control process further including: an NPA/NV existence probability map generation step (for example, step S14 in <FIG>) of generating an NPA/NV existence probability map (for example, the NPA/NV existence probability map E in <FIG>) in which the NPA existence probability and the NV existence probability are identifiably displayed in the fundus image information.

In addition, the program may cause the computer to further execute: an accompanying annotation acquisition step (for example, step S9 in <FIG>) of acquiring, as accompanying finding annotation information, a diagnostic note of the ophthalmologist regarding an accompanying finding, attached to the fluorescein angiography image information and the fundus image information; an accompanying finding training information generation step (for example, step S10 in <FIG>) of generating accompanying finding training information that serves as training information for calculating an existence probability of the accompanying finding in the fundus image information, on the basis of the fluorescein angiography image information and the fundus image information, and the accompanying finding annotation information corresponding to the fluorescein angiography image information and the fundus image information; and an accompanying finding existence probability calculation step (for example, step S11 in <FIG>) of calculating the existence probability of the accompanying finding in the fundus image information on the basis of the accompanying finding training information, and in the NPA/NV existence probability calculation step, a control process of calculating the existence probability of the retinal non-perfusion area (NPA) and the existence probability of the neovascularization (NV) in the fundus image information on the basis of the accompanying finding existence probability and the NPA/NV training information may be executed.

Further, the program may cause the computer to further execute: an accompanying finding existence probability map generation step (for example, step S12 in <FIG>) of generating an accompanying finding existence probability map in which the existence probability of the accompanying finding is identifiably displayed in the fundus image information.

Hereinafter, a modified example of the embodiment of the present invention will be described with reference to the drawings.

<FIG> is a configuration diagram of an information processing system of a modified example according to the embodiment of the present invention. The information processing system of the modified example according to the embodiment of the present invention includes the ophthalmologist terminal <NUM>, the examination device <NUM>, a learning device <NUM>, and a diagnosis support device <NUM>. These devices are connected to each other through a network N.

The learning device <NUM> generates, on the basis of a fundus image that is an image of the fundus and an area of abnormality in blood circulation specified on the basis of a fluorescein angiography image of the fundus, a trained model indicating a relationship between the fundus image and the area of abnormality in blood circulation in the fundus image through learning. Here, the area of abnormality in blood circulation is an area in which blood circulation is abnormal due to blood damage on the retina that occurs in ocular ischemic diseases such as diabetic retinopathy.

The learning device <NUM> acquires information indicating a fundus image of a patient and information indicating a fluorescein angiography image of the patient, and stores the information indicating the fundus image and the information indicating the fluorescein angiography image acquired above in association with each other. Specifically, in a fundus examination of a patient, the examination device <NUM> captures a fundus image of the patient, creates fundus image notification information including a patient ID and information indicating the captured fundus image, to be addressed to the learning device <NUM>, and transmits the created fundus image notification information to the learning device <NUM>. In addition, in the fundus fluorescein angiography of the patient, the examination device <NUM> captures a fluorescein angiography image of the patient, creates fluorescein angiography image notification information including the patient ID and information indicating the captured fluorescein angiography image, to be addressed to the learning device <NUM>, and transmits the created fluorescein angiography image notification information to the learning device <NUM>.

The learning device <NUM> acquires the patient ID and the information indicating the fundus image included in the fundus image notification information transmitted from the examination device <NUM> to the learning device <NUM>, the patient ID and the information indicating the fluorescein angiography image included in the fluorescein angiography image notification information, and stores the patient ID, the information indicating the fundus image, and the information indicating the fluorescein angiography image acquired above in association with each other.

The learning device <NUM> acquires a diagnostic note of the ophthalmologist D regarding any one or both of the retinal non-perfusion area (NPA) and the neovascularization (NV), attached to the fluorescein angiography image of the patient, as NPA/NV annotation information. Specifically, in the fundus fluorescein angiography, in a case where the diagnostic note of the ophthalmologist D regarding the retinal non-perfusion area (NPA) and the neovascularization (NV) is attached to the fluorescein angiography image, the ophthalmologist terminal <NUM> creates NPA/NV annotation notification information including the patient ID and the diagnostic note, to be addressed to the learning device <NUM>, on the basis of an operation of the ophthalmologist D, and transmits the created NPA/NV annotation notification information to the learning device <NUM>.

The learning device <NUM> receives the NPA/NV annotation notification information transmitted by the ophthalmologist terminal <NUM>, and stores the NPA/NV annotation information included in the received NPA/NV annotation notification information. The information indicating the fluorescein angiography image and the NPA/NV annotation information attached to the fluorescein angiography image are stored in association with each other.

The learning device <NUM> acquires a diagnostic note of the ophthalmologist D regarding the accompanying finding, attached to the fundus image and the fluorescein angiography image, as accompanying finding annotation information. Specifically, in a case where the diagnostic note of the ophthalmologist D regarding the accompanying findings is attached to the information indicating the fluorescein angiography image and the information indicating the fundus image, the ophthalmologist terminal <NUM> creates accompanying finding annotation notification information including the patient ID and the diagnostic note, to be addressed to the learning device <NUM>, on the basis of an operation of the ophthalmologist D, and transmits the created accompanying finding annotation notification information to the learning device <NUM>. The learning device <NUM> receives the accompanying finding annotation notification information transmitted by the ophthalmologist terminal <NUM>, acquires the patient ID and the accompanying finding annotation information included in the received accompanying finding annotation notification information, and stores the acquired patient ID and the accompanying finding annotation information. The information indicating the fundus image, the information indicating the fluorescein angiography image, and the accompanying finding annotation information are stored in association with each other.

The learning device <NUM> acquires the information indicating the fundus image, the information indicating the fluorescein angiography image associated with the information indicating the fundus image, the NPA/NV annotation information, and the accompanying finding annotation information, and specifies an area of abnormality in blood circulation on the basis of the information indicating the fluorescein angiography image, the NPA/NV annotation information, and the accompanying finding annotation information acquired above. The learning device <NUM> generates NPA/NV learning information in which the information correlating the fundus image and the area of abnormality in blood circulation specified on the basis of the fluorescein angiography image corresponding to the fundus image, and stores the generated NPA/NV learning information.

The learning device <NUM> generates, using the information indicating the fundus image included in the NPA/NV learning information as input information, and using the area of abnormality in blood circulation specifiedon the basis of the fluorescein angiography image corresponding to the fundus image as training information, a trained model indicating a relationship between the fundus image and the area of abnormality in blood circulation in the fundus image through learning. A specific method of generating the trained model is not particularly limited. For example, a method for extracting a feature common in the fundus images having the area of abnormality in blood circulation from the NPA/NV learning information, and deriving a relationship between the extracted common feature and the area of abnormality in blood circulation in the fundus image may be used. The relationship between the fundus image and the area of abnormality in blood circulation in the fundus image may be derived using a neural network or a deep learning technique. The learning device <NUM> stores the generated trained model, creates trained model notification information including the trained model, to be addressed to the diagnosis support device <NUM>, and transmits the created trained model notification information to the diagnosis support device <NUM>.

The diagnosis support device <NUM> receives the trained model transmitted by the learning device <NUM>, and stores the received trained model.

The ophthalmologist terminal <NUM> creates patient information including the patient ID and the information indicating the fundus image of the patient, to be addressed to the diagnosis support device <NUM>, and transmits the created patient information to the diagnosis support device <NUM>.

The diagnosis support device <NUM> receives the patient information transmitted by the ophthalmologist terminal <NUM>, and acquires the patient ID and the information indicating the fundus image of the patient included in the received patient information. The diagnosis support device <NUM> specifies the area of abnormality in blood circulation in the fundus image on the basis of the acquired information indicating the fundus image, using the stored trained model. The diagnosis support device <NUM> creates a diagnosis result including the information indicating an area of abnormality in blood circulation in the fundus image identified using the patient fundus image and the trained model and the patient ID, to be addressed to the ophthalmologist terminal <NUM>, and transmits the created diagnosis result to the ophthalmologist terminal <NUM>.

Hereinafter, the learning device <NUM> and the diagnosis support device <NUM> included in the information processing system will be described.

<FIG> is a block diagram illustrating an example of a learning device according to a modified example of the embodiment of the present invention.

The learning device <NUM> includes a communication unit <NUM>, a storage unit <NUM>, an operation unit <NUM>, an information processing unit <NUM>, a display unit <NUM>, and a bus line <NUM> such as an address bus or a data bus for electrically connecting the respective components as shown in <FIG>.

The communication unit <NUM> is realized by a communication module. The communication unit <NUM> communicates with external communication devices such as the ophthalmologist terminal <NUM>, the examination device <NUM>, or the diagnosis support device <NUM> through the network N. Specifically, the communication unit <NUM> receives fundus image notification information transmitted by the examination device <NUM>, and outputs the received fundus image notification information to the information processing unit <NUM>. The communication unit <NUM> receives fluorescein angiography image notification information transmitted by the examination device <NUM>, and outputs the received fluorescein angiography image notification information to the information processing unit <NUM>. The communication unit <NUM> receives NPA/NV annotation notification information transmitted by the ophthalmologist terminal <NUM>, and outputs the received NPA/NV annotation notification information to the information processing unit <NUM>. The communication unit <NUM> receives accompanying finding annotation notification information transmitted by the ophthalmologist terminal <NUM>, and outputs the received accompanying finding annotation notification information to the information processing unit <NUM>. The communication unit <NUM> acquires trained model notification information output by the information processing unit <NUM>, and transmits the acquired trained model notification information to the diagnosis support device <NUM>.

The storage unit <NUM> is realized by, for example, a random access memory (RAM), a read only memory (ROM), a hard disk drive (HDD), a flash memory, or a hybrid storage device in which a plurality of these memories are combined. The storage unit <NUM> stores a program <NUM> executed by the information processing unit <NUM>, an application <NUM>, an image DB <NUM>, an annotation DB <NUM>, learning information storage <NUM>, and a trained model <NUM>.

The program <NUM> is, for example, an operating system, which is located between a user or an application program and hardware, provides a standard interface for the user or the application program, and performs efficient management for each resource such as hardware.

The application <NUM> causes the learning device <NUM> to receive the fundus image notification information transmitted by the examination device <NUM>, and to store the patient ID and the information indicating the fundus image included in the received fundus image notification information in association with each other. The application <NUM> causes the learning device <NUM> to receive the fluorescein angiography image notification information transmitted by the examination device <NUM>, and to store the patient ID and the information indicating the fluorescein angiography image included in the received fluorescein angiography image notification information in association with each other. The application <NUM> causes the learning device <NUM> to receive the NPA/NV annotation notification information transmitted by the ophthalmologist terminal <NUM>, and to store the patient ID and the NPA/NV annotation information included in the received NPA/NV annotation notification information in association with each other.

The application <NUM> causes the learning device <NUM> to receive the accompanying finding annotation notification information transmitted by the ophthalmologist terminal <NUM>, and to store the patient ID and the accompanying finding annotation information included in the received accompanying finding annotation notification information in association with each other. The application <NUM> causes the learning device <NUM> to acquire the information indicating the fundus image, the information indicating the fluorescein angiography image, the NPA/NV annotation information, and the accompanying finding annotation information associated with the patient ID. The application <NUM> causes the learning device <NUM> to specify the area of abnormality in blood circulation on the basis of the information indicating the fluorescein angiography image, the NPA/NV annotation information, and the accompanying finding annotation information acquired above.

The application <NUM> causes the learning device <NUM> to generate NPA/NV learning information in which the information indicating the fundus image and the area of abnormality in blood circulation specified on the basis of the fluorescein angiography image corresponding to the fundus image are associated with each other, and to store the generated NPA/NV learning information. The application <NUM> causes the learning device <NUM> to generate, using the information indicating the fundus image included in the NPA/NV learning information as input information, and using the area of abnormality in blood circulation specified on the basis of the fluorescein angiography image corresponding to the fundus image as training information, a trained model indicating a relationship between the fundus image and the area of abnormality in blood circulation in the fundus image through learning. The application <NUM> causes the learning device <NUM> to store the generated trained model, and to transmit the trained model to the diagnosis support device <NUM>.

The image DB <NUM> stores the patient ID, the information indicating the fundus image, and the information indicating the fluorescein angiography image in association with each other.

The annotation DB <NUM> stores the patient ID, the NPA/NV annotation information, and the accompanying finding annotation information in association with each other.

The learning information storage <NUM> stores the NPA/NV learning information in which the information indicating the fundus image and the area of abnormality in blood circulation specified on the basis of the fluorescein angiography image corresponding to the fundus image in association with each other.

The trained model <NUM> stores, using the information indicating the fundus image included in the NPA/NV learning information as input information, and using the area of abnormality in blood circulation specified on the basis of the fluorescein angiography image corresponding to the fundus image as training information, the trained model indicating the relationship between the fundus image and the area of abnormality in blood circulation in the fundus image.

The operation unit <NUM> is configured by, for example, a touch panel, which detects a touch operation on a screen displayed on the display unit <NUM>, and outputs a detection result of the touch operation to the information processing unit <NUM>.

The display unit <NUM> is configured by, for example, a touch panel, which displays a screen for receiving the information indicating the fundus image received by the learning device <NUM> and the information indicating the fluorescein angiography image. Further, the display unit <NUM> displays a screen for receiving an operation of processing the NPA/NV annotation information received by the learning device <NUM> and the accompanying finding annotation information.

The entirety or a part of the information processing unit <NUM> is, for example, a function unit (hereinafter, referred to as a software function unit) realized as a processor such as a central processing unit (CPU) executes the program <NUM> stored in the storage unit <NUM> and the application <NUM>. The entirety or a part of the information processing unit <NUM> may be realized by hardware such as a large scale integration (LSI), an application specific integrated circuit (ASIC), or a field-programmable gate array (FPGA), or may be realized by a combination of a software function unit and hardware. The information processing unit <NUM> includes, for example, an image acquisition unit <NUM>, an annotation acquisition unit <NUM>, a learning information generation unit <NUM>, and a learning unit <NUM>.

The image acquisition unit <NUM> acquires the fundus image notification information output by the communication unit <NUM>, and acquires the patient ID and the information indicating the fundus image included in the acquired fundus image notification information. The image acquisition unit <NUM> stores the patient ID and the information indicating the fundus image acquired above in association with each other in the image DB <NUM>.

The image acquisition unit <NUM> acquires the fluorescein angiography image notification information output by the communication unit <NUM>, and acquires the patient ID and the information indicating the fluorescein angiography image included in the acquired fluorescein angiography image notification information. The image acquisition unit <NUM> stores the patient ID and the information indicating the fluorescein angiography image acquired above in association with each other in the image DB <NUM>.

The annotation acquisition unit <NUM> acquires the NPA/NV annotation notification information output by the communication unit <NUM>, and acquires the patient ID and the NPA/NV annotation information included in the acquired NPA/NV annotation notification information. The annotation acquisition unit <NUM> stores the patient ID and the NPA/NV annotation information acquired above in association with each other in the annotation DB <NUM>.

The annotation acquisition unit <NUM> acquires the accompanying finding annotation notification information output by the communication unit <NUM>, and acquires the patient ID and the accompanying finding annotation information included in the acquired accompanying finding annotation notification information. The annotation acquisition unit <NUM> stores the patient ID and the accompanying finding annotation information acquired above in association with each other in the annotation DB <NUM>.

The learning information generation unit <NUM> acquires the patient ID, the information indicating the fundus image associated with the patient ID, and the information indicating the fluorescein angiography image stored in the image DB <NUM> of the storage unit <NUM>. The learning information generation unit <NUM> acquires the patient ID, the NPA/NV annotation information associated with the patient ID, and the accompanying finding annotation information stored in the annotation DB <NUM> of the storage unit <NUM>. The learning information generation unit <NUM> specifies the area of abnormality in blood circulation on the basis of the information indicating the fluorescein angiography image, the NPA/NV annotation information, and the accompanying finding annotation information acquired above.

Hereinafter, each configuration of the learning device <NUM> will be specifically described.

<FIG> is a diagram illustrating an example of a fundus image, and <FIG> is a diagram illustrating an example of a fluorescein angiography image.

The learning information generation unit <NUM> extracts a green component of the fundus image, and removes a noise component from a green component extraction fundus image that is a fundus image obtained by the extraction of the green component. The learning information generation unit <NUM> divides the green component extraction fundus image from which the noise component is removed into rectangles. The learning information generation unit <NUM> resizes the green component extraction fundus image from which the noise component is removed to a predetermined size on the basis of the image divided into rectangles. In resizing the green component extraction fundus image into the predetermined size, interpolation is performed by an interpolation method such as bicubic interpolation.

The learning information generation unit <NUM> extracts a green component of the fluorescein angiography image, and removes a noise component from a green component extraction fluorescein angiography image which is a fluorescein angiography image obtained by the extraction of the green component. The learning information generation unit <NUM> divides the green component extraction fluorescein angiography image from which the noise component is removed into rectangles. The learning information generation unit <NUM> resizes the green component extraction fluorescein angiography image from which the noise component is removed to a predetermined size on the basis of the image divided into rectangles. In resizing the green component extraction fluorescein angiography image into the predetermined size, interpolation is performed by an interpolation method such as bicubic interpolation.

The learning information generation unit <NUM> corrects rotational components of the green component extraction fundus image from which the noise component is removed and the green component extraction fluorescein angiography image from which the noise component is removed so that positions of eyeballs coincide with each other, and performs resizing to a predetermined size. The learning information generation unit <NUM> corrects the rotational components so that the positions of the eyeballs coincide with each other, and specifies an area of abnormality in blood circulation on the basis of the green component extraction fundus image from which the noise component is removed and the green component extraction fluorescein angiography image from which the noise component is removed, resized to the predetermined size.

<FIG> is a diagram illustrating an example of an area of abnormality in blood circulation.

The learning information generation unit <NUM> generates NPA/NV learning information in which the information indicating the fundus image and the area of abnormality in blood circulation specified on the basis of the fluorescein angiography image corresponding to the fundus image are associated with each other, and stores the generated NPA/NV learning information in the learning information storage <NUM> of the storage unit <NUM>. Returning to <FIG>, the description will be continued.

The learning unit <NUM> acquires the NPA/NV learning information stored in the learning information storage <NUM> of the storage unit <NUM>. The learning unit <NUM> acquires information indicating the fundus image and the information indicating the area of abnormality in blood circulation included in the acquired NPA/NV learning information. The learning unit <NUM> generates, using the acquired information indicating the fundus image as input information, and using the area of abnormality in blood circulation specified on the basis of the fluorescein angiography image corresponding to the fundus image as training information, a trained model indicating a relationship between the fundus image and the area of abnormality in blood circulation in the fundus image through learning.

Specifically, in the modified example of the present embodiment, assuming that it is difficult to perform learning and prediction for entire images at once due to memory limitation of a graphics processing unit (GPU), a case where a method for extracting patches from images and causing a neural network to learn the extracted patches is used will be described. Here, as an example, the size of the patch is set to <NUM> px × <NUM> px, and its stride (interval for moving a frame for patch extraction) is set to <NUM> px.

The generated patches are divided into two groups of a group containing a positive area and a group containing no positive area. The patches are selected so that the proportions of both groups used for learning become equal.

A phenomenon in which a neural network shows good performance only for learning data itself or an image very similar to the learning data and shows extremely low performance for unknown images is called over-fitting, which may be improved by collecting a larger amount of samples or applying a geometrical operation such as rotation to the learning data.

In the modified example of the present embodiment, after the patches were generated, a rotation angle was determined on the basis of a normal distribution of σ=<NUM> deg. , a horizontal inversion was performed with a probability of <NUM>% and a vertical inversion was performed with a probability of <NUM>%.

<FIG> is a diagram illustrating an example of a structure of a neural network.

One example of the structure of the neural network is based on U-Net. In <FIG>, b represents convolution (kernel_size=(<NUM>,<NUM>)), g represents max pooling (pool_size=(<NUM>,<NUM>)), and o represents up sampling (size=(<NUM>,<NUM>)). After each convolution layer, a batch normalization was performed using an activation function ReLU. However, in the last convolution layer, the batch normalization was not performed by using sigmoid as an activation function.

Further, arrows indicated by a1, a2, a3, and a4 represent skip connection by concatenation. This is considered to contribute to restoration of image position information.

The learning unit <NUM> stores the generated trained model in the trained model <NUM> of the storage unit <NUM>.

The learning unit <NUM> creates trained model notification information including the created trained model, to be addressed to the diagnosis support device <NUM>, and outputs the created trained model notification information to the communication unit <NUM>.

<FIG> is a block diagram illustrating an example of a diagnosis support device according to a modified example of the embodiment of the present invention.

The diagnosis support device <NUM> includes a communication unit <NUM>, a storage unit <NUM>, an operation unit <NUM>, an information processing unit <NUM>, a display unit <NUM>, and a bus line <NUM> such as an address bus or a data bus for electrically connecting the respective components as shown in <FIG>.

The communication unit <NUM> is realized by a communication module. The communication unit <NUM> communicates with external communication devices such as the ophthalmologist terminal <NUM> or the learning device <NUM> through the network N. Specifically, the communication unit <NUM> receives trained model notification information transmitted by the learning device <NUM>, and outputs the received trained model notification information to the information processing unit <NUM>. The communication unit <NUM> receives patient information transmitted by the ophthalmologist terminal <NUM>, and outputs the received patient information to the information processing unit <NUM>. The communication unit <NUM> acquires diagnostic information output by the information processing unit <NUM>, and transmits the acquired diagnostic information to the ophthalmologist terminal <NUM>.

The storage unit <NUM> is realized by, for example, a RAM, a ROM, an HDD, a flash memory, or a hybrid storage device in which a plurality of these memories are combined. The storage unit <NUM> stores a program <NUM> executed by the information processing unit <NUM>, an application <NUM>, and a trained model <NUM>.

The application <NUM> causes the diagnosis support device <NUM> to receive the trained model notification information transmitted by the learning device <NUM>, and to store the trained model included in the received trained model notification information. The application <NUM> causes the diagnosis support device <NUM> to receive the patient information transmitted by the ophthalmologist terminal <NUM>, and to acquire a patient ID and a fundus image included in the received patient information. The application <NUM> causes the diagnosis support device <NUM> to recognize an area of abnormality in blood circulation in the acquired fundus image using the stored trained model.

The application <NUM> causes the diagnosis support device <NUM> to create diagnostic information including the fundus image of the patient, information indicating the area of abnormality in blood circulation identified using the fundus image of the patient and the trained model, and the patient ID, to be addressed to the ophthalmologist terminal <NUM>, and to transmit the created diagnostic information to the ophthalmologist terminal <NUM>.

The operation unit <NUM> includes, for example, a touch panel, which detects a touch operation on a screen displayed on the display unit <NUM> and outputs a detection result of the touch operation to the information processing unit <NUM>.

The display unit <NUM> includes, for example, a touch panel, which displays a screen that receives information indicating the fundus image included in the patient information received by the diagnosis support device <NUM>. In addition, the display unit <NUM> displays a result of the diagnosis made by the diagnosis support device <NUM>.

The entirety or a part of the information processing unit <NUM> is a software function unit realized as a processor such as a CPU executes the program <NUM> stored in the storage unit <NUM> and the application <NUM>, for example. The entirety or a part of the information processing unit <NUM> may be realized by hardware such as LSI, ASIC, or FPGA, or may be realized by a combination of a software function unit and hardware. The information processing unit <NUM> includes, for example, a reception unit <NUM>, a identification unit <NUM>, and a creation unit <NUM>.

The reception unit <NUM> acquires the trained model notification information output by the communication unit <NUM>, and acquires the trained model included in the acquired trained model notification information. The reception unit <NUM> receives the acquired trained model, and stores the received trained model in the trained model <NUM> of the storage unit <NUM>.

The reception unit <NUM> acquires the patient information output by the communication unit <NUM>, and acquires the patient ID and the information indicating the fundus image included in the acquired patient information. The reception unit <NUM> receives the acquired patient ID and the information indicating the fundus image, and outputs the patient ID and the information indicating the fundus image received above to the identification unit <NUM>.

The identification unit <NUM> acquires the patient ID and the information indicating the fundus image output by the reception unit <NUM>. The identification unit <NUM> acquires the trained model stored in the trained model <NUM> of the storage unit <NUM>, and specifies the area of abnormality in blood circulation in the acquired fundus image using the acquired trained model. The identification unit <NUM> outputs the information indicating the area of abnormality in blood circulation in the identified fundus image and the patient ID to the creation unit <NUM>.

Specifically, in the modified example of the present embodiment, similarly to the learning device <NUM>, a case where a patch is extracted from an image and an area of abnormality in blood circulation in a fundus image is identified using the extracted patch and a trained model will be described. Here, as an example, the size of the patch is set to <NUM> px × <NUM> px, and its stride (interval for moving a frame for patch extraction) is set to <NUM> px. All of the generated patches are selected.

The identification unit <NUM> acquires an image of <NUM> × <NUM> × <NUM> on the basis of the trained model. The identification unit <NUM> votes the acquired pixel values to corresponding pixels of an original image and performs averaging therefor. Here, the identification unit <NUM> may convert the acquired image into color display.

The creation unit <NUM> acquires the patient ID and the information indicating the area of abnormality in blood circulation in the fundus image output by the identification unit <NUM>. The creation unit <NUM> creates the diagnostic information including the acquired patient ID and the information indicating the area of abnormality in blood circulation in the fundus image, to be addressed to the ophthalmologist terminal <NUM>. The creation unit <NUM> outputs the created diagnostic information to the communication unit <NUM>.

An example of an operation of the information processing system according to the modified example of this embodiment will be described with reference to <FIG>.

<FIG> is a flowchart illustrating an example of an operation of a learning device included in the information processing system of the modified example of the present embodiment. <FIG> illustrates an operation after the ophthalmologist terminal <NUM> transmits NPA/NV annotation notification information and accompanying finding annotation notification information to the learning device <NUM> and the examination device <NUM> transmits fundus image notification information and fluorescein angiography image notification information to the learning device <NUM>.

The communication unit <NUM> of the learning device <NUM> receives the fundus image notification information transmitted by the examination device <NUM>, and outputs the received fundus image notification information to the information processing unit <NUM>. The image acquisition unit <NUM> of the information processing unit <NUM> acquires the fundus image notification information output by the communication unit <NUM>, stores the patient ID and the information indicating the fundus image included in the acquired fundus image notification information in association with each other in the image DB <NUM> of the storage unit <NUM>.

The communication unit <NUM> of the learning device <NUM> receives the fluorescein angiography image notification information transmitted by the examination device <NUM>, and outputs the received fluorescein angiography image notification information to the information processing unit <NUM>. The image acquisition unit <NUM> of the information processing unit <NUM> acquires the fluorescein angiography image notification information output by the communication unit <NUM>, and stores the patient ID and the information indicating the fluorescein angiography image included in the acquired fluorescein angiography image notification information in association with each other in the image DB <NUM> of the storage unit <NUM>.

The communication unit <NUM> of the learning device <NUM> receives the NPA/NV annotation notification information transmitted by the ophthalmologist terminal <NUM>, and outputs the received NPA/NV annotation notification information to the information processing unit <NUM>. The annotation acquisition unit <NUM> of the information processing unit <NUM> acquires the NPA/NV annotation notification information output by the communication unit <NUM>, and stores the patient ID and the NPA/NV annotation information included in the acquired NPA/NV annotation notification information in association with each other in the annotation DB <NUM> of the storage unit <NUM>.

The communication unit <NUM> of the learning device <NUM> receives the accompanying finding annotation notification information transmitted by the ophthalmologist terminal <NUM>, and outputs the received accompanying finding annotation notification information to the information processing unit <NUM>. The annotation acquisition unit <NUM> of the information processing unit <NUM> acquires the accompanying finding annotation notification information output by the communication unit <NUM>, stores the patient ID and the accompanying finding annotation information included in the acquired accompanying finding annotation notification information in association with each other in the annotation DB <NUM> of the storage unit <NUM>.

The learning information generation unit <NUM> of the learning device <NUM> acquires the patient ID, the information indicating the fundus image associated with the patient ID, and the information indicating the fluorescein angiography image stored in the image DB <NUM> of the storage unit <NUM>. The learning information generation unit <NUM> acquires the patient ID, the NPA/NV annotation information associated with the patient ID, and the accompanying finding annotation information stored in the annotation DB <NUM> of the storage unit <NUM>. The learning information generation unit <NUM> specifies the area of abnormality in blood circulation on the basis of the information indicating the fluorescein angiography image, the NPA/NV annotation information, and the accompanying finding annotation information acquired above. The learning information generation unit <NUM> generates NPA/NV learning information in which the information indicating the fundus image and the area of abnormality in blood circulation specified on the basis of the fluorescein angiography image corresponding to the fundus image are associated with each other, and stores the generated NPA/NV learning information in the learning information storage <NUM> of the storage unit <NUM>.

The learning unit <NUM> of the learning device <NUM> acquires the NPA/NV learning information stored in the learning information storage <NUM> of the storage unit <NUM>. The learning unit <NUM> acquires information indicating the fundus image and the information indicating the area of abnormality in blood circulation included in the acquired NPA/NV learning information. The learning unit <NUM> generates, using the acquired information indicating the fundus image as input information, and using the area of abnormality in blood circulation specified on the basis of the fluorescein angiography image corresponding to the fundus image as training information, a trained model indicating a relationship between the fundus image and the area of abnormality in blood circulation in the fundus image through learning. The learning unit <NUM> stores the generated trained model in the trained model <NUM> of the storage unit <NUM>.

In the flowchart shown in <FIG>, the order of steps S201, S202, and S203 may be changed.

According to the flowchart shown in <FIG>, the learning device <NUM> may specify the area of abnormality in blood circulation on the basis of the information indicating the fluorescein angiography image, the NPA/NV annotation information, and the accompanying finding annotation information. The learning device <NUM> may generate, using the information indicating the fundus image as input information, and using the area of abnormality in blood circulation specified on the basis of the fluorescein angiography image corresponding to the fundus image as training information, a trained model indicating a relationship between the fundus image and the area of abnormality in blood circulation in the fundus image through learning.

<FIG> is a flowchart illustrating an example of an operation of a diagnosis support device included in the information processing system of the modified example of the present embodiment. <FIG> shows an operation after the learning device <NUM> transmits the trained model notification information to the diagnosis support device <NUM> and the ophthalmologist terminal <NUM> transmits the patient information to the diagnosis support device <NUM>.

The communication unit <NUM> of the diagnosis support device <NUM> receives the trained model notification information transmitted by the learning device <NUM>, and outputs the received trained model notification information to the information processing unit <NUM>. The reception unit <NUM> acquires the trained model notification information output by the communication unit <NUM>, and acquires the trained model included in the acquired trained model notification information. The reception unit <NUM> receives the acquired trained model, and stores the received trained model in the trained model <NUM> of the storage unit <NUM>.

The communication unit <NUM> receives patient information transmitted by the ophthalmologist terminal <NUM>, and outputs the received patient information to the information processing unit <NUM>. The reception unit <NUM> acquires the patient information output by the communication unit <NUM>, and acquires the patient ID and the information indicating the fundus image included in the acquired patient information. The reception unit <NUM> receives the acquired patient ID and the information indicating the fundus image, and outputs the patient ID and the information indicating the fundus image received above to the identification unit <NUM>.

The identification unit <NUM> acquires the patient ID and the information indicating the fundus image output by the reception unit <NUM>. The identification unit <NUM> acquires the trained model stored in the trained model <NUM> of the storage unit <NUM>.

The identification unit <NUM> acquires the trained model stored in the trained model <NUM> of the storage unit <NUM>, and recognizes the area of abnormality in blood circulation in the fundus image that is a identification target using the acquired trained model. The identification unit <NUM> outputs the information indicating the area of abnormality in blood circulation in the identified fundus image and the patient ID to the creation unit <NUM>.

The creation unit <NUM> acquires the patient ID and the information indicating the area of abnormality in blood circulation in the fundus image output by the identification unit <NUM>. The creation unit <NUM> creates diagnostic information including the patient ID and the information indicating the area of abnormality in blood circulation in the acquired fundus image, to be addressed to the ophthalmologist terminal <NUM>. The creation unit <NUM> outputs the created diagnostic information to the communication unit <NUM>.

The communication unit <NUM> acquires the diagnostic information output by the creation unit <NUM>, and transmits the acquired diagnostic information to the ophthalmologist terminal <NUM>.

According to the flowchart shown in <FIG>, the diagnosis support device <NUM> may recognize the area of abnormality in blood circulation in the fundus image that is a identification target, using the trained model indicating the relationship between the fundus image generated by using the information indicating the fundus image as input information and using the area of abnormality in blood circulation specified on the basis of the fluorescein angiography image corresponding to the fundus image as training information, and the area of abnormality in blood circulation in the fundus image.

In the above-described modified example, the diagnosis support device <NUM> receives the trained model notification information transmitted by the learning device <NUM>, and stores the trained model included in the received trained model notification information in the storage unit <NUM>. A case where the diagnosis support device <NUM> specifies the area of abnormality in blood circulation in the fundus image using the fundus image included in the patient information transmitted by the ophthalmologist terminal <NUM> and the stored trained model and transmits the diagnostic result including information indicating the identified area of abnormality in blood circulation to the ophthalmologist terminal <NUM> has been described, but the present invention is not limited to this example. For example, a configuration in which the diagnosis support device <NUM> transmits the patient information to the learning device <NUM> may be used. The learning device <NUM> receives the patient information transmitted by the diagnosis support device <NUM>, and specifies the area of abnormality in blood circulation in the fundus image using the fundus image included in the received patient information and the trained model. The learning device <NUM> creates a diagnostic result including the information indicating the identified area of abnormality in blood circulation, and transmits the created diagnostic result to the diagnosis support device <NUM>. The diagnosis support device <NUM> may receive the diagnostic result transmitted by the learning device <NUM>, and may transmit the received diagnostic result to the ophthalmologist terminal <NUM>.

According to the modified example of the present embodiment, the diagnosis support device <NUM> includes an identification unit that specifies an area of abnormality in blood circulation in a fundus image that is an image of the fundus, using a trained model obtained by learning a relationship between the fundus image and the area of abnormality in blood circulation in the fundus image, on the basis of the fundus image and the area of abnormality in blood circulation specified on the basis of a fluorescein angiography image of the fundus, and an output unit that outputs information indicating the fundus image of a patient and the area of abnormality in blood circulation in the fundus image of the patient identified by the identification unit using the trained model. With this configuration, the diagnosis support device <NUM> can estimate an abnormal circulation area with high accuracy from a normal fundus image using a trained model in which abnormal circulation area information specified by a doctor from a fluorescein angiography image and a fundus image corresponding thereto are used as training information.

The area of abnormality in blood circulation is generated on the basis of the fluorescein angiography image and an ophthalmologist's diagnostic note regarding one or both of a retinal non-perfusion area and a neovascularization attached to the fluorescein angiography image. With this configuration, it is possible to generate the area of abnormality in blood circulation that serves as the training information on the basis of the fluorescein angiography image and the ophthalmologist's diagnostic note regarding one or both of the retinal non-perfusion area and the neovascularization attached to the fluorescein angiography image.

The identification unit recognizes one or both of the retinal non-perfusion area and an area corresponding to the neovascularization in the fundus image. With this configuration, it is possible to identify one or both of the retinal non-perfusion area and the area corresponding to the neovascularization with high accuracy from the normal fundus image.

The output unit outputs an image in which the area of abnormality in blood circulation identified by the identification unit is overlaid on the fundus image. With this configuration, it is possible to acquire an image in which the abnormal circulation area is superimposed on the normal fundus image from the normal fundus image.

According to the modified example of the present embodiment, the learning device <NUM> includes a learning unit that generates, on the basis of a fundus image that is an image of the fundus and an area of abnormality in blood circulation specified on the basis of a fluorescein angiography image of the fundus, a trained model indicating a relationship between the fundus image and the area of abnormality in blood circulation in the fundus image, through learning. With this configuration, the learning device <NUM> can generate the trained model indicating the relationship between the fundus image and the area of abnormality in blood circulation in the fundus image through machine learning.

Note that the server <NUM>, the ophthalmologist terminal <NUM>, the examination device <NUM>, the learning device <NUM>, and the diagnosis support device <NUM> have a computer therein. Further, steps of the respective processes of each device described above are stored in a computer-readable recording medium in the form of a program, and the above processes are performed as the computer reads and executes the program. Here, the computer-readable recording medium includes a magnetic disc, a magneto-optical disc, a CD-ROM, a DVD-ROM, a semiconductor memory, or the like. Further, the computer program may be distributed to the computer through a communication line, and the computer that receives the distribution may execute the program.

In addition, the program may be a program for realizing some of the functions described above.

Claim 1:
A method for supporting diagnosis comprising:
receiving a fundus image,
processing the fundus image using a trained model (<NUM>) configured to recognize an area of abnormality in blood circulation in the fundus image,
wherein the trained model has been trained based upon a training data set comprising of a training fundus image, a fluorescent angiography image, and information of blood circulation abnormality associated with the fluorescent angiography image; and
outputting information relating to the area of abnormality in blood circulation in the fundus image,
wherein the fluorescent angiography image corresponds to the training fundus image and
wherein the fluorescent angiography image and the training fundus image is acquired from same patient.