Patent Description:
<CIT> hereinafter Patent Document <NUM> discloses technology for detecting retinal vascular infraction in a test subject.

Each of <CIT>, <NPL>, and <NPL> forms part of the state of the art.

According to a first aspect of the present invention, there is provided an image processing method as recited in claim <NUM>.

According to a second aspect of the present invention, there is provided an image processing program as recited in claim <NUM>.

According to a third aspect of the present invention, there is provided an image processing device as recited in claim <NUM> below.

The dependent claims define particular embodiments and implementations of each respective aspect.

Detailed description follows regarding exemplary embodiments of the present invention, with reference to the drawings. In the following, for ease of explanation a scanning laser ophthalmoscope will be referred to as "SLO". Moreover, for ease of explanation optical coherence tomography will be referred to as "OCT".

A configuration of an ophthalmic system <NUM> will now be described with reference to <FIG>. As illustrated in the example of <FIG>, the ophthalmic system <NUM> includes an ophthalmic device <NUM>, a field of view measurement instrument <NUM>, a laser photocoagulator <NUM>, a management server device (hereafter referred to as "management server") <NUM>, and an image display device (hereafter referred to as "image viewer") <NUM>.

The ophthalmic device <NUM> acquires fundus images and tomographic images. The field of view measurement instrument <NUM> measures the field of view of a patient. The laser photocoagulator <NUM> uses a laser to coagulate pathological lesions on a fundus of the patient in order to suppress pathological progression. The management server <NUM> stores plural fundus images, obtained by imaging the fundus of plural patients using the ophthalmic device <NUM>, in association with IDs of the patients, and predicts any non perfusion areas (NPAs) in a specified fundus image. The image viewer <NUM> displays an image of non perfusion area (NPA) candidates predicted by the management server <NUM>.

These non perfusion areas (NPAs) are areas on a fundus where there is no blood flow or hardly any blood flow due to retina capillary vascular bed obstruction or the like, and may also be avascular areas (AVAs) which are areas on a fundus where there are no blood vessels or only sparse blood vessels.

The ophthalmic device <NUM>, the field of view measurement instrument <NUM>, the laser photocoagulator <NUM>, the management server <NUM>, and the image viewer <NUM> are all interconnected over a network <NUM>.

The management server <NUM> is an example of an "image processing device" of the technology disclosed herein. The image viewer <NUM> is an example of an "image display device" of the technology disclosed herein.

Next, description follows regarding a configuration of the ophthalmic device <NUM>, with reference to <FIG>. As illustrated in <FIG>, the ophthalmic device <NUM> includes an imaging device <NUM> and a control device <NUM>. The imaging device <NUM> images the fundus of a subject eye. The control device <NUM> is realized by a computer including a central processing unit (CPU) 16A, random access memory (RAM) 16B, read only memory (ROM) 16C, and an input/output (I/O) port 16D.

A storage device <NUM> is connected to the input/output (I/O) port 16D. Note that the storage device <NUM> is configured, for example, by non-volatile memory ((NVM) or a hard disk). The input/output (I/O) port 16D is connected to the network <NUM> through a communication interface (I/F) <NUM>.

The control device <NUM> includes an input/display device 16E connected to the CPU 16A through the I/O port 16D. The input/display device 16E displays images obtained by imaging, and includes a graphical user interface to receive various instructions including an instruction to perform imaging. Examples of the graphical user interface include a touch panel display. Note that in the following, for convenience, "imaging" refers to a user using the ophthalmic device <NUM> to acquire an image of an imaging subject.

The imaging device <NUM> operates under control from the control device <NUM>. The imaging device <NUM> includes a SLO unit <NUM>, a wide angled optical system <NUM>, and an OCT unit <NUM>.

In the following description, when the ophthalmic device <NUM> is installed on a horizontal plane, the horizontal direction is referred to as the "X direction", a direction perpendicular to the horizontal direction is referred to as the "Y direction", and a direction connecting a pupil center <NUM> at the anterior segment of a subject eye <NUM> and an eyeball center O of the subject eye <NUM> is referred to as the "Z direction". Accordingly, the X direction, Y direction, and Z direction are mutually perpendicular directions.

The ophthalmic device <NUM> according to the present exemplary embodiment includes two functions, i.e. a first function and a second function, as examples of main functions that can be implemented by the ophthalmic device <NUM>. The first function is a function (hereafter referred to as the SLO imaging system function) in which the ophthalmic device <NUM> is operated as a scanning laser ophthalmoscope (hereafter referred to as a SLO) to perform SLO imaging. The second function is a function (hereafter referred to as the OCT imaging system function) in which the ophthalmic device <NUM> operates in optical coherence tomography (hereafter OCT) to perform OCT imaging. Note that for ease of explanation the function of performing imaging by SLO will be referred to as the "SLO imaging system function". Moreover, for ease of explanation the function of performing imaging by OCT will be referred to as the "OCT imaging system function".

The SLO imaging system function is implemented by the control device <NUM>, the SLO unit <NUM>, and the wide angled optical system <NUM> in the configuration of the ophthalmic device <NUM>. The SLO unit <NUM> includes a light source 18A, a detection element 18B, a dichroic mirror 18C and the like, and is configured to perform imaging of the fundus of the subject eye <NUM> Namely, the fundus (for example an imageable region 12A) of the subject eye <NUM> is imaged as an imaging subject by operating the ophthalmic device <NUM> in the SLO imaging system function. Specifically, light from the SLO unit <NUM> (referred to hereafter as SLO light) is passed through the pupil of the subject eye <NUM> and onto the imageable region 12A by the wide angled optical system <NUM>, while being scanned in the X direction (horizontal direction) by a first optical scanner <NUM> and being scanned in the Y direction (vertical direction) by a third optical scanner <NUM>. A fundus image (SLO image (an UWFSLO fundus image, described later)) configured by this reflected light is acquired by the SLO unit <NUM>. Note that the SLO imaging system function is a known function, and so detailed description thereof will be omitted. The imageable region 12A is within a range of approximately <NUM> degrees when converted into an internal illumination angle from the eyeball center O.

The OCT imaging system function is implemented by the control device <NUM>, the OCT unit <NUM>, and the wide angled optical system <NUM>. The OCT unit <NUM> includes a reference optical system 21E including a light source 20A, a sensor 20B, a fiber coupler 20C, and a polarized light adjuster 21D, and the like, and images plural tomographic regions in the fundus layer thickness direction. Namely, the ophthalmic device <NUM> images tomographic regions, which are regions in the fundus layer thickness direction (for example, the imageable region 12A), by being operated in the OCT imaging system function. Specifically, the light from the light source 20A of the OCT unit <NUM> (hereafter referred to as signal light (LS)) is branched by the fiber coupler 20C. One signal light therefrom is passed through the pupil of the subject eye <NUM> and onto the imageable region 12A by the wide angled optical system <NUM>, while being scanned in the X direction (horizontal direction) by the second optical scanner <NUM> and being scanned in the Y direction (horizontal direction) by the third optical scanner <NUM>. The one signal is reflected at the tomographic region, and the reflected light proceeds through the fiber coupler 20C and along a path incident to the sensor 20B.

The optical path length of the signal light (LS) is determined by the distance from the light source 20A to the tomographic region, and by the distance from the tomographic region to the spectroscope 20B.

Note that in the signal light, the reflected light that has been reflected by the tomographic region and is incident to the spectroscope 20B is in particular called return light.

Moreover the other signal light branched by the fiber coupler 20C has a light path length adjusted by the polarized light adjuster 21D, and proceeds along an optical path incident to the sensor 20B.

Note that the other signal light, namely the signal light proceeding from the light source 20A, through the fiber coupler 20C and the polarized light adjuster 21D, to the sensor 20B is referred to as reference light (LR).

The return light and the reference light interfere at the sensor 20B to form incident interference light. The sensor 20B detects each of the spectral components of the interference light. The control device <NUM> uses the detection results of the sensor 20B to acquire a tomographic image (hereafter referred to as an "OCT image") illustrating a tomographic region.

The acquired SLO image and OCT image are transmitted, together with the patient ID, through the communication interface (I/F) <NUM> to the management server <NUM> over the network <NUM>.

Next, description follows regarding a configuration of the wide angled optical system <NUM> included in the ophthalmic device <NUM>, with reference to <FIG>. As illustrated in <FIG>, a common optical system <NUM> includes, in addition to the third optical scanner <NUM>, a slit mirror <NUM> and an elliptical mirror <NUM>. Note that side view end faces are illustrated of a dichroic mirror <NUM>, the slit mirror <NUM>, and the elliptical mirror <NUM>. Note that a configuration may also be adopted in which plural lens groups are employed instead of the common optical system <NUM>, the slit mirror <NUM>, and the elliptical mirror <NUM>.

The slit mirror <NUM> includes an elliptical shaped first reflection surface 30A. The first reflection surface 30A includes a first focal point P1 and a second focal point P2. The elliptical mirror <NUM> also includes an elliptical shaped second reflection surface 32A. The second reflection surface 32A includes a first focal point P3 and a second focal point P4.

The slit mirror <NUM>, the elliptical mirror <NUM>, and the third optical scanner <NUM> are arranged so that the first focal point P3 and the second focal point P2 lie at a common position on the third optical scanner <NUM>. Moreover, the slit mirror <NUM>, the elliptical mirror <NUM>, and the third optical scanner <NUM> are arranged so that the second focal point P4 is positioned at a central portion of the pupil of the subject eye <NUM>. Furthermore, the first optical scanner <NUM>, the second optical scanner <NUM>, and the slit mirror <NUM> arranged so that the first focal point P1 is positioned on the first optical scanner <NUM> and the second optical scanner <NUM>.

Namely, the first optical scanner <NUM>, the second optical scanner <NUM>, and the third optical scanner <NUM> are arranged at conjugate positions to the central portion of the pupil of the subject eye <NUM>.

Note that the wide angled optical system <NUM>, as well as being a wide angled optical system employing an elliptical mirror, may also be a wide angled optical system combining an optical system employing a wide angled lens in combination with plural lenses.

In the present exemplary embodiment, a field of view (FOV) of the fundus is an angle of a fundus region over a wide range from the fundus center to the fundus periphery that is observable by the wide angled optical system <NUM> illustrated in <FIG>. The size of this wide range fundus region is determined by the internal illumination angle and the external illumination angle.

The external illumination angle is an illumination angle of light from the ophthalmic device <NUM> side, namely from the exterior of the subject eye <NUM>. Namely, the external illumination angle is the angle of scanned light onto the fundus of the subject eye <NUM> heading toward a pupil center <NUM> of the subject eye <NUM> (namely, a center point of the pupil as viewed face-on (see also <FIG>)). The external illumination angle is equivalent to the angle of light reflected from the fundus so as to head out from the pupil center <NUM> and be emitted from the subject eye <NUM> toward the ophthalmic device <NUM>.

The internal illumination angle is an illumination angle of light effectively imaged when the scanning light is illuminated onto the fundus of the subject eye <NUM>, with respect to the eyeball center O of the subject eye <NUM> as a reference position. Although an external illumination angle A and an internal illumination angle B are in a correspondence relationship, since in the following description a description of an ophthalmic imaging device is given, the external illumination angle is employed as an illumination angle corresponding to the field of view angle of the fundus.

The ophthalmic device <NUM> images within the imageable region 12A (see <FIG>), which is a fundus region of the subject eye <NUM>. The imageable region 12A is the maximum scannable region with the scanning light using the wide angled optical system <NUM>, and the external illumination angle A is approximately <NUM> degrees (corresponding to an internal illumination angle of approximately <NUM> degrees). The SLO image obtained by imaging the imageable region 12A is referred to as an UWFSLO image. Note that UWF is an abbreviation for Ultra Widefield.

Next, description follows regarding a configuration of an electrical system of the image viewer <NUM>, with reference to <FIG>. As illustrated in <FIG>, the image viewer <NUM> is equipped with a computer main unit <NUM>. The computer main unit <NUM> includes a CPU <NUM>, RAM <NUM>, ROM <NUM>, and an input/output (I/O) port <NUM>. A storage device <NUM>, a display <NUM>, a mouse <NUM>, a keyboard <NUM>, and a communication interface (I/F) <NUM> are connected to the input/output (I/O) port <NUM>. The storage device <NUM> is, for example, configured by non-volatile memory. The input/output (I/O) port <NUM> is connected to the network <NUM> through the communication interface (I/F) <NUM>. This thereby enables the image viewer <NUM> to communicate with the ophthalmic device <NUM> and the management server <NUM>.

The display <NUM> of the image viewer <NUM> is an example of a "display section" of the technology disclosed herein.

The configuration of the electrical system of the management server <NUM> is, similarly to the configuration of the electrical system of the image viewer <NUM>, equipped with a computer main unit <NUM>, including the CPU <NUM>, the RAM <NUM>, the ROM <NUM>, and the input/output (I/O) port <NUM>, and with the storage device <NUM>, the display <NUM>, the mouse <NUM>, and the keyboard <NUM> that are connected to the input/output (I/O) port <NUM>.

The CPU <NUM> of the management server <NUM> is an example of an "image processing device" of the technology disclosed herein.

Fundus image data for test subjects and an image processing program 154P are stored in the storage device <NUM> of the management server <NUM>.

Although a description follows of a case in which the image processing program 154P is stored in the storage device <NUM>, the technology disclosed herein is not limited thereto, and the image processing program 154P may be stored on the ROM <NUM>.

The image processing program 154P is an example of an image processing program according to technology disclosed herein.

Elements corresponding to the display <NUM>, the mouse <NUM>, and the keyboard <NUM> of the image viewer <NUM> may be omitted for a configuration of the electrical system of the management server <NUM>.

Next, description follows regarding various functions implemented by the CPU <NUM> of the management server <NUM> executing the image processing program 154P, with reference to <FIG>. The image processing program 154P is equipped with a reception function, an acquisition function, an enhancement image processing function, a prediction processing function, a generation function, and a transmission function. The CPU <NUM> functions as a reception section <NUM>, an acquisition section <NUM>, an enhancement image processing section <NUM>, a prediction processing section <NUM>, a generation section <NUM>, and a transmission section <NUM> as illustrated in <FIG> by the CPU <NUM> executing the multi-function image processing program 154P.

The enhancement image processing section <NUM>, the prediction processing section <NUM>, and the generation section <NUM> may be configured by an integrated image processing chip (an IC, a hardware configuration such as circuit, or the like).

Next, description follows regarding overall operation of the ophthalmic system <NUM> illustrated in <FIG>.

First, the ophthalmic system <NUM> collects basic information about the subject eye <NUM> of a patient in order to perform a diagnosis on the subject eye <NUM> of the patient (see <FIG>). More specifically, the eye axial length etc. is measured using a non-illustrated eye axial length measurement instrument, or the like. Furthermore, on instruction by a doctor, the patient goes to a room where the field of view measurement instrument <NUM> is installed. The field of view measurement instrument <NUM> measures a visible range (field of view map) by looking at responses of the patient when a light stimulus is imparted to the retina. The field of view measurement instrument <NUM> transmits the measured field of view map to the management server <NUM> together with the patient ID. The management server <NUM> stores the field of view map in the storage device <NUM> (see <FIG>) in association with the patient ID. On instruction by the doctor, the patient then goes to a room where the ophthalmic device <NUM> is installed. The ophthalmic device <NUM> images the subject eye <NUM> of the patient to acquire the fundus image (SLO image (UWFSLO fundus image) and OCT image). The ophthalmic device <NUM> transmits the acquired fundus image to the management server <NUM> together with the patient ID. The management server <NUM> stores the fundus image in the storage device <NUM> associated with the patient ID.

When examining the subject eye <NUM> of the patient, the doctor employs the fundus image of the subject eye <NUM>, and sometimes employs information about whether or not there is a non perfusion area on the fundus. First, the patient ID is input to the image viewer <NUM>, then the image viewer <NUM> acquires the fundus image of the patient from the management server <NUM>, and the acquired fundus image is displayed on the display <NUM>. An instruction to generate a perfusion area candidate image for the fundus image being displayed is then transmitted from the image viewer <NUM> to the management server <NUM>.

The management server <NUM> transmits image data for the generated non perfusion area candidate image based on the specified fundus image to the image viewer <NUM>.

The non perfusion area candidate image is a fundus image with any predicted non perfusion areas displayed so as to be superimposed thereon.

Although described in detail later, the management server <NUM> reads the specified fundus image from the storage device <NUM>, predicts any non perfusion areas in the read fundus image, and generates a final image of candidate areas for non perfusion areas (either as a non perfusion area candidate image or a candidate group image), and transmits the image data for the generated final image to the image viewer <NUM>.

The image viewer <NUM> receives the image data for the non perfusion area candidate image using the communication interface I/F <NUM>, and displays the non perfusion area candidate image on the display <NUM>.

The fundus image is an UWFSLO fundus image imaged with the ophthalmic device <NUM>, and enables the prediction of whether or not there are any NPAs, serving as targets, present across a wide range of the fundus. Obviously there is no limitation to using an UWFSLO fundus image, and the NPAs may be predicted using fundus images of the same patient (previously imaged UWFSLO fundus images, fundus images imaged with another instrument, or the like).

The doctor diagnoses the subject eye <NUM> of the patient based on the non perfusion area candidate image displayed on the display <NUM>. If there is no problem with the subject eye <NUM> then the consultation is ended. However, in cases in which there is a problem with the subject eye <NUM> and the doctor has diagnosed a need for an OCT image, then a tomographic image of a fundus layer needs to be acquired using the ophthalmic device <NUM>, and so an instruction to perform imaging by OCT may be output through the image viewer <NUM> to the ophthalmic device <NUM>.

When instructed to perform imaging by OCT, the ophthalmic device <NUM> transmits an OCT image acquired with the ophthalmic device <NUM> to the management server <NUM> together with the patient ID. The management server <NUM> stores the OCT image in the storage device <NUM> associated with the patient ID, and also transmits the OCT image to the image viewer <NUM>.

Although described in detail later, briefly the image viewer <NUM> performs screen display in various display modes on the screen <NUM> of the display <NUM> of the image viewer <NUM>, as illustrated in <FIG>.

Next, description follows regarding the image processing program executed by the CPU <NUM> of the management server <NUM>, with reference to <FIG>. The image processing illustrated in <FIG> is implemented by the CPU <NUM> of the management server <NUM> executing the image processing program 154P.

The image processing illustrated in <FIG> is an example of an "image processing method" of technology disclosed herein.

The image processing program 154P is started when the reception section <NUM> (see also <FIG>) has received command data from the image viewer <NUM>. The command data is issued from the image viewer <NUM>, and relates to a command to generate a non perfusion area (NPA) candidate image from the specified UWFSLO image, and to transmit image data for the non perfusion area (NPA) candidate image to the image viewer <NUM>.

When the image processing program 154P has started, at step <NUM> the acquisition section <NUM> acquires the specified UWFSLO image from out of the plural UWFSLO images stored in the storage device <NUM> in association with the plural respective patient IDs. The UWFSLO images, as illustrated in <FIG>, include structures such as retina vascular structures, the optic nerve head, and the like.

Next at step <NUM>, the enhancement image processing section <NUM> performs enhancement image processing on the acquired UWFSLO image to enhance vascular portions thereof. This processing is processing performed to make the blood vessels including capillary blood vessels more prominent in order to predict non perfusion areas (NPAs) with good precision.

The enhancement image processing may employ various methods, such as enhancement processing in which an image histogram is employed, as in histogram averaging and contrast limited adaptive histogram equalization (CLAHE), or alternatively such as contrast conversion processing based on gradation conversion, frequency enhancement processing for a particular frequency band such as processing by unsharp masking, deconvolution processing such processing by Weiner filter, morphology processing to enhance the shape of vascular portions, or the like. Preferably histogram averaging or adaptive histogram equalization is employed therefor. The blood vessels are enhanced by the enhancement image processing, and the capillary blood vessels are also enhanced as illustrated in <FIG>.

This enables the non perfusion areas (NPAs) to be predicted with good precision from the UWFSLO image in which the blood vessels have been enhanced. Thus in the technology disclosed herein, at the next steps <NUM> to <NUM>, the prediction processing section <NUM> predicts plural non perfusion areas (NPAs) in the UWFSLO image in which the blood vessels have been enhanced.

Specifically, at step <NUM>, the prediction processing section <NUM> selects primary candidates for non perfusion areas (NPAs). More specifically, the prediction processing section <NUM> extracts plural pixels of a first darkness or darker from the UWFSLO image in which the blood vessels have been enhanced (see <FIG>), and selects as primary candidates for non perfusion areas (NPAs) a single or plural areas having a surface area of a prescribed surface area or greater of contiguous pixels of the first darkness or darker.

The pixels of the first darkness or darker referred to here are pixels for which the pixel value of the respective pixel is a first specific value or lower.

Note that as the pixel values, for example, brightness values expressing lightness may be employed therefor, however, values expressing at least one out of saturation or hue may also be employed therefor instead of brightness values, or as well as the brightness values. A primary candidate is an example of a "first non perfusion area candidate" of the technology disclosed herein.

<FIG> is a schematic image illustrating an enlarged display of a portion of an UWFSLO image as a simplified representation of results of processing of step <NUM>. There are six primary candidates <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> for the non perfusion areas (NPAs) shown together with blood vessels <NUM> in <FIG>.

After step <NUM> the image processing proceeds to step <NUM> and step <NUM>.

At step <NUM>, from the single or plural primary candidates of the non perfusion areas (NPAs), the prediction processing section <NUM> selects only dark candidates from the primary candidates based on a respective average value of the pixel values in each of the candidate areas. Specifically, the prediction processing section <NUM> calculates an average value of the pixel values in each of the areas of the single or plural primary candidates of non perfusion areas (NPAs), and selects as a dark area a single or plural candidate whose calculated average value is smaller than a second specific value. The second specific value is a value of a specific value smaller than the first specific value. Namely, only candidates that are dark areas having a darkness that is a second darkness darker than the first darkness, or darker (i.e. candidates having a specific average pixel value or less), are extracted from the primary candidates of the first darkness, to yield first secondary candidates.

The first secondary candidates are examples of "second non perfusion area candidates" of technology disclosed herein.

When the processing of step <NUM> is executed after the processing of step <NUM>, for example as illustrated in <FIG>, the primary candidate <NUM> is excluded from the six primary candidates <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> (see <FIG>) extracted at step <NUM>. This thereby narrows down the candidates to the first secondary candidates 402A, 404A, 406A, 408A, 410A.

At step <NUM>, the prediction processing section <NUM> narrows down the plural primary candidates for non perfusion areas (NPAs) to only areas having blood vessels running alongside. More specifically, first the prediction processing section <NUM> (<NUM>) extracts blood vessels. The blood vessels are extracted based on the pixel values using a method such as morphology processing or binarization or the like. Note that the areas extracted thereby are referred to as vascular areas. Then the prediction processing section <NUM> (<NUM>) uses a method such as distance conversion to compute a distance between such vascular areas and the peripheral edges of a single or plural primary candidates for non perfusion areas (NPAs) or for, or of each area group of candidate groups for non perfusion areas (NPAs), and selects areas in which this computed distance is within a fixed range.

The fixed range referred to here is a first range that is larger than a first specific distance, but smaller than a second specific distance larger than the first specific distance (namely, cases in which the areas have blood vessels running alongside).

Thus at step <NUM>, the prediction processing section <NUM> extracts from the primary candidates areas for which a distance to blood vessels is a first distance or lower, as second secondary candidates areas. The candidates <NUM>, <NUM> are, for example, accordingly excluded as they are not areas having blood vessels running alongside, as illustrated in <FIG>. This thereby narrows down the candidates to the second secondary candidates 406B, 408B, 410B, 412B as candidates having blood vessels running alongside.

Note that for the second secondary candidates, as illustrated in <FIG>, an area 450B that is a fixed range away from blood vessel terminal ends 400E1, 400E2, 400E3 may be employed as a second secondary candidate.

The second secondary candidates are an example of "second non perfusion area candidates" of the technology disclosed herein.

Step <NUM> and step <NUM> may be executed one after each other, or may be executed at the same time as each other. The image processing proceeds to step <NUM> after the processing of step <NUM> and step <NUM> has been completed.

At step <NUM>, the prediction processing section <NUM> performs consolidation processing to consolidate the first secondary candidates and the second secondary candidates. Specifically, areas are extracted that are members of the first secondary candidates (plural dark areas) and are also members of the second secondary candidates (plural areas having the blood vessels running alongside), and identifies these as predicted non perfusion areas.

In the example of <FIG>, the six primary candidates <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> are obtained as non perfusion areas (NPAs) at step <NUM>. However, as illustrated in <FIG>, the candidate <NUM> is excluded at step <NUM>, and the candidates <NUM>, <NUM> are excluded at step <NUM>. Thus, as illustrated in <FIG>, the candidates <NUM>, <NUM>, <NUM> are identified at step <NUM> as predicted non perfusion areas 406NPA, 408NPA, 410NPA.

Next at step <NUM>, the generation section <NUM> applies a colored border to the periphery of the predicted non perfusion areas in the UWFSLO image to generate the non perfusion area candidate image. The non perfusion area candidate image refers to a fundus image in which predicted non perfusion areas are displayed superimposed on the UWFSLO image so as to enable positions of the predicted non perfusion areas on the UWFSLO image to be easily ascertained. The predicted non perfusion areas are displayed surrounded by borders in the image in order to enable the positions of the predicted non perfusion areas on the UWFSLO image to be easily ascertained. As an example of such display, part of the UWFSLO image is, as illustrated in <FIG>, displayed with colored borders applied to the peripheries of the predicted non perfusion areas 406NPA, 408NPA, 410NPA.

Furthermore, the non perfusion area candidate image may be generated by the generation section <NUM> applying light color shading (a color of a first density darker than the density of the UWFSLO image) to the predicted non perfusion areas 406NPA, 408NPA, 410NPA, as illustrated in <FIG>. Note that at step <NUM>, the generation section <NUM> may shade the predicted non perfusion areas 406NPA, 408NPA, 410NPA with a similar color to real color, or a different color to real color, as illustrated in <FIG>. Note that broken lines may be employed for the borders at the periphery of the predicted non perfusion areas 406NPA, 408NPA, 410NPA.

The non perfusion area candidate image is an example of information employed by a doctor to diagnose or determine the progress of diabetic retinopathy, retinal vein occlusion, or the like.

At the next step <NUM>, the transmission section <NUM> transmits the image data for the non perfusion area candidate image generated at step <NUM> to the image viewer <NUM>.

On receipt of data for the non perfusion area candidate image, the image viewer <NUM> displays the non perfusion area candidate image on the display <NUM>.

Detailed description now follows regarding a method to display the non perfusion area candidate image, with reference to the screen <NUM> on the display <NUM> of the image viewer <NUM> as illustrated in <FIG> and <FIG>.

The display methods illustrated in <FIG> and <FIG> are examples of an "image display method" of technology disclosed herein.

<FIG> illustrates a display content of an NPA analysis mode of the display <NUM> of the image viewer <NUM> displayed on the screen <NUM>. The screen <NUM> includes a patient information display field <NUM>, a UWFSLO image display field <NUM>, an OCT image display field <NUM>, a non perfusion area candidate image display field <NUM>, and two enlarged image display fields <NUM>, <NUM>. The screen <NUM> also includes a menu button <NUM>, an NPA tracking observation button <NUM>, an NPA analysis button <NUM>, an NPA laser coagulation button <NUM>, an NPA field of view analysis button <NUM>, and an OCT analysis button <NUM>. Furthermore, the screen <NUM> also includes a tool button display field <NUM>.

<FIG> illustrates a display screen for an NPA analysis mode and so, from out of the menu button <NUM>, the NPA tracking observation button <NUM>, the NPA analysis button <NUM>, the NPA laser coagulation button <NUM>, the NPA field of view analysis button <NUM>, and the OCT analysis button <NUM>, the NPA analysis button <NUM> is accordingly displayed in a first display state to indicate being active, and the other buttons are displayed in a second display state to indicate being inactive. The first display state and the second display state can be various display states, such as being displayed in different colors (e.g. the first display state green and the second display state red), being displayed in 3D (e.g. the first display state displayed in relief from the screen using shadow-display or the like, and the second display state displayed without shadows).

The patient information display field <NUM> includes a patient ID display field 502A, a patient name display field 502B, a gender display field 502C, an age display field 502D, and an attendance history display field 502E. The image viewer <NUM> acquires the patient ID, patient name, gender, age, and attendance history as stored in the management server <NUM>. The image viewer <NUM> then displays the acquired patient ID, patient name, gender, age, and attendance history in the patient ID display field 502A, the patient name display field 502B, the gender display field 502C, the age display field 502D, and the attendance history display field 502E, respectively.

The image viewer <NUM> displays a UWFSLO image of the patient in the UWFSLO image display field <NUM> of the screen <NUM>, and displays the non perfusion area candidate image in the non perfusion area candidate image display field <NUM> thereof.

In order to facilitate observation/diagnosis performed by a doctor, the non perfusion area candidate image is displayed in the non perfusion area candidate image display field <NUM> at a different magnification to the UWFSLO image displayed in the UWFSLO image display field <NUM>. For example, the non perfusion area candidate image is displayed at a size of a specific magnification enlargement in comparison to the size of the UWFSLO image displayed in the UWFSLO image display field <NUM>. The non perfusion area candidate image may also be displayed at a size shrunk by a specific magnification in comparison to the size of the UWFSLO image displayed in the UWFSLO image display field <NUM>. Note that each of these specific magnifications may also be configured so as to be variable.

Situations arise in which the doctor wishes to see a portion in the non perfusion area candidate image in an enlarged view. Thus as illustrated in <FIG>, for example, the doctor operates an input means such as a mouse to specify areas 508A, 508B desired to be viewed enlarged as areas within the non perfusion area candidate image display field <NUM> that the doctor wishes to display enlarged. When the areas 508A, 508B are specified, the image viewer <NUM> magnifies and displays images 508AA, 508BB that are portions of the specified areas 508A, 508B in the non perfusion area candidate image in the enlarged image display fields <NUM>, <NUM>.

In cases in which the doctor has viewed the non perfusion area candidate image and determined that an OCT image needs to be acquired, the doctor specifies a specified area 508C for acquiring an OCT image in the non perfusion area candidate image being displayed in the non perfusion area candidate image display field <NUM>. The specified area may be a straight line or may be a rectangle. In such cases a specified area 504A may also be displayed on the UWFSLO image being displayed in the UWFSLO image display field <NUM> at a position corresponding to the 508C specified on the non perfusion area candidate image.

To specify the area 508C for acquiring an OCT image, an instruction to acquire the OCT image of the specified area 508C is output to an operator of the ophthalmic device <NUM> through the management server <NUM>. A patient then revisits the room in which the ophthalmic device <NUM> is disposed. The ophthalmic device <NUM> acquires the OCT image in response to the instruction and transmits the OCT image to the management server <NUM>. The OCT image data is then transmitted from the management server <NUM> to the image viewer <NUM>, and the image viewer <NUM> displays the OCT image in the OCT image display field <NUM>. The doctor accordingly checks the OCT image at the location where a problem in the subject eye <NUM> has been determined, and examines the subject eye <NUM>.

Note that buttons to call various input aid tools are displayed in the tool button display field <NUM>, such as an icon for displaying a line, an icon for displaying a rectangle, a text input icon to input text on the image, and a pen input icon or the like to display a handwritten sketch superimposed on the image.

The attendance history display field 502E is provided in the patient information display field <NUM>. In the example illustrated in <FIG>, attendance histories are being displayed for June <NUM><NUM>, September <NUM><NUM>, and December <NUM><NUM>. A doctor who has determined that they wish to view the attendance history clicks on the NPA tracking observation button <NUM>. When the NPA tracking observation button <NUM> is clicked, the screen <NUM> of the display <NUM> of the image viewer <NUM> is changed from the display state of the NPA analysis mode of <FIG> to the display state of an NPA tracking observation mode of <FIG>.

As illustrated in <FIG>, an NPA tracking observation mode screen <NUM> includes a UWFSLO tracking image display field 524A, and a non perfusion area candidate image display field 524D in addition to the patient information display field <NUM> and the tool button display field <NUM>. Note that patient information display field <NUM>, the tool button display field <NUM>, and the non perfusion area candidate image display field 524D are display fields similar to those of <FIG>. <FIG> is a NPA tracking observation mode display screen, and so from out of the menu button <NUM>, the NPA tracking observation button <NUM>, the NPA analysis button <NUM>, the NPA laser coagulation button <NUM>, the NPA field of view analysis button <NUM>, and the OCT analysis button <NUM>, the NPA tracking observation button <NUM> is accordingly displayed in the first display state to indicate being active, and the other buttons are displayed in a second display state to indicate being inactive.

The UWFSLO tracking image display field 524A includes plural (three in the above example) tracking image display fields 524A1 to 524A3 for past UWFSLO images (tracking images), and a slider bar 524B including a slider 524C.

In cases in which the patient has attended four or more times, then a configuration might be considered in which all of four or more UWFSLO images are displayed at the same time. However, to display all the images the display field for each of the images would need to be made small, making it less visible. Thus the three tracking image display fields 524A1 to 524A3 are provided as described above. The position of the slider 524C is such that the right end corresponds to the current time, further to the left thereof corresponding to past times. The image viewer <NUM> is configured to displays three UWFSLO images that correspond to the position of the slider 524C in the tracking image display fields 524A1 to 524A3.

The doctor uses an input means such as a mouse to select an image desired for NPA analysis from out of the three UWFSLO images being displayed in the UWFSLO image history display field 524A. For example, when the tracking image 524A3 has been selected, the color of the border of 524A3 is displayed in a color different to the color of the borders of 524A1 and 524A2. The thickness of line for the border may also be changed instead of the color of the border, or both the color and the thickness of the border may be changed.

Next, the doctor presses (clicks on) the NPA prediction button 525E using a mouse or the like, and the image viewer <NUM> issues a command to the management server <NUM> so as to generate a non perfusion area candidate image for the selected tracking image 524A3. The management server <NUM> reads the image of the tracking image 524A3 and performs the image processing explained with reference to <FIG>. The obtained non perfusion area candidate image corresponding to the tracking image 524A3 is saved in the storage device <NUM> of the management server, and image data for the image processed non perfusion area candidate image is transmitted to the image viewer <NUM>.

The image viewer <NUM> displays the non perfusion area candidate image of the image processed tracking image 524A3 in the non perfusion area candidate image display field 524D of the screen <NUM> based on the image data received for the non perfusion area candidate image of the image processed tracking image 524A3. The image viewer <NUM> may be configured to display the display border of the selected tracking image 524A3 and the display border of the non perfusion area candidate image display field 524D in the same color and line style or thickness.

Note that any of the three non perfusion area candidate images resulting from image processing the three UWFSLO images (tracking images) may be displayed in the non perfusion area candidate image display field 524D.

Moreover, if the doctor presses the NPA prediction button 525E without selecting a tracking image from out of the three UWFSLO tracking image display fields 524A, the image viewer <NUM> outputs a command to the management server <NUM> to generate non perfusion area candidate images for all of the UWFSLO images associated with the patient ID (namely, the 524A1, 524A2, 524A3 stored in an image folder for the patient ID). When there are, from out of the UWFSLO images stored in the image folder of the patient ID, UWFSLO images for which non perfusion area candidate images have already been generated, a command may be issued so as to generate non perfusion area candidate images for any UWFSLO images for which no non perfusion area candidate image has yet been generated.

The management server <NUM> sequentially reads the images of the tracking images 524A1, 524A2, 524A3, and performs the image processing explained with reference to <FIG> sequentially thereon. Then the obtained tracking image 524A3 is saved in the management server storage device <NUM>, and the image data of the image processed non perfusion area candidate image is transmitted to the image viewer <NUM>. The respective non perfusion area candidate images obtained for the tracking images 524A1, 524A2, 524A3 are each saved in the storage device <NUM> of the management server, and image data for the three new image processed non perfusion area candidate images are transmitted to the image viewer <NUM>.

On receipt of the image data for the image processed three new non perfusion area candidate images, the image viewer <NUM> displays the three received non perfusion area candidate images in the UWFSLO image tracking image display fields of the screen <NUM> based on the attendance history. Furthermore, when one non perfusion area candidate image has been selected from out of the three non perfusion area candidate images being displayed in the tracking image display fields, the selected non perfusion area candidate image is enlarged and displayed in the non perfusion area candidate image display field 524D. For example, in cases in which the non perfusion area candidate image 524A3 having the most recent attendance history has been selected, the display border of the non perfusion area candidate image 524A3 may be displayed in the same color and line style or thickness as the display border of the non perfusion area candidate image display field 524D.

Moreover, non perfusion area candidate images with different attendance dates may be compared, and image processing performed so as to display a newly appearing predicted non perfusion area in a changed color, and to display plural non perfusion area candidate images in the tracking observation image display fields. For example, the UWFSLO image imaged on June <NUM><NUM> displayed in 524A1 of <FIG> is processed to display the obtained predicted non perfusion areas in blue. Processing may be performed on the UWFSLO image imaged on September <NUM><NUM> displayed in 524A2 of <FIG> that enables, from out of the predicted non perfusion areas obtained thereby, predicted non perfusion areas that are predicted to be the same as the predicted non perfusion areas of June <NUM><NUM> to be displayed in blue, and predicted non perfusion areas appearing for the first time under the image processing of September <NUM><NUM> to be displayed in red. Such image processing enables valuable information for diagnosing the rate of progression of symptoms in a patient to be provided to a doctor, enabling support to be given to diagnosis.

Next description will be given regarding the function of the menu button <NUM>, the NPA laser coagulation button <NUM>, the NPA field of view analysis button <NUM>, and the OCT analysis button <NUM> displayed on the screen <NUM> of <FIG> and on the screen <NUM> of <FIG>.

The menu button <NUM> is a button to return to the menu screen of an ophthalmic electronic medical record. Tools for selecting initial settings and user recording, fundus observation mode, anterior segment diagnostic mode, and the like are displayed on the menu screen. Transition is made to the screen <NUM> of <FIG> when the "fundus observation mode" is selected on the menu screen.

The OCT analysis button <NUM> is a button for transitioning to a screen to perform image analysis using retinal tomographic images, a thickness map of the optic nerve layer, and the like obtained with the ophthalmic device <NUM>, B scan data obtained by imaging the retina and OCT volume data.

The NPA laser coagulation button <NUM> is a button for transitioning to a screen to perform analysis related to treatment using the laser photocoagulator <NUM>. In cases in which the result of diagnosis is that the doctor has determined there to be a pathological lesion on fundus of the patient and that there is a need to suppress pathological progression, sometimes treatment is performed to cause coagulation by illuminating a laser onto the fundus using the laser photocoagulator <NUM>. The NPA laser coagulation button <NUM> is clicked in such cases. When the NPA laser coagulation button <NUM> is clicked, transition is made to a screen equipped with a simulation function so as to use the non perfusion area candidate image to present an appropriate laser illumination position in order to determine a position to illuminate the laser onto.

The NPA field of view analysis button is a button for transitioning to a screen to perform analysis related to the field of view using the field of view map (a map representing a visible range) obtained using the field of view measurement instrument <NUM> and the non perfusion area candidate image. When the NPA field of view analysis button <NUM> is clicked, transition is made to an NPA field of view analysis mode to combine and display the field of view map from the field of view measurement instrument <NUM> combined with the non perfusion area candidate image. Performing analysis processing in which the field of view map is combined with an image including enhanced non perfusion areas enables correlation between the position of the NPAs on the retina and the field of view to be investigated.

The present exemplary embodiment as described above enables non perfusion areas to be predicted by performing image processing on fundus images.

Moreover, due to plural non perfusion areas being predicted from a fundus image that has been subjected to enhancement image processing to enhance the vascular portions, the present exemplary embodiment also enables plural non perfusion areas to be predicted with good precision.

Furthermore, employing UWFSLO images to predict the non perfusion areas enables early stage discovery of NPAs in areas around the retina periphery, and enables not only diabetic retinopathy, but also retinal vein occlusion, and retinal artery occlusion to be discovered at an early stage.

Moreover, the present exemplary embodiment is also useful in specifying laser illumination positions for laser coagulation surgery using the laser photocoagulator <NUM>. Furthermore, the present exemplary embodiment enables a correlation between NPAs and field of view to be investigated by performing analysis processing of the field-of-view examination data of the field of view measurement instrument <NUM>, and the field-of-view examination data combined with an image in which the non perfusion areas have been enhanced.

Furthermore, performing successive observations of the non perfusion areas using tracking observations enables information to be provided to the doctor to support checking of the effects of treatment and the progression of symptoms.

In the technology disclosed herein, either step <NUM> or step <NUM> may be executed first, with a group of non perfusion area candidates predicted thereby employed as secondary candidates, and then at the other step continuing therefrom, these secondary candidates may be narrowed down to tertiary candidates. For example, after executing the processing of step <NUM>, the candidates resulting therefrom may be further narrowed down at step <NUM>, or in reverse, after executing the processing of step <NUM>, the candidates resulting therefrom may be further narrowed down at step <NUM>. This enables step <NUM> to be omitted.

Moreover, the predicted non perfusion areas may be areas narrowed down by step <NUM> and step <NUM>. The predicted non perfusion areas may also be areas narrowed down by step <NUM> and step <NUM>.

In technology disclosed herein, the above described contents of the processing of step <NUM> may be modified, and as a consolidated step <NUM>, the prediction processing section <NUM> may select plural areas that are at least dark areas or areas having blood vessels running alongside as a group of non perfusion area candidates.

In the above exemplary embodiment, at step <NUM>, in the UWFSLO image a colored border is applied to the periphery of plural predicted non perfusion areas and light color shading is applied thereto. However, technology disclosed herein is not limited thereby, and an image may be generated that omits the vascular portions and indicates only plural non perfusion areas, or an image may be generated that omits the vascular portions and only includes plural non perfusion areas enhanced in the manner described above.

In the technology disclosed herein, information for display may also include left-right eye information, a graph of change over time in the number of non perfusion areas appearing, a graph of change over time in the average surface area of non perfusion areas, and a graph of change over time in the total surface area of non perfusion areas. Furthermore, during laser coagulation surgery, the non perfusion area candidate image may be displayed superimposed on the UWFSLO fundus image.

Although in the above exemplary embodiment the image processing program of <FIG> is executed by the management server <NUM>, the technology disclosed herein is not limited thereto. For example, in a first case the image processing program of <FIG> may be stored in the storage device <NUM> of the ophthalmic device <NUM>, and the ophthalmic device <NUM> execute the image processing program of <FIG> every time an UWFSLO image is acquired. In a second case the image viewer <NUM> may execute steps <NUM> to <NUM>. Note that at step <NUM> in the second case, the image data of the specified fundus image is acquired from the management server <NUM>, and instead of the content of step <NUM>, an image in which plural non perfusion areas have been enhanced is displayed on the display <NUM>. In the first case, the ophthalmic device <NUM> is an example of an image processing device of technology disclosed herein. In the second case, the image viewer <NUM> is an example of an image processing device of technology disclosed herein.

Although in the above exemplary embodiment the image viewer <NUM> displays on the display <NUM> the image in which plural non perfusion areas are enhanced, the technology disclosed herein is not limited thereto. For example, an UWFSLO image of <FIG>, an image in which the blood vessels have been enhanced in the UWFSLO image of <FIG>, or an image of plural enhanced non perfusion areas as in <FIG> may be displayed. In such cases, the images of <FIG>, <FIG>, and <FIG> to <FIG> may be displayed in a display array, may be displayed so as to be switched sequentially every time clicked, or may be selectively displayed.

In the technology disclosed herein the primary candidates for non perfusion area acquired at step <NUM>, and the images of candidates removed from the candidates at steps <NUM>, <NUM> may be displayed in a display array, may be displayed so as to be switched sequentially every time clicked, or may be selectively displayed.

Although in the above exemplary embodiment an image of the entire fundus is acquired as the UWFSLO image, technology disclosed herein is not limited thereto. An UWFSLO image of only a specific area including an already appearing pathological lesion may be acquired as a tracking observation.

Although an example has been described in which an UWFSLO image is subjected to image processing, obviously a fundus image from a fundus camera may be appropriately employed therefor, and fundus images imaged by various ophthalmic devices, such as an SLO ophthalmic device or fundus camera with relatively narrow angle (for example, an internal illumination angle of <NUM> degrees of less) may be appropriately employed therefor.

Although in the above exemplary embodiment an example has been described of the ophthalmic system <NUM> equipped with the ophthalmic device <NUM>, the field of view measurement instrument <NUM>, the laser photocoagulator <NUM>, the management server <NUM>, and the image viewer <NUM>, the technology disclosed herein is not limited thereto. For example, as a first example the ophthalmic device <NUM> may further include at least one function from either the field of view measurement instrument <NUM> or the laser photocoagulator <NUM>. Moreover, as a second example, the ophthalmic device <NUM> of the above exemplary embodiment may be equipped with the field of view measurement instrument <NUM> and the laser photocoagulator <NUM>. In both the first example and the second example, the ophthalmic device <NUM> may further include at least one function of the management server <NUM> or the image viewer <NUM>. Adopting such an approach enables at least one device from out of the management server <NUM> or the image viewer <NUM> to be omitted, i.e. the device corresponding to the function the ophthalmic device <NUM> is equipped with to be omitted.

Moreover, the management server <NUM> may be omitted, and the image viewer <NUM> configured so as to execute the function of the management server <NUM>.

Note that the image processing described in the above exemplary embodiment is merely an example thereof. Thus obviously unnecessary steps may be omitted, new steps may be added, and the sequence of processing may be changed within a scope not departing from the spirit of technology disclosed herein.

Claim 1:
An image processing method comprising:
performing first image processing on a fundus image of a subject eye to extract a first non perfusion area candidate;
characterised by:
performing second image processing on the fundus image to extract a second non perfusion area candidate; and
extracting as a predicted non perfusion area any candidate that is both the first non perfusion area candidate and the second non perfusion area candidate.