Image processing method, program, and image processing device

A first image is projective transformed into a reference image by using a projective transformation matrix on each pixel of the first image. A projective transformation matrix for transforming a tilted image n into a central image G0 is computed based on positions of combinations of twelve corresponding points indicated in a combination of the central image G0 with the tilted image n. The computed projective transformation matrix is then employed to perform a projective transformation of the tilted image n. A projective-transformed tilted image n1 is created thereby.

TECHNICAL FIELD

Technology disclosed herein relates to an image processing method, a program, and an image processing device.

RELATED ART

Japanese Patent Application Laid-Open (JP-A) No. 2009-112617 discloses technology relating to a panoramic fundus image synthesizing device and method.

SUMMARY

An image processing method of a first aspect of technology disclosed herein includes: a step of specifying respective corresponding points in a reference image obtained by imaging a fundus with a first gaze and a first image obtained by imaging the fundus with a second gaze different to the first gaze; a step of computing a projective transformation matrix for projective transformation of the first image onto the reference image based on the corresponding points; and a step of employing the projective transformation matrix to perform projective transformation of the first image.

A program of a second aspect of technology disclosed herein causes a computer to execute the image processing method of the first aspect.

An image processing device of a third aspect of technology disclosed herein includes an image processing section configured to execute: a step of specifying respective corresponding points in a reference image obtained by imaging a fundus with a first gaze and a first image obtained by imaging the fundus with a second gaze different to the first gaze; a step of computing a projective transformation matrix for projective transformation of the first image onto the reference image based on the corresponding points; and a step of employing the projective transformation matrix to perform projective transformation of the first image.

DETAILED DESCRIPTION

Detailed explanation follows regarding exemplary embodiments of the present invention, with reference to the drawings. In the following, for ease of explanation, a scanning laser ophthalmoscope is referred to as an “SLO”.

The configuration of an ophthalmic system100will now be described with reference toFIG.1. As illustrated inFIG.1, the ophthalmic system100includes an ophthalmic device110, an eye axial length measuring instrument120, a management server device (hereinafter referred to as “management server”)140, and an image display device (hereinafter referred to as “image viewer”)150. The ophthalmic device110acquires fundus images. The eye axial length measuring instrument120measures the eye axial length of a patient. The management server140stores plural fundus images and eye axial lengths obtained by imaging the fundi of plural patients using the ophthalmic device110in association with respective patient IDs. The image viewer150displays fundus images acquired by the management server140.

The image viewer150is an example of an “image processing device” of technology disclosed herein.

The ophthalmic device110, the eye axial length measuring instrument120, the management server140, and the image viewer150are connected to each other over a network130.

Note that other ophthalmic instruments (instruments for tests such as field of view measurement and intraocular pressure measurement) and a diagnostic support device that performs image analysis using artificial intelligence may be connected to the ophthalmic device110, the eye axial length measuring instrument120, the management server140and the image viewer150over the network130.

Explanation follows regarding a configuration of the ophthalmic device110, with reference toFIG.2. As illustrated inFIG.2, the ophthalmic device110includes a control unit20, a display/operation unit30, and an SLO unit40, and images the posterior eye portion (fundus) of the examined eye12. Furthermore, a non-illustrated OCT unit may be provided for acquiring OCT data of the fundus.

The control unit20includes a CPU22, memory24, a communication interface (I/F)26, and the like. The display/operation unit30is a graphical user interface that displays an image obtained by imaging and receives various instructions including an imaging instruction. The display/operation unit30also includes a display32and an input/instruction device34.

The SLO unit40includes a light source42for green light (G-light: wavelength 530 nm), a light source44for red light (R-light: wavelength 650 nm), and a light source46for infrared radiation (IR-light (near-infrared light): wavelength 800 nm). The light sources42,44,46respectively emit light as commanded by the control unit20.

The SLO unit40includes optical systems50,52,54and56that reflect or transmit light from the light sources42,44and46in order to guide the reflected light into a single optical path. The optical systems50and56are mirrors, and the optical systems52and54are beam splitters. The G-light is reflected by the optical systems50and54, the R-light is transmitted through the optical systems52and54, and the IR-light is reflected by the optical systems52and56, such that all are guided into a single optical path.

The SLO unit40includes a wide-angle optical system80for two-dimensionally scanning light from the light sources42,44,46across the posterior eye portion (fundus) of the examined eye12. The SLO unit40includes a beam splitter58that, from out of the light from the posterior eye portion (fundus) of the examined eye12, reflects the G-light and transmits light other than the G-light. The SLO unit40includes a beam splitter60that, from out of the light transmitted through the beam splitter58, reflects the R-light and transmits light other than the R-light. The SLO unit40includes a beam splitter62that, from out of the light that has passed through the beam splitter60, reflects IR-light. The SLO unit40is provided with a G-light detection element72that detects the G-light reflected by the beam splitter58, an R-light detection element74that detects the R-light reflected by the beam splitter60, and an IR-light detection element76that detects IR-light reflected by the beam splitter62.

The wide-angle optical system80includes an X-direction scanning device82configured by a polygon mirror to scan the light from the light sources42,44,46in an X direction, a Y-direction scanning device84configured by a galvanometer mirror to scan the light from the light sources42,44,46in a Y direction, and an optical system86including a non-illustrated slit mirror and elliptical mirror to widen the angle over which the light is scanned. The optical system86is capable of achieving a field of view (FOV) of the fundus with a larger angle than in conventional technology, enabling a fundus region to be imaged over a wider range than when employing conventional technology. More specifically, a fundus region can be imaged over a wide range of approximately 120 degrees of external light illumination angles from outside the examined eye12(in practice approximately 200 degrees about a center O of the eyeball of the examined eye12as a reference position for an internal light illumination angle capable of being imaged by illuminating the fundus of the examined eye12with scanning light). The optical system86may be configured employing plural lens sets instead of a slit mirror and elliptical mirror. Each scanning device of the X-direction scanning device82and the Y-direction scanning device84may also be scanning devices employing two-dimensional scanners configured by MEMS mirrors.

A system using an elliptical mirror as described in International Applications PCT/JP2014/084619 or PCT/JP2014/084630 may be used in cases in which a system including a slit mirror and an elliptical mirror is used as the optical system86. The respective disclosures of International Application PCT/JP2014/084619 (International Publication WO2016/103484) filed on Dec. 26, 2014 and International Application PCT/JP2014/084630 (International Publication WO2016/103489) filed on Dec. 26, 2014 are incorporated by reference herein in their entireties.

The ophthalmic device110includes fixation targets92U,92D,92L,92R (see alsoFIG.6AtoFIG.6F) that are lit up to attract the gaze of a patient and are configured by light-emitting devices (for example LEDs) provided at positions at the periphery of the optical system86, offset above, below, and to the left and right of the optical axis thereof. The ophthalmic device110further includes a fixation target control device90that lights up the fixation targets92U,92D,92L,92R under the control of the control unit20.

Note that when the ophthalmic device110is installed on a horizontal plane, the “X direction” corresponds to a horizontal direction and the “Y direction” corresponds to a direction perpendicular to the horizontal plane. A direction connecting the center of the pupil of the anterior eye portion of the examined eye12and the center of the eyeball is referred to as the “Z direction”. The X direction, the Y direction, and the Z direction are accordingly perpendicular to one another.

A color fundus image is obtained by imaging the fundus of the examined eye12using G-light and R-light simultaneously. More specifically, the control unit20controls the light sources42,44such that the light sources42,44emit light at the same time, and scans the G-light and R-light across the fundus of the examined eye12using the wide-angle optical system80. G-light reflected from the fundus of the examined eye12is detected by the G-light detection element72, and image data of a second fundus image (a green fundus image) is generated by an image processing section182. Similarly, R-light reflected from the fundus of the examined eye12is detected by the R-light detection element74, and image data of a first fundus image (a red fundus image) is generated by the CPU22of the ophthalmic device110. In cases in which IR-light is illuminated, IR-light reflected from the fundus of the examined eye12is detected by the IR-light detection element76, and image data of an IR fundus image is generated by the CPU22of the ophthalmic device110.

The eye axial length measuring instrument120inFIG.1has two modes for measuring the eye axial length, this being the length of the examined eye12in an eye axial direction, namely a first mode and a second mode. In the first mode, light from a non-illustrated light source is guided into the examined eye12, and interference light generated from interference between reflected light from the fundus and reflected light from the cornea is received, and the eye axial length is measured based on an interference signal represented by the interference light received. The second mode is a mode in which non-illustrated ultrasound waves are employed to measure the eye axial length. The eye axial length measuring instrument120transmits the eye axial length measured using either the first mode or the second mode to the management server140. The eye axial length may be measured using both the first mode and the second mode, in which case an average of the eye axial lengths measured by the two modes is transmitted to the management server140as the eye axial length.

Next, explanation follows regarding a configuration of the management server140, with reference toFIG.3. As illustrated inFIG.3, the management server140includes a control unit160, and a display/operation unit170. The control unit160includes a computer including a CPU162, memory164configured by a storage device, a communication interface (I/F)166, and the like. The display/operation unit170is a graphical user interface for displaying images and for receiving various instructions. The display/operation unit170includes a display172and an input/instruction device174such as a touch panel.

Configuration of the image viewer150is similar to that of the management server140, and so explanation thereof is omitted. An analysis processing program is stored in the memory164of the image viewer150.

The analysis processing program is an example of a “program” of technology disclosed herein.

Next, with reference toFIG.4, explanation follows regarding each of various functions implemented by the CPU162of the image viewer150executing the analysis processing program. The analysis processing program includes an analysis processing function, a display control function, and a processing function. By the CPU162executing the analysis processing program including each of these functions, the CPU162functions as an image processing section182, a display control section184, and a processing section186, as illustrated inFIG.4.

Next, explanation follows regarding operation of the ophthalmic system100, with reference toFIG.5.

The examined eye of the patient is positioned so as to allow imaging of the examined eye of the patient using the ophthalmic device110. As illustrated inFIG.5, at step202the doctor uses the input/instruction device34to instruct the ophthalmic device110to start imaging the fundus of the examined eye12. At step204, the ophthalmic device110images the fundus of the examined eye12(an image1) using the SLO unit40. The fixation targets92U,92D,92L,92R are not lit up at step204. Accordingly, at step204the fundus is imaged in a state in which the optical axis of the examined eye12is aligned with the optical axis of the ophthalmic system100, as illustrated inFIG.6AandFIG.6D. More specifically, as described above, the fundus of the examined eye12is imaged with G light and R light simultaneously to acquire the first fundus image (red fundus image) and the second fundus image (green fundus image), and the image1is obtained as a color fundus image from the first fundus image (red fundus image) and the second fundus image (green fundus image). The image1acquired by such imaging is, as illustrated inFIG.6A, a central image G0(see alsoFIG.8) that is imaged over a range from an up position U0to a down position D0in the up-down direction as viewed from the ophthalmic system100, and from a left position L0to a right position R0in the left-right direction with respect thereto.

A state in which the optical axis of the examined eye12is aligned with the optical axis of the ophthalmic system100is an example of a “first gaze” of technology disclosed herein, and the central image G0is an example of a “reference image” of technology disclosed herein.

When imaging of the image1has been completed, at the next step206the ophthalmic device110transmits the image1to the management server140.

When imaging the image1at step204, various information, such as a patient ID, patient name, age, information as to whether each image is from the right or left eye, the date/time of imaging and visual acuity before treatment, and the date/time of imaging and visual acuity after treatment, is also input to the ophthalmic device110. The various information described above are transmitted from the ophthalmic device110to the management server140during image transmission at step206.

The examined eye of the patient is positioned so as to allow imaging of the examined eye of the patient using the ophthalmic device110. At step208, the doctor uses the input/instruction device34to instruct the ophthalmic device110to start imaging. At step210the ophthalmic device110performs imaging of images2. In the processing of step210, the fundus of the examined eye12is imaged as described above with the gaze of the patient looking up, down, left, or right. A left-tilted image (GL (see alsoFIG.8)), an up-tilted image (GU), a right-tilted image (GR), and a down-tilted image (GD) are thereby obtained. The left and right directions are as viewed from the ophthalmic device110. Accordingly, the left-tilted image (GL) is an image obtained by imaging the fundus in a state in which the patient is looking diagonally toward the right, and the right-tilted image (GR) is an image obtained by imaging the fundus in a state in which the patient is looking diagonally toward the left.

For example, in order to direct the gaze of the patient diagonally upward, the fixation target92U is lit up, such that the gaze of the patient is directed diagonally upward as illustrated inFIG.6B. The fundus is then imaged in this state. Since the gaze of the patient is directed diagonally upward, imaging is performed as far as an upper side position U1further to the upper side than the up position U0. An image of a surface MU01not present in the central image G0is thus included in the up-tilted image (GU).

In order to direct the gaze of the patient diagonally downward, the fixation target92D is lit up, such that the gaze of the patient is directed diagonally downward as illustrated inFIG.6C. The fundus is then imaged in this state. Since the gaze of the patient is directed diagonally downward, imaging is performed as far as a lower side position D1further to the lower side than the down position D0. An image of a surface MD02not present in the central image G0is thus included in the down-tilted image (GD).

In order to direct the gaze of the patient diagonally leftward as viewed from the ophthalmic device110, the fixation target92L is lit up. Accordingly, as illustrated inFIG.6E, the gaze of the patient is directed diagonally leftward (toward the right as viewed from the patient). The fundus is then imaged in this state. Since the gaze of the patient is directed diagonally leftward, imaging is performed as far as a left side position L1further to the left side than the left position L0. An image of a surface ML01not present in the central image G0is thus included in the left-tilted image (GL).

In order to direct the gaze of the patient diagonally rightward as viewed from the ophthalmic device110, the fixation target92R is lit up. Accordingly, as illustrated inFIG.6F, the gaze of the patient is directed diagonally rightward (toward the left as viewed from the patient). The fundus is then imaged in this state. Since the gaze of the patient is directed diagonally rightward, imaging is performed as far as a right side position R1further to the right side than the left position R0. An image of a surface MR02not present in the central image G0is thus included in the right-tilted image (GR).

The gazes diagonally upward, downward, leftward, and rightward are examples of a “second gaze” and a “third gaze” of the present disclosure, and the up-tilted image (GU), the down-tilted image (GD), the left-tilted image (GL), and the right-tilted image (GR) are examples of a “first image” and a “second image” of technology disclosed herein.

At step212, the ophthalmic device110transmits the images2to the management server140, including the up-tilted image (GU), the down-tilted image (GD), the left-tilted image (GL), and the right-tilted image (GR).

At step214, the user (an ophthalmologist or the like) uses the input/instruction device174to instruct the image viewer150to perform analysis processing.

At step216, the image viewer150instructs the management server140to perform image transmission. On being instructed to perform image transmission, at step218the management server140reads the image1and images2, and at step220transmits image data of the image1and images2to the image viewer150.

At step222, the management server140, to which the image data of the image1and images2has been transmitted, displays the image1and images2on the display172. Specifically, the central image G0is centrally disposed, and the up-tilted image (GU), the down-tilted image (GD), the left-tilted image (GL), and the right-tilted image (GR) are respectively disposed at the upper side, the lower side, the left side, and the right side of the central image G0.

At step224, the doctor indicates corresponding points in various combinations of the central image G0with the up-tilted image (GU), the down-tilted image (GD), the left-tilted image (GL), or the right-tilted image (GR).

For example, the user (an ophthalmologist or the like) looks at the central image G0and the up-tilted image (GU) displayed on the display172and uses the input/instruction device174to indicate plural feature points that respectively correspond with each other in the central image G0and the up-tilted image (GU). The feature points may, for example, be a branch point of blood vessels on the fundus, or the optic nerve head. Moreover, plural points (for example twelve points) are indicated for the feature points. The image viewer150receives the plural combinations of feature points (for example twelve pairs thereof) that corresponding across the central image G0and the up-tilted image (GU). At step224, the user (ophthalmologist or the like) uses the input/instruction device174to indicate plural feature points in each of the combinations of the central image G0with the down-tilted image (GD), the central image G0with the left-tilted image (GL), and the central image G0with the right-tilted image (GR) in a similar manner to as described above. The image viewer150receives the combinations plural feature points (for example twelve pairs thereof) for each of the respective combinations.

Indication of the corresponding points at step224is not limited to being performed by the user (ophthalmologist or the like), and may be performed automatically based on the image data of the above images. For example, the image processing section182may identify corresponding feature points by performing template matching based on the image data for the respective combinations of the central image G0with the up-tilted image (GU), the down-tilted image (GD), the left-tilted image (GL), or the right-tilted image (GR).

At step226, the image viewer150executes analysis processing, described in detail later, and at step228displays an analysis results screen300(seeFIG.8) on the display172. The analysis results screen300will be described later.

Next, explanation follows regarding the analysis processing. After the image data of the image1and images2has been received and displayed (step222) as described above, when the plural feature point combinations for the respective combinations of images have been received at step224, the image viewer150executes the analysis program at step266. The analysis processing method illustrated in the flowchart ofFIG.7is implemented by the CPU162of the image viewer150executing the analysis processing program. The analysis processing method is an example of an “image processing method” of technology disclosed herein.

At step242inFIG.7, the processing section186initializes a variable n for discriminating the tilted images to 0, and at step244, the processing section186increments the variable n by 1. Note that the left-tilted image (GL), the up-tilted image (GU), the right-tilted image (GR), and the down-tilted image (GD) are each respectively discriminated by the variable n=1, 2, 3, or 4. The total number N of instances of the variable n is four.

At step246, the image processing section182computes a projective transformation matrix to transform the tilted image n discriminated by variable n onto the central image G0, based on the positions in the twelve corresponding points combinations specified on combination of the central image G0with the tilted image n. The projective transformation matrix is a matrix employed for projective transformation so as to transform each pixel in the tilted image n, so that the position of each pixel in the tilted image n is positioned at a corresponding position in the central image G0.

At step248, the image processing section182employs the projective transformation matrix computed at step248to perform projective transformation of the tilted image n. A projective-transformed tilted image n1is created as a result.

At step250, a reference projection point is identified as a datum point in the central image G0(for example a center position) projected onto the projective-transformed tilted image n1.

At step252, the processing section186determines whether or not the variable n is the same as the total number N. Cases in which the variable n is not the same as the total number N mean that there is still a tilted image remaining that has not been subjected to the above processing (steps246to250), and so the analysis processing returns to step244. However, cases in which the variable n is the same as the total number N mean that all of the tilted images have been subjected to the above processing (steps246to250), and so the analysis processing proceeds to step254.

Through the above processing, different projective transformation matrices are computed to transform each of the left-tilted image (GL), the up-tilted image (GU), the right-tilted image (GR), and the down-tilted image (GD) onto the central image G0. The respective projective transformation matrices are employed on each of the left-tilted image (GL), the up-tilted image (GU), the right-tilted image (GR), and the down-tilted image (GD) to create a post-transformation left-tilted image (GL1), a post-transformation up-tilted image (GU1), a post-transformation right-tilted image (GR1), and a post-transformation down-tilted image (GD1). Reference projection points are then identified as the datum point (for example the center position) of the central image G0projected onto the post-transformation left-tilted image (GL1), the post-transformation up-tilted image (GU1), post-transformation right-tilted image (GR1), and post-transformation down-tilted image (GD1).

At step254, the display control section184takes the post-transformation left-tilted image (GL1), the post-transformation up-tilted image (GU1), the post-transformation right-tilted image (GR1), and the post-transformation down-tilted image (GD1) and uses these to create an assembled montage image based on the respective reference projection points therein.

Note that the post-transformation left-tilted image (GL1), the post-transformation up-tilted image (GU1), the post-transformation right-tilted image (GR1), and the post-transformation down-tilted image (GD1) each include the first fundus image (red fundus image) and the second fundus image (green fundus image). The montage image accordingly also includes the first fundus image (red fundus image) and the second fundus image (green fundus image).

The structure of the eye is configured by the vitreous body covered by plural layers that each have a different structure. These plural layers include the retina, the choroid, and the sclera from the side closest to the vitreous body outward. R-light passes through the retina and travels as far as the choroid. Accordingly, the first fundus image (red fundus image) includes information relating to blood vessels (retinal blood vessels) present in the retina and information relating to blood vessels (choroidal blood vessels) present in the choroid. By contrast, G-light only travels as far as the retina. Accordingly, the second fundus image (green fundus image) includes information relating to the blood vessels (retinal blood vessels) present in the retina.

At step256the display control section184creates a choroidal vascular image from the montage image.

The choroidal vascular image is generated in the following manner. The image processing section182of the management server140subjects the second fundus image (green fundus image) in the montage image to black hat filter processing so as to extract the retinal blood vessels from the second fundus image (green fundus image). Next, the image processing section182performs in-painting processing employing the retinal blood vessels extracted from the second fundus image (green fundus image) to remove these retinal blood vessels from the first fundus image (red fundus image) of the montage image. Namely, processing is performed that uses position information relating to the retinal blood vessels extracted from the second fundus image (green fundus image) to infill the retinal blood vessel structures in the first fundus image (red fundus image) with the same pixel values to those of surrounding pixels. The image processing section182then subjects the image data of the first fundus image (red fundus image) from which the retinal blood vessels have been removed to contrast-limited adaptive histogram equalization, thereby emphasizing the choroidal blood vessels in the first fundus image (red fundus image). A choroidal vascular image is obtained thereby. The generated choroidal vascular image is stored in the memory164.

Regarding the method used to generate the choroidal fundus image, the disclosure of Japanese Patent Application No. 2018-052246, filed on Mar. 20, 2018, is incorporated in its entirety by reference herein.

At step254, the display control section184creates analysis result screen data.

FIG.8illustrates the analysis results screen300. As illustrated inFIG.8, the analysis results screen300includes an image display region302, a patient information display region304, and a folder display region306.

The folder display region306includes a pre-processing folder360and an analysis folder370. The pre-processing folder360is provided with an individual display icon362and a montage display icon3642. Various icons are provided in the analysis folder370, as illustrated inFIG.10.

The image display region302illustrated inFIG.8is displaying content corresponding to a case in which the individual display icon362in the pre-processing folder360has been operated. As illustrated inFIG.8, the image display region302includes a pre-transformation-image display region322, and a post-transformation-image display region324. The pre-transformation-image display region322is includes a region330to display the central image G0, and regions330n1to330n4arranged around the region330at the center to display the left-tilted image (GL), the up-tilted image (GU), the right-tilted image (GR), and the down-tilted image (GD) at the left side, upper side, right side, and lower side of the region330, respectively.

The left-tilted image (GL) contains an image extending as far as the left side position L1further to the left side than the left position L0. The up-tilted image (GU) contains an image extending as far as the upper side position U1further to the upper side than the up position U0. The right-tilted image (GR) contains an image extending as far as the right side position R1further to the right side than the left position R0. The down-tilted image (GD) contains an image extending as far as the lower side position D1further to the lower side than the down position D0.

The post-transformation-image display region324includes a region330to display the central image G0, and regions340n1to340n4arranged around the region330at the center to display the post-transformation left-tilted image (GL1), the post-transformation up-tilted image (GU1), the post-transformation right-tilted image (GR1), and the post-transformation down-tilted image (GD1) at the left side, upper side, right side, and lower side of the region330, respectively. The left side position L1in the left-tilted image (GL) is positioned at a position L11in the post-transformation left-tilted image (GL1). The upper side position U1in the up-tilted image (GU1) is positioned at a position U11in the post-transformation up-tilted image (GU1). The right side position R1in the right-tilted image (GR1) is positioned at a position R11in the post-transformation right-tilted image (GR1). The lower side position D1in the left-tilted image (GD1) is positioned at a position D11in the post-transformation down-tilted image (GD1).

Icons and buttons for instructing image generation, described later, are displayed on the display screen of the image viewer150, also described later. When the user of the image viewer150(an ophthalmologist or the like) clicks on one of the icons etc., an instruction signal corresponding to the clicked icon etc. is transmitted from the image viewer150to the management server140. On receipt of the instruction signal from the image viewer150, the management server140generates an image corresponding to the instruction signal, and transmits image data of the generated image to the image viewer150. The image viewer150that has received the image data from the management server140then displays an image based on the received image data on a display. Display screen generation processing is performed in the management server140by the CPU162executing a display screen generation program.

When the montage display icon364in the pre-processing folder360of the analysis results screen300illustrated inFIG.8is operated, the image display region302is changed to the content illustrated inFIG.9. As illustrated inFIG.9, the pre-transformation-image display region322of the image display region302is similar content to that of the region322inFIG.8. On the other hand, the post-transformation-image display region324includes a region380to display a montage image G01created at step254.

The montage image G01includes the central image G0and a portion ML011further to the left side than the central image G0. The portion ML011is configured by the post-transformation left-tilted image (GL1) from which the central image G0has been subtracted, and is delineated by lines at the center of respective common portions for portions that are common to the up-tilted image (GU1) and the down-tilted image (GD1), respectively.

The montage image G01includes a portion MU011further to the upper side than the central image G0. The portion MU011is configured by the post-transformation up-tilted image (GU1) from which the central image G0has been subtracted, and is delineated by lines at the center of respective common portions for portions that are common to the left-tilted image (GL1) and the right-tilted image (GR1), respectively.

The montage image G01includes a portion MR011further to the right side than the central image G0. The portion R011is configured by the post-transformation right-tilted image (GR1) from which the central image G0has been subtracted, and is delineated by lines at the centers of respective common portions for portions that are common to the up-tilted image (GU1) and the down-tilted image (GD1), respectively.

The montage image G01includes a portion MD011further to the lower side than the central image G0. The portion MD011is configured by the post-transformation down-tilted image (GD1) from which the central image G0has been subtracted, and is delineated by lines at the centers of respective common portions for portions that are common to the right-tilted image (GR1) and the left-tilted image (GL1).

In this manner, the montage image G01includes the portion ML011, the portion MU011, the portion MR011, and the portion MD011arranged around the central image G0. The montage image G01accordingly includes portions not in the central image G0obtained by imaging the fundus with the optical axis of the examined eye12aligned with the optical axis of the ophthalmic system100, i.e. the portion ML011, the portion MU011, the portion MR011, and the portion MD011. This enables a greater amount of information to be obtained about the fundus.

FIG.10illustrates a display screen displayed in a case in which the analysis folder370of the analysis results screen300on the image viewer150has been operated. As illustrated inFIG.10, the image display region302includes an RG image display region382and a choroidal vascular image384. A montage image G02is displayed as an RG image in the RG image display region382. A choroidal vascular image G03created at step256is displayed as the choroidal vascular image384.

Conventional method technology relating to a panoramic fundus image synthesizing device and method has been disclosed. Such conventional technology does not enable a wider panoramic image to be generated from plural fundus images obtained by imaging the fundus region over a wide range. However, the present exemplary embodiment enables a wider panoramic image (montage image) to be generated from the four fundus images obtained by imaging the fundus while moving the gaze up, down, left, and right.

The conventional technology accordingly does not enable computation of a projective transformation matrix to perform projective transformations on each of the pixels of a first image so that the positions of the respective pixels in the first image are positioned at corresponding positions in a reference image. However, in the present exemplary embodiment, such a projective transformation matrix can be computed, thereby enabling each of the pixels of the first image to be projection transformed using this projective transformation matrix.

Explanation follows regarding various modified examples of the technology disclosed herein.

First Modified Example

Although in the exemplary embodiment described above the wider panoramic image (montage image) is generated from the four fundus images obtained by imaging the fundus while moving the gaze up, down, left, and right, the technology disclosed herein is not limited thereto. For example, a wider panoramic image (montage image) may be generated from four fundus images obtained by imaging the fundus while moving the gaze in other directions such as, for example, diagonally up and to the right, diagonally up and to the left, diagonally down and to the left, and diagonally down and to the right, instead of, or in addition to up, down, left, and right.

Note that the number of fundus images obtained by imaging the fundus while moving the gaze is not limited to four or eight, and is merely one or more.

Second Modified Example

Although a choroidal vascular image is created in the exemplary embodiment described above (at step256inFIG.7), the technology disclosed herein is not limited thereto. For example, additional processing may be executed between step256and step258inFIG.7.FIG.11is a flowchart illustrating a VV quantitative value computation program executed between step256and step258ofFIG.7in a second modified example.

At step272inFIG.11, the image processing section182identifies the position of a vortex vein (hereafter VV) in the choroidal vascular image created at step256.

The image processing section182sets a movement direction (blood vessel running direction) for each choroidal blood vessel appearing in the choroidal vascular image G03montage image. Specifically, firstly the image processing section182executes the following processing for each pixel in the choroidal vascular image G03. Namely, for each pixel the image processing section182sets a region (cell) centered on this pixel and creates a histogram of the directions of brightness gradient at each of the pixels in the cell. Next, the image processing section182takes the gradient direction having the lowest count in the respective histogram of each cell as the movement direction at the pixel for the cell. The gradient direction corresponds to the blood vessel running direction. Note that the following reason is why the gradient direction having the lowest count corresponds to the blood vessel running direction. Namely, the brightness gradient is small along the blood vessel running direction, whereas the brightness gradient is large along other directions (for example there is a large difference in brightness between blood vessels and tissue other than blood vessels). When the respective histogram of brightness gradients has been created for each of the pixels, then the count in the histogram is small for the direction along the blood vessel running direction. The above processing is performed to set the blood vessel running direction at each pixel in the choroidal vascular image.

The image processing section182sets initial positions for M (natural number)×N (natural number) individual elements (=L). Specifically, the image processing section182sets M individual positions at uniform spacings in the choroidal vascular image G03vertically and N individual positions therein horizontally, namely a total of L individual initial positions.

The image processing section182then estimates a VV position. Specifically the image processing section182performs the following processing at each of the L individual positions. Namely, the image processing section182acquires the blood vessel running direction at an initial position (one of the L individual positions), moves this element by a specific distance along the acquired blood vessel running direction, repeats acquisition of the blood vessel running direction but this time at the position moved to, and then moves the element by the specific distance along this acquired blood vessel running direction. Such movements of a specific movement distance along the acquired blood vessel running directions are repeated a preset number of times. The above processing is executed for all of the L individual positions. At this point in time, a point where a given number of elements or more have collected together is taken as the VV position.

At step272, the image processing section182detects the optic nerve head in the green fundus image of the montage image. Specifically, since the optic nerve head is the brightest region in the green fundus image, the image processing section182detects as the optic nerve head a region of a specific number of pixels where the pixel values are greatest in the green fundus image that has been read as described above. A center position of the region containing the brightest pixels is computed as the coordinates of the position of the optic nerve head and stored in the memory164.

Also at step272, the image processing section182detects the macula. Specifically, since the macula is a dark region in the choroidal vascular image, the image processing section182detects as the macula a region of a specific number of pixels where the pixel values are smallest in the choroidal vascular image G03read as described above. The center position of the region containing the darkest pixels is computed as the coordinates of the position of the macula and stored in the memory164.

Also at step272, the image processing section182identifies positions corresponding to the VV position, the optic nerve head position, and the macula position in the post-transformation left-tilted image (GL1), the post-transformation up-tilted image (GU1), the post-transformation right-tilted image (GR1), and the post-transformation down-tilted image (GD1).

At step274, the image processing section182projects the post-transformation left-tilted image (GL1), the post-transformation up-tilted image (GU1), the post-transformation right-tilted image (GR1), and the post-transformation down-tilted image (GD1) onto a three-dimensional (3D) eyeball surface based on the respective reference projection points.

At step276, the image processing section182computes VV quantitative values from the three-dimensional (3D) eyeball surface having the post-transformation left-tilted image (GL1), the post-transformation up-tilted image (GU1), the post-transformation right-tilted image (GR1), and the post-transformation down-tilted image (GD1) projected thereon.

Computation processing for the VV quantitative values is firstly VV distance computation processing, and is secondly VV angle computation processing.

First, explanation follows regarding the VV distance computation processing. The VV distances are firstly a distance between the optic nerve head position and the VV position, and is secondly a distance between the macula position and the VV position.

Firstly the distance between the optic nerve head position and the VV position, and secondly the distance between the macula position and the VV position, are computed on the three-dimensional (3D) eyeball surface.

Next, explanation follows regarding the VV angle computation processing. VV angles include an angle formed between a line from the macula position to the optic nerve head position and a line from the optic nerve head position to the VV position, and an angle formed between a line from the optic nerve head position to the macula position and a line from the macula position to the VV position. The method of computing the angle θ formed between a line from the macula position to the optic nerve head position and a line from the optic nerve head position to the VV position may employ computation by conformal projection or spherical trigonometry.

The VV quantitative values are included in the analysis result screen data created at step258inFIG.7in the second modified example. The VV quantitative values are accordingly displayed on the display172of the image viewer150.

Third Modified Example

In the exemplary embodiment described above an example has been described in which a fundus image is acquired by the ophthalmic device110with an internal light illumination angle of about 200 degrees. However, the technology disclosed herein is not limited thereto, and the technology disclosed herein may, for example, be applied even when the fundus image imaged by an ophthalmic device has an internal illumination angle of 100 degrees or less.

Fourth Modified Example

In the exemplary embodiment described above the ophthalmic device110uses SLO to image the fundus. However, the technology disclosed herein is not limited thereto, and for example the fundus may be imaged using a fundus camera.

Fifth Modified Example

The exemplary embodiment described above describes an example of the ophthalmic system100equipped with the ophthalmic device110, the eye axial length measuring instrument120, the management server140, and the image viewer150; however the technology disclosed herein is not limited thereto. For example, as a first example, the eye axial length measuring instrument120may be omitted, and the ophthalmic device110may be configured so as to further include the functionality of the eye axial length measuring instrument120. Moreover, as a second example, the ophthalmic device110may be configured so as to further include the functionality of one or both of the management server140and the image viewer150. For example, the management server140may be omitted in cases in which the ophthalmic device110includes the functionality of the management server140. In such cases, the analysis processing program is executed by the ophthalmic device110or the image viewer150. Moreover, the image viewer150may be omitted in cases in which the ophthalmic device110includes the functionality of the image viewer150. As a third example, the management server140may be omitted, and the image viewer150may be configured so as to execute the functionality of the management server140.

Other Modified Examples

The data processing described in the exemplary embodiment described above is merely an example thereof. Obviously, unnecessary steps may be omitted, new steps may be added, and the sequence of processing may be changed within a range not departing from the spirit thereof.

Moreover, although in the exemplary embodiment described above an example has been given of a case in which data processing is implemented by a software configuration utilizing a computer, the technology disclosed herein is not limited thereto. For example, instead of a software configuration utilizing a computer, the data processing may be executed solely by a hardware configuration of FPGAs or ASICs. Alternatively, a portion of processing in the data processing may be executed by a software configuration, and the remaining processing may be executed by a hardware configuration.