Patent ID: 12211186

DETAILED DESCRIPTION OF THE EMBODIMENTS

Example of Involuntary Eye Movement Elimination and Video Emphasis

FIG.1is a descriptive view showing an example of involuntary eye movement elimination and video magnification (video emphasis). T is the time axis. The arrow of the time axis T is the time axis direction, or in other words, the direction in which time progresses. (A) shows fundus video data100. The fundus video data100is video data attained by capturing a region including a macula101, an optic disc102, and blood vessels103of the fundus of a subject. The fundus video data100includes fundus image frames F1to F4in chronological order, for example. The smaller the suffix number is, the earlier the frame is. If not distinguishing between the fundus image frames F1to F4, the fundus image frames are referred to as the fundus image frames F. For ease of description, only fundus image frames up to the fundus image frame F4are included here, but there may be fundus image frames F beyond the fundus image frame F4.

The fundus image frames F1to F4include, as image data, the macula101, the optic disc102, and the blood vessels103. Tissue other than the macula101, the optic disc102, and the blood vessels103are omitted from the depiction. The fundus image frame F1serves as a reference frame for position-matching the fundus image frames F2to F4. The fundus image frames F2to F4depict the macula101, the optic disc102, and the blood vessels103of the fundus image frame F1with dotted lines.

The fundus is imaged in a state where the imaging device is fixed in place, and thus, the macula101, the optic disc102, and the blood vessels103of the fundus image frames F2to F4are offset from the positions of the macula101, the optic disc102, and the blood vessels103of the fundus image frame F1due to involuntary eye movement.

Involuntary eye movement is movement of the eye, referred to as a saccade, in which the gaze repeatedly shifts rapidly (approximately 100-500 instances per second) over short periods of time (approximately 20-80 ms). Depending on the dynamic characteristics of the involuntary eye movement, involuntary eye movement can be classified into the following types: microsaccades in which the eye movement is relatively large and fast; drift, in which the movement is large and slow; and tremors, in which the movement is small and high frequency.

Also, the amount of blood flowing through the blood vessels103increases or decreases depending on the heartbeat. For example, the amount of blood flowing through the blood vessels103is greater in the fundus image frames F2and F4compared to the fundus image frames F1and F3. Thus, the blood vessels103are wider and darker in color in the fundus image frames F2and F4than in the fundus image frames F1and F3.

In (A), the blood vessels103in the fundus image frame F1are the lightest in color, and the blood vessels103in the fundus image frames F2and F4are the darkest in color. Also, the color of the blood vessels103in the fundus image frame F3is darker than the blood vessels103of the fundus image frame F1but lighter than the blood vessels103of the fundus image frames F2and F4.

Video data V is the fundus video data100from playing back the fundus image frames F1to F4in the time direction. The video data V is the fundus video data100at the display timing of the fundus image frame F4, and the macula101, the optic disc102, and the blood vessels103of the previous fundus image frames F1to F3are visible as residual images. Thus, a physician who is the user would have difficulty knowing whether the width and color of the blood vessels103is changing due to the heartbeat or involuntary eye movement. That is, in reality, it is difficult to distinguish between changes in the width and color of the blood vessels103and positional shifts and color changes in the blood vessels between frames resulting from involuntary eye movement.

(B) shows fundus video data110in which involuntary eye movement has been eliminated from the fundus video data100of (A). The fundus video data110includes fundus image frames G1to G4in chronological order. The fundus image frames G1to G4are frames in which involuntary eye movement was eliminated from the fundus image frames F1to F4, respectively, through image processing. If not distinguishing between the fundus image frames G1to G4, the fundus image frames are referred to as the fundus image frames G.

By correcting for positional offset between frames resulting from involuntary eye movement, the involuntary eye movement is eliminated, and thus, the macula101, the optic disc102, and the blood vessels103of the fundus image frames G2to G4are, respectively, displayed in the same positions as the macula101, the optic disc102, and the blood vessels103of the fundus image frame F1, which are indicated with the dotted lines in the fundus image frames F2to F4. On the other hand, movement and color changes in the blood vessels103resulting from the heartbeat are not eliminated. Thus, confusion between involuntary eye movement and pulsation and color changes in specific tissue such as the blood vessels103is mitigated, and the user such as a physician can observe the pulsation and color changes of the specific tissue without the influence of the involuntary eye movement.

(C) shows fundus video data120generated by performing processing in which the blood vessels103are subjected to video emphasis as specific tissue within the fundus video data110of (B), with changes in the width and color of the blood vessels103being displayed with emphasis. The fundus video data120includes fundus image frames H1to H4in chronological order. The fundus image frames H1to H4are frames generated by performing video emphasis on the fundus image frames G1to G4, respectively, through image processing. If not distinguishing between the fundus image frames H1to H4, the fundus image frames are referred to as the fundus image frames H. Video emphasis is a technique by which minute changes in movement that are one pixel or less in the footage and small changes in color and movement are displayed with emphasis. By performing video emphasis on the fundus video, it is possible to display with emphasis changes in the width (movement) and color of the blood vessels resulting from the heartbeat. As a result, the user such as a physician can view with ease the color changes in the blood vessel103resulting from the heartbeat. Below, the mechanism by which the involuntary eye movement is eliminated will be described.

<Ophthalmic System>

FIG.2is a system configuration drawing showing an example of an ophthalmic system. In the ophthalmic system200, a slit lamp202(slit lamp microscope) and a surgical microscope203constituting an ophthalmic apparatus201, a management server204, and a terminal205are connected in a manner enabling communication therebetween via a network206such as a LAN (local area network), a WAN (wide area network), or the internet. The slit lamp202is a microscope in which a subject eye is illuminated with slit light, and an illuminated cross section of the subject eye is imaged and observed from the side. The surgical microscope203is a microscope specialized for surgery. Both the slit lamp202and the surgical microscope203can generate, transmit, store, and display the fundus video data100,110, and120as subject eye video data. As long as the ophthalmic apparatus201can image the subject eye of the patient, the ophthalmic apparatus201may be a fundus camera, a scanning laser ophthalmoscope (SLO), or an optical coherence tomography (OCT) device.

The management server204acquires and stores fundus video data from the ophthalmic apparatus201, or transmits the fundus video data to the ophthalmic apparatus201or the terminal205. The terminal205receives and plays back the fundus video data from the ophthalmic apparatus201or the management server204, or transmits the fundus video data100,110, and120to the ophthalmic apparatus201or the management server204.

At least one of the ophthalmic apparatus201, the management server204, and the terminal205can execute the involuntary eye movement elimination described inFIG.1, and at least one of the ophthalmic apparatus201, the management server204, and the terminal205can execute the video emphasis described inFIG.1. That is, the involuntary eye movement elimination and the video emphasis may be executed by the same device or by different devices.

Computer Hardware Configuration Example

Next, a computer hardware configuration example will be described. A computer is a collective term for the ophthalmic apparatus201, the management server204, and the terminal205shown inFIG.2. If the computer is the ophthalmic apparatus201, then a light source, an optical system, and a sensor (not shown) are included.

Hardware Configuration Example of Computer

FIG.3is a block diagram for illustrating a hardware configuration example of each of a computer. A computer300includes a processor301, a storage device302, an input device303, an output device304, and a communication interface (communication IF)305. The processor301, the storage device302, the input device303, the output device304, and the communication IF305are coupled to one another through a bus306. The processor301is configured to control the computer300. The storage device302serves as a work area for the processor301. The storage device302is also a non-transitory or transitory recording medium configured to store various programs and various kinds of data. Examples of the storage device302include a read only memory (ROM), a random access memory (RAM), a hard disk drive (HDD), and a flash memory. The input device303is configured to input data. Examples of the input device303include a keyboard, a mouse, a touch panel, a numeric keypad, and a scanner. The output device304is configured to output data. Examples of the output device304include a display, and a printer. The communication IF305is coupled to the network206, and is configured to transmit and receive data.

Functional Configuration Example of Video Processing Device

FIG.4is a block diagram showing a mechanical configuration example of a video processing apparatus, andFIG.5is a flowchart showing an example of image processing steps performed by the video processing apparatus. InFIG.4, the video processing apparatus400has an acquisition unit401, an elimination unit402, an emphasis unit403, and an output unit. The video processing apparatus400has at least one of the elimination unit402and the emphasis unit403in a computer300. The video processing apparatus400is constituted of one computer300, or a plurality of linked computers300.

The acquisition unit401, the elimination unit402, the emphasis unit403, and the output unit are specifically realized by a processor301executing programs stored in a storage device302shown inFIG.3, for example.

The acquisition unit401acquires the fundus video data100from the storage device302in the video processing apparatus400or from another computer300outside of the video processing apparatus400(step S501). The elimination unit402eliminates involuntary eye movement from the fundus video data100on the basis of the fundus video data100acquired by the acquisition unit401, or in other words, corrects the positional offset in the fundus region between frames resulting from involuntary eye movement to eliminate the effect of involuntary eye movement from the video data (step S502).

The emphasis unit403executes emphasis processing through VM for the fundus video data110in which the effect of involuntary eye movement was eliminated by the elimination unit402(step S503). Specifically, for example, the emphasis unit403emphasizes specific frequency components in the time direction. The emphasis unit403may emphasize the entirety of each of the fundus image frames F of the fundus video data100, or may emphasize regions including specific tissue such as the macula101, the optic disc102, the blood vessels103, or the like.

The output unit404outputs the fundus video data120in which the specific tissue was emphasized by the emphasis unit403(step S504). Specifically, the output unit404displays the fundus video data120in the display device of the video processing apparatus400, or transmits the fundus video data120from the video processing apparatus400to another computer300, for example.

FIG.6Ais a block diagram showing a detailed functional configuration example of the elimination unit402. The elimination unit402includes a first elimination unit601, a second elimination unit602, and a third elimination unit603. First, the first elimination unit601will be described.

The first elimination unit601eliminates involuntary eye movement in order for the emphasis unit403to emphasize the movement and color change of specific tissue such as the blood vessels103. Specifically, the first elimination unit601is also an involuntary eye movement elimination unit, for example. As described inFIG.1, the first elimination unit601sets the fundus image frame F1as the reference frame, and executes position-matching with the other fundus image frames F2to F4as comparison frames. Here, the reference frame was set as the oldest fundus image frame F1among the fundus image frames F1, but the reference frame may be set to any of the fundus image frames F2to F4. Setting the oldest fundus image frame F1as the reference frame would improve real-time processing.

As a result of the positional offset of the fundus due to involuntary eye movement being corrected through position-matching of the frames, the effect of involuntary eye movement is eliminated from the corrected video data, and the blood vessels103, the optic disc102, and the macula101are in the same position for all of the frames. Thus, the emphasis unit403, to be described later, emphasizes the change in width and color of the blood vessels103resulting from the heartbeat.

Position-matching between the reference frame and the comparison frames is executed through non-rigid position-matching using an affine transformation matrix including transformation through translation, rotation, and expansion/contraction. Also, the elimination unit402may use an algorithm such as scale-invariant feature transform (SIFT) or speeded up robust features (SURF), which is a speeded up version of SIFT.

Additionally, the first elimination unit601may use, as the evaluation function, the sum of squared differences (SSD) for pixel values at the same position, the sum of absolute differences (SAD) for pixel values at the same position, mutual information, or cross-correlation in order to perform position-matching between the reference frame and comparison frames. If mutual information or cross-correlation is used, for example, then the greater than 0 the correlation value is, the more similar the reference frame and the comparison frames are, and the less the correlation value is, the less similar the reference frame and the comparison frames are.

Thus, by executing position-matching between the reference frame and the comparison frames, the positions of the macula101, the optic disc102, and the blood vessels103of the fundus image frames F2to F4that are the comparison frames ofFIG.1are made to match the respective positions of the macula101, the optic disc102, and the blood vessels103of the reference frame (fundus image frame F1) depicted with the dotted lines. As a result, the positions of the macula101, the optic disc102, and the blood vessels103match in the fundus image frames F1to F4, and therefore, involuntary eye movement is eliminated from the fundus video data100.

Also, the first elimination unit601may eliminate, from the fundus video data100, comparison frames in which the above-mentioned correlation value is less than or equal to a threshold. A comparison frame with a correlation value less than or equal to the threshold has a low degree of similarity to the reference frame, and thus, can be said to be frames showing microsaccades, which are a relatively large and fast eye movements, microsaccades being one of the dynamic characteristics of the involuntary eye movement. Thus, by eliminating comparison frames with a correlation value less than or equal to the threshold from the fundus video data100, the first elimination unit601can generate the fundus video data110in which microsaccades are not displayed.

Next, the second elimination unit602will be described. The second elimination unit602executes elimination processing for the emphasis unit403to emphasize only the color change, among the movement and color change, of specific tissue such as the blood vessels103. Specifically, the second elimination unit602has a separation unit621, a time filtering unit622, a phase noise elimination unit623, an attenuation unit624, and a reconstruction unit625, for example.

The separation unit621uses a known filtering process such as complex steerable pyramids to separate the fundus image frames G into a high frequency component hpr, a low frequency component lpr, and a plurality of orthogonal components every time a fundus image frame G that was subjected to involuntary eye movement elimination by the first elimination unit601is inputted.

That is, the separation unit621separates localized wave amplitudes (high frequency component hpr, low frequency component lpr) from the phases (plurality of orthogonal components) of the wave. The separation unit621outputs the high frequency component hpr and the low frequency component lpr to the reconstruction unit625and outputs the plurality of orthogonal components to the time filtering unit622.

The time filtering unit622independently filters, by time, the phases (plurality of orthogonal components) of the fundus image frames F by position, direction, and scale. The phase noise elimination unit623applies spatial smoothing that is weighted by amplitude to increase the S/N ratio of the phases.

The attenuation unit624attenuates the phases that were band-passed by time by the time filtering unit622. As a result, movement of specific tissue (e.g., the blood vessels103) due to errors (corresponding to heartbeats) occurring due to position-matching by the first elimination unit601is suppressed. The reconstruction unit625reconstructs the fundus image frames G using the output from the attenuation unit624, the high frequency component hpr, and the low frequency component lpr. Thus, the reconstructed fundus image frames G are image frames in which the movement of specific tissue is suppressed.

Next, the third elimination unit603will be described. The third elimination unit603executes elimination processing for the emphasis unit403to emphasize only the movement, among the movement and color change, of specific tissue such as the blood vessels103. Specifically, for example, the third elimination unit603accumulates the fundus image frames G that were subjected to involuntary eye movement elimination by the first elimination unit601, and determines whether the difference in color density in specific tissue (e.g., the blood vessels103) between two consecutive fundus image frames G is greater than or equal to a threshold. If the difference is greater than or equal to the threshold, then the fundus image frame G with the lighter color density for the specific tissue (e.g., the blood vessels103), among the two consecutive fundus image frames G, is eliminated.

In the case of the fundus image frames G1to G4shown in (B) ofFIG.1, for example, if the difference in color density of the blood vessels103between the fundus image frames G1and G2is greater than or equal to the threshold, the third elimination unit603eliminates the fundus image frame G1, which has the lighter color density for the blood vessels103. If the difference in color density of the blood vessels103between the fundus image frames G2and G3and between the fundus image frames G3and G4is not greater than or equal to the threshold, the fundus image frames G2and G3are not eliminated.

FIG.6Bis a block diagram showing a detailed functional configuration example of the emphasis unit403. The emphasis unit403has a spatial separation unit701, a time filtering unit702, amplification units703-1,703-2. . .703-n(n being an integer of 2 or greater), addition units704-1,704-2. . .704-n, and a reconstruction unit705. If not distinguishing between the amplification units703-1,703-2. . .703-n, these are referred to simply as the amplification units703. If not distinguishing between the addition units704-1,704-2. . .704-n, these are referred to simply as the addition units704. First, the spatial separation unit701will be described.

The spatial separation unit701separates the fundus image frames G into a plurality of different spatial frequency bands (band1, band2. . . band n) every time a fundus image frame G subjected to involuntary eye movement elimination by the elimination unit402is inputted. A known filtering process such as bandpass filters and complex steerable pyramids can be used. The spatial frequency increases in the order of the band1, the band2. . . the band n. The image data g1allocated to the band1, the image data g2allocated to the band2. . . and the image data gn allocated to the band n by the spatial separation unit701are outputted to the time filtering unit702.

Next, the time filtering unit702uses a known filtering process such as a second-order infinite impulse response (IIR) filter to extract 60-80 kHz frequency components, which are frequencies at which the human heart beats, from the image data g1, g2. . . gn.

The image data g1that has passed through the time filtering unit702is amplified on the basis of an emphasis coefficient set by the amplification unit703-1. The emphasis coefficient is set to a factor of 10, for example, and the amplitude of the image data g1is amplified by 10 times. The image data g1that has passed through the amplification unit703-1is added by the addition unit704-1to the image data g1outputted from the spatial separation unit701and outputted to the reconstruction unit705. Similarly, the image data g2is also outputted to the reconstruction unit705via the amplification unit703-2and the addition unit704-2. The image data gn is also outputted to the reconstruction unit705via the amplification unit703-nand the addition unit704-n.

The emphasis coefficient of the amplification unit703is the same for the entirety of the fundus image frame G subjected to involuntary eye movement elimination by the elimination unit402, but the emphasis coefficient may be set spatially (for each pixel). Specifically, an emphasis region such as the optic disc102or the blood vessels103is extracted by image processing employing artificial intelligence or the like or by user instruction, and the emphasis coefficient of the pixels of the extracted emphasis region is set to a differing value than other regions in the fundus image frame G. As a result, video emphasis is performed only on the emphasis regions of the fundus image frame G.

The reconstruction unit705uses the image data h1, h2. . . hn outputted from the addition units704-1,704-2. . .704-nto reconstruct the fundus image frames G. In this manner, the reconstructed fundus image frames G are the image frames H1to H4in which periodic fluctuations resulting from the heartbeat are emphasized for specific tissue.

The user (ophthalmologist or the like) can select among a plurality of video emphasis modes via an input device303. At least the following video emphasis modes are made available: normal mode in which both movement and color changes are emphasized; color emphasis mode in which only color change is emphasized; and movement emphasis mode in which only movement is emphasized. If normal mode is selected, the fundus video data100is subjected by the first elimination unit601of the elimination unit402to correction of positional offset among frames resulting from involuntary eye movement, and subjected to video emphasis by the emphasis unit403.

If color mode is selected, the fundus video data100is subjected by the first elimination unit601of the elimination unit402to correction of positional offset among frames resulting from involuntary eye movement, and then movement of specific tissue is eliminated by the second elimination unit602. Then, video emphasis is performed by the emphasis unit403.

If movement emphasis mode is selected, the fundus video data100is subjected by the first elimination unit601of the elimination unit402to correction of positional offset among frames resulting from involuntary eye movement, and then color change is eliminated by the third elimination unit603. Then, video emphasis is performed by the emphasis unit403.

<Fundus Video Data120Subjected to Video Emphasis>

FIG.7is a descriptive drawing showing the fundus video data120subjected to video emphasis by the emphasis unit403. InFIG.7, periodic fluctuation resulting from the heartbeat is emphasized for specific tissue such as the blood vessels103. Color change of the blood vessels103is observed as a darkening of the blood vessel103when blood is sent therethrough and a lightening of the blood vessel103thereafter, for example. The movement of the blood vessels103involves expansion when blood is pumped therethrough (increased width of the blood vessels103), and then contraction of the blood vessels103. Movement and color change tend to be greater for veins than arteries.(A) shows the fundus video data120generated by performing video emphasis on the fundus video data110subjected to involuntary eye movement elimination by the first elimination unit601. In this fundus video data120, both movement and color changes of the blood vessels103are emphasized.(B) shows the fundus video data120generated by performing video emphasis on the fundus video data110subjected to involuntary eye movement elimination by the first elimination unit601and movement elimination for specific tissue by the second elimination unit602. In this fundus video data120, the specific tissue is the blood vessels103, and only the color change of the blood vessels103is emphasized.(C) shows the fundus video data120generated by performing video emphasis on the fundus video data110subjected to involuntary eye movement elimination by the first elimination unit601and color change elimination by the third elimination unit603. In this fundus video data120, the color change is eliminated (the fundus image frame G1with the lighter color density is eliminated), and only the movement of the blood vessels103(change in width of the blood vessels) is emphasized.

Display Examples of Fundus Video Data

Next, display examples of the fundus video data100,110, and120will be described with reference toFIGS.8to10. The computer300may display the fundus video data110and120in real time during examination, treatment, and surgery, or may read the fundus video data110and120stored in the storage device302and play back the fundus video data, for example. The fundus video data100not subjected to involuntary eye movement elimination may be 2-dimensional video data or 3-dimensional video data.

FIG.8is a descriptive drawing showing a display example 1 of the fundus video data. A display screen800includes a video data display region801, a patient information display region802, and a parameter information display region803. The video data display region801displays the fundus video data100and120. Thus, the user can view while comparing the fundus video data100, which is live footage, and the fundus video data120, which was subjected to involuntary eye movement elimination and video emphasis.

The patient information display region802displays patient information. The patient information is information identifying the patient such as the personal name, the address, and the like. The patient is a person having the subject eye being imaged as the fundus video data100.

The parameter information display region803displays parameter information. The parameter information includes an emphasis parameter and biological monitoring information, for example. The emphasis parameter is a parameter indicating the frequency domain to be emphasized by video emphasis by the emphasis unit403and the degree of emphasis. The biological monitoring information is information from monitoring the body of the patient such as the pulse of the patient.

FIG.9is a descriptive drawing showing a display example 2 of the fundus video data. A display screen900includes a first video data display region901, a second video data display region902, the patient information display region802, and the parameter information display region803.

The first video data display region901displays the fundus video data100prior to involuntary eye movement elimination. The first video data display region901is the main screen and is larger than the second video data display region902. The second video data display region902displays the fundus video data120after involuntary eye movement elimination and video emphasis. The second video data display region902is a subscreen and is smaller than the first video data display region901. As a result, the user can focus on the first video data display region901while viewing the second video data display region902as necessary.

FIG.10is a descriptive drawing showing a display example 3 of the fundus video data. The display screen1000has the first video data display region901that displays the fundus video data100prior to involuntary eye movement elimination and the second video data display region902that displays the fundus video data120after involuntary eye movement elimination and video emphasis, for example. The second video data display region902is displayed so as to be superimposed on the upper right corner of the first video data display region901. As a result, the user can focus on the first video data display region901while viewing the second video data display region902as necessary.

If a changeover switch such as a foot switch is provided to the ophthalmic apparatus201, then the display content of the video data display region801and the second video data display region902may be switched from the fundus video data120to the fundus video data110or from the fundus video data110to the fundus video data120according to the changeover switch being switched ON or OFF. As a result, it is possible to see the difference before and after the video emphasis.

Also, inFIGS.9and10, switching may be performed according to the switching of the changeover switch ON or OFF such that the fundus video data120is displayed in the first video data display region901and the fundus video data100is displayed in the second video data display region902. As a result, it is possible for the user to view the fundus video data120in the larger main screen.

Also, if viewing the fundus video data through an eyepiece of the ophthalmic apparatus201, the fundus video data110and120may respectively be viewable through either one of the lenses of the eyepiece for each eye. For example, in the case ofFIG.10, the fundus video data100is viewed with the right eye and the fundus video data100and120is viewed with the left eye.

Also, the display screens800,900, and1000ofFIGS.8to10may display a mode switching button for switching between video emphasis modes, enabling the user to select between the normal mode in which both movement and color changes are emphasized, color emphasis mode in which only color change is emphasized, and movement emphasis mode in which only movement is emphasized. The fundus video data120in the selected video emphasis mode is displayed in the video data display region801or the second video data display region902.

Also, the display screens800,900, and1000ofFIGS.8to10display live video (fundus video data100prior to involuntary eye movement elimination), but may display the fundus video data120after involuntary eye movement elimination.

In this manner, it is possible to improve flexibility for the user in viewing the video data before and after video emphasis after being subjected to involuntary eye movement elimination.

The present invention is not limited to the content above, and the content above may be freely combined. Also, other aspects considered to be within the scope of the technical concept of the present invention are included in the scope of the present invention.

EXPLANATION OF REFERENCES

100,110,120fundus video data,101macula,102optic disc,103blood vessel,200ophthalmic system,201ophthalmic apparatus,204management server,205terminal,400video processing apparatus,401acquisition unit,402elimination unit,403emphasis unit,404output unit,601first elimination unit,602second elimination unit,603third elimination unit,621separation unit,622time filtering unit,623phase noise elimination unit,624attenuation unit,625reconstruction unit, F,G,H fundus image frame