Predicting Ophthalmologist Gaze Patterns on OCT Data with Masked Autoencoders and Long Short-Term Memory Networks

Authors: tri  le
kuang  sung
kaveri a thakoor
Venue: IEEE_org_EMBS_BHI_2024_Conference
Published: 2024-07-05 19:57:09

Abstract: Scanpath prediction is crucial in the medical domain as it captures the visual attention patterns of experienced clinicians, offering insights into diagnostic processes and enhancing training programs. Understanding where experts focus can lead to improved medical imaging interpretation and decision-making. However, scanpath prediction is extremely challenging due to the inherent noise in eye-tracking data, individual variability among clinicians, and the complexity of medical images.

This work introduces a pioneering adaptation of the “Show, Attend and Tell” (SAT) framework to analyze the gaze patterns of ophthalmologists on Optical Coherence Tomography (OCT) reports. Instead of using Convolutional Neural Networks (CNNs) for visual feature extraction, we integrate self-supervised learning through a Masked Autoencoder (MAE). The MAE reconstructs masked regions of OCT images, enabling the encoder to generate robust image representations despite limited labeling in medical imaging datasets. We trained separate LSTM models for each clinician to account for individual inspection patterns.

The model demonstrated strong evaluation results, with the best-performing model achieving a ScanMatch score up to 0.5595 and Pearson correlation of up to 0.866 in predicting expert gaze on OCT reports. We showcase a downstream use-case of predicting the sequence of expert-fixated regions on an OCT report and visualizing these for ophthalmic resident education. Our findings highlight the framework’s potential to enhance the understanding and emulation of expert-level diagnostic mechanisms, aiding in the explanation of AI-based predictions in the clinic and guiding novice residents in ophthalmic education, especially in resource-diverse environments with limited access to expert ophthalmologists or labeled datasets.

Main paper: 
 i. introduction: 
 I N the realm of medical imaging, particularly Optical Co- herence Tomography (OCT), the interpretation of complex visual data plays a crucial role in diagnostic processes. Eye tracking data has been a great source of information in medical applications of machine learning including detection [10] , [12] and segmentation [11] . Previous works have successfully integrated clinician's gaze information to inform Vision Transformer [3] based models for both OCTs [7] and X-ray images [8] , enhancing both detection accuracy and computational efficiency. Expert sonographers' gaze during ultrasound image inspection is used for model compression without significant loss in performance [9] . These examples show that eye gaze from seasoned clinicians carries a wealth of information across different types of medical imaging. However, limited work has attempted to predict experienced ophthalmologists' gaze fixation location and order during image inspection for downstream diagnostic and educational purposes. Scanpath prediction, which forecasts the sequence of eye movements during image viewing, has been extensively studied in natural images [16] - [27] . However, its application to medical imaging remains underexplored. To the best of our knowledge, we are the first to develop a method for scanpath prediction in Optical Coherence Tomography (OCT) reports that is also agnostic to different types of medical images.Through years of experience, expert ophthalmologists develop nuanced gaze patterns that guide their image analysis and diagnoses. However, the scarcity of experienced ophthalmologists poses a significant challenge, especially in remote or under-served areas. This project seeks to bridge this gap by leveraging the "Show, Attend, and Tell" [1] framework, traditionally used for image captioning, to predict gaze patterns on OCT reports using gaze data from experienced ophthalmologists. Such a predictive model could serve as a virtual advisor to novice ophthalmologists, guiding their inspection of OCT reports in the absence of attendings or experienced doctors, or as a second opinion/corroboration even for expert doctors in ambiguous cases.A novel aspect of this project is the replacement of the conventional Convolutional Neural Network (CNN) encoder with an encoder derived from a Masked Autoencoder, specifically trained on OCT images. This approach exploits the advantages of self-supervised learning, which does not rely on labeled data. This is particularly beneficial in the medical imaging domain, where acquiring labeled data is not only challenging but also labor-intensive. Through this innovation, we aim to harness extensive, unlabeled OCT datasets, thereby overcoming one of the significant hurdles in medical image analysis. The integration of self-supervised learning into our predictive model promises to enhance the accessibility and accuracy of OCT report interpretations, democratizing expertlevel diagnostic guidance across the medical community.In deep learning, the Masked Autoencoder (MAE) represents a significant step up in the context of self-supervised learning and representation learning in the computer vision domain. Originally conceptualized to address the inefficiencies and data constraints of supervised learning, MAE has rapidly gained traction across various domains, including medical imaging [4] . At its core, the MAE solves the pretext task in the image domain by intentionally obscuring parts of the input data, compelling the model to predict these missing segments based solely on the available information. This methodology not only enhances the model's feature extraction capabilities but also its generalization to unseen data, making it particularly suited for domains where labeled data is scarce or expensive to obtain.In medical imaging, specifically in the analysis of Optical Coherence Tomography (OCT) images, the application of MAE has opened new vistas. The encoder component of MAE, adept at capturing intricate details from partial data, has shown promise in understanding complex visual patterns intrinsic to medical diagnostics. Studies leveraging MAE for natural images highlight its superiority in extracting relevant features without explicit labeling, showcasing its potential to be applied to medical image interpretation, specifically the prediction of expert gaze on OCT images.The "Show, Attend, and Tell" framework [1] , initially introduced for image captioning tasks, has established itself as a cornerstone in computer vision and natural language processing. Its innovative use of attention enables models to focus on specific parts of an image while generating corresponding textual descriptions, mimicking the human ability to correlate visual elements with linguistic annotations. This approach has not only enhanced the interpretability of deep learning models but also has significantly improved their performance by allowing them to prioritize salient features in images.Adapting the "Show, Attend, and Tell" methodology to the domain of scanpath prediction, particularly within medical imaging, marks an effort to translate visual attention mechanisms from textual annotation to the prediction of visual focus areas. In the context of analyzing Optical Coherence Tomography (OCT) reports, this adaptation could change how clinicians engage with medical imagery. By predicting where an experienced ophthalmologist's gaze might linger on an OCT image, this framework offers a tool for training and assisting less experienced clinicians, while also potentially providing corroboration/confirmation for the reasoning behind a diagnosis for experts. This application of "Show, Attend, and Tell" in scanpath prediction underscores the versatility of attention mechanisms and highlights the potential of machine learning to enhance diagnostic practice within ophthalmology.The original "Show, Attend, and Tell" framework utilizes an LSTM to generate a probability distribution over a vocabulary through softmax, framing the problem as one of classification. In this study, we compared two independent approaches to capture the expert gaze patterns that trained the LSTM to be an autoregressive model and a classification model. We used the autoregressive version as a convenient sanity check to make sure that the model could capture the gaze pattern and that the attention network (fully connected layers) used during the training of the LSTM is paying attention to the critical regions of OCT reports. We then used qualitative results of the autoregressive model as a premise to re-frame the training as a classification problem which is the main application of this study.This study also examines the robustness of the SAT framework through the substitution of its conventional CNN encoder with the encoder from a Masked Autoencoder (MAE). While CNNs have been the standard for extracting image representations, they are inherently suited for classification tasks that require labeled data. In the SAT framework, obtaining image embeddings is an intermediary step that does not necessarily rely on labels, making the MAE, which can learn from unlabeled data, an apt choice for medical imaging contexts where labels may be scarce. This shift not only challenges the traditional reliance on CNNs but also aligns with the evolving needs of medical image analysis to learn from extensive, unlabeled datasets.More about model training is explained in the sub-section III-B.In this study, two types of internal datasets were utilized, both centred around Optical Coherence Tomography (OCT) reports. The first dataset is comprised of approximately 13,000 OCT reports (6,000 labeled as glaucomatous vs. not glaucomatous and 7000 labeled as acceptable vs. unacceptable, with some overlap with the 6000-dataset and including poor scans), which were used to conduct self-supervised learning on the Masked Autoencoder (MAE). The second dataset includes gaze patterns from seven experienced ophthalmologists, collected while each expert examined 20 OCT reports. Each fixation in each pattern is in normalized coordinates with respect to computer screen resolution. The 20 OCT reports (per clinician) were randomly sub-selected from a larger pool of OCT reports collected between 2010 and 2023 at our institution (through an IRB-approved study: Protocol AAAU4079 approved by the Columbia University Irving Medical Center Institutional Review Board). A different set of 20 reports was The second image is the randomly masked version of the OCT report that is used as input during MAE's training. The third image is the reconstruction result of both visible and masked patches. The fourth image is the combination between the visible masks and reconstructed patches that were masked. used for each clinician; however, due to small dataset size, there was some overlap in reports viewed by clinicians; on average, each report was viewed by 3 clinicians. For each LSTM model corresponding to a clinician, we trained on 15 gaze patterns and tested on 5 gaze patterns. We also performed 4-fold cross-validation on each LSTM to retrieve the 95% confidence interval to ensure statistical significance.Fig. 1 illustrates the real-time fixation prediction pipeline. In the inference stage, an OCT report is initially processed by the MAE encoder to extract the image representation. This representation, along with the LSTM's hidden state, is then fed into the attention network, generating an attention heatmap. Utilizing this heatmap and gaze information at each timestep, the model predicts the coordinates of the next gaze. The study encompasses two primary training phases, outlined below.The MAE is trained on 13,000 OCT reports to learn data representation. Initially, it randomly masks 75% of the patches from an OCT report. The remaining visible patches are processed through 12 Vision Transformer (ViT) [3] blocks during the encoding phase. The encoder's output, or image embedding, combined with mask tokens (place holders for masked patches at the beginning), is input into a decoder comprising 8 ViT blocks. The decoder's output reconstructs the masked OCT report, as depicted in Fig. 2 . A more powerful or deeper decoder is unnecessary for the MAE, as we only employ its encoder for our primary application. During inference, unlike the autoregressive LSTM where error propagation is inherent, we used beam search to significantly remedy this issue which yields a more globally optimal gaze pattern.3) Beam Search: Beam search is a heuristic search algorithm used in natural language processing for decoding sequences. It explores multiple possible outputs by maintaining a fixed number of best partial hypotheses (beams) at each step, enhancing the likelihood of generating the most probable sequence in tasks like machine translation.In gaze pattern prediction using LSTM, beam search mitigates error propagation by evaluating multiple gaze paths simultaneously. This approach retains the most promising trajectories, preventing early mistakes from significantly influencing future predictions, thereby enhancing the accuracy and reliability of predicting gaze sequences over time.4) Consideration and Simplification: During the training of the MAE, OCT reports were resized to a square aspect ratio. Gaze coordinates were not affected by this change since they were originally normalized based on the image resolution during the data collection phase. During training of the LSTM on gaze data, we only used the MAE's encoder to extract OCT report embeddings. Since the MAE's decoder is used to reconstruct masked images, it cannot be utilized in this application.We conducted both quantitative and qualitative evaluations of the model's predictions to ensure its ability to learn and replicate the gaze patterns of clinicians. To measure the spatial alignment and consistency between the model's output fixations with the regions deemed critical by clinicians during their inspections, we used the Pearson Correlation on saliency maps [14] . To assess the similarity between the predicted and actual scanpaths, various methods are available [15] . We opted for ScanMatch [13] , which effectively captures both the positional and temporal aspects of scanpaths [15] .1) Saliency maps consistency with Pearson Correlation: Previous work [5] used Pearson correlation coefficients to quantitatively evaluate the consistency of the regions of interest (ROIs) identified during fundus image inspections for glaucoma diagnosis by different ophthalmologists. They computed these coefficients between Gaussian-filtered fixation maps, with each map representing the gaze pattern of an individual ophthalmologist. We took a similar approach with a slight modification to evaluate how well the model performs in learning to capture individual clinician gaze patterns and their focused regions during the inspection. Detailed steps are explained below.The smoothed grid S is retrieved by applying the Gaussian filter with a standard deviation set to one on the original binary grid cell I. This is done on both predicted and ground-truth gaze patterns.EQUATIONAfter smoothing, the smoothed grid is converted into a vector format: s = [SThe Pearson correlation coefficient between the flattened ground truth and prediction grids is calculated as:EQUATIONwhere n = 49 (since the grid is 7x7), s gt is the flattened ground truth grid, and ŝ is the flattened prediction grid.2) Scanpaths similarity with ScanMatch: ScanMatch is a method designed to assess the similarity between scanpaths, utilizing the Needleman-Wunsch algorithm, originally developed for DNA sequence comparison in bioinformatics [13] . This approach involves spatially and temporally binning saccadic eye movement sequences, which are then recoded into sequences of letters that encapsulate information about fixation locations and order. The method evaluates the similarity between these sequences by maximizing a similarity score derived from a substitution matrix. This matrix scores all possible letter pair substitutions and incorporates a penalty for gaps. Scanpath prediction models widely adopt ScanMatch for evaluation due to its robustness to inherent noise in saccadic eye movements [18] .Given OCT-report patches predicted by our algorithm, we developed a user interface (UI) using Unity, a versatile crossplatform game engine, to present a sequence of bounding boxes on OCT reports. The model's prediction, a sequence of coordinates on a 7x7 grid, is translated into corresponding bounding boxes and overlaid on top of the OCT reports. The UI facilitates the visualization of annotated regions of interest, allowing users to interactively explore and analyze spatial and temporal relationships between regions of interest (ROIs) within OCT data. Fig. 2 shows the reconstruction result of MAE trained on OCT reports. The training of the Masked Autoencoder (MAE) on 13,000 Optical Coherence Tomography (OCT) reports yielded qualitatively impressive results demonstrating its transferability from natural images to medical image applications. This further shows its potential as a viable encoder within the "Show, Attend, and Tell" framework. By effectively learning the data representation, the MAE showcases its capability to capture the intricate details and nuances present in OCT images. This proficiency not only enhances the model's understanding of the visual content but also significantly improves the subsequent attention-driven processes. Fig. 3 displays the model's gaze pattern predictions for two distinct reports. On closer inspection of the ophthalmologist's gaze sequence (3a), one can observe a pattern where the inspection begins at the report's center and proceeds in an anticlockwise direction. The model successfully captures these patterns, as depicted to the right (3b). Intriguingly, the model appears to have learned a regression function tailored to each report's gaze points, resulting in a series of discrete points that follow a smooth, complex curve. While these results affirm the method's effectiveness and provide insights into an expert clinician's gaze pattern, they exhibit lower variance compared to the actual collected gaze data. The model's predictions do not isolate specific regions within the OCT report that could potentially assist less experienced ophthalmologists.In Fig. 4 , the attention mechanism's role in predicting gaze patterns for report 3a and its associated gaze pattern 3b is evident. Notably, the LSTM's attention network identifies key areas (indicated by brighter regions) in the OCT report, such as the circumpapillary retinal nerve fiber layer (RNFL) scan-visible as a grayscale rectangular image in the report's upper left. This validates the effectiveness of the MAE in supplanting traditional CNNs within the conventional SAT framework. Whereas, in the original SAT framework for image captioning, the attention is sharply focused-for instance, on a balloon when predicting the word "balloon"-the adapted framework for gaze prediction utilizes all relevant areas on the OCT image to anticipate the next gaze point.Table I shows promising results on seven different seasoned ophthalmologists. The Pearson correlation calculated on the hold-out test set shows a strong correlation between prediction and ground-truth gaze patterns. Moreover, beam search serves a critical role which helps retrieve the more globally optimal gaze pattern. This could be qualitatively visualized in Fig. 5 .Compared to the autoregressive version of the model, more variance in the pattern is introduced in the LSTM trained as a classification problem. The predicted pattern post-processed by beam search showed a clear pattern that resembles the ground truth gaze pattern of the experienced clinician.2) Scanpaths similarity with ScanMatch: [18] , [27] .error propagation during LSTM inference and producing scanpath patterns that more closely resemble those of clinicians.As we are the first to develop a model for scanpath prediction on medical images, particularly for ophthalmic OCT reports, there are no standardized metrics to perform direct benchmarking on this data. However, the ScanMatch score is used for benchmarking scanpath prediction on natural images and accounts for both spatial and temporal aspects of gaze. We therefore compiled Table III with results from previous works on scanpath predictions on different tasks and datasets of natural images. Although the comparison does not fully establish the superiority of our model over others due to significant differences between natural and medical images, it demonstrates our model's strong performance and provides a baseline for future research.AiR [28] is a Visual Question Answering dataset consisting of images and questions, as well as eye-tracking data from 20 participants. OSIE [29] is a free-viewing dataset including 700 images with eye-tracking data from 15 participants. COCO-Search18 [25] is a visual search dataset including images annotated with the fixations of people searching for targetobject goals. MASSVIS [30] is a popular dataset covering various types of visualizations which also provides gaze data recorded from human viewers.Our model's mean ScanMatch metric from Table II shows a competitive result (0.4032) compared to other models trained on natural images in Table III . While Chen et al. [18] reaches a ScanMatch score of 0.554 in the COCO-Search18 dataset, we argue that scanpaths of experienced clinicians on OCT reports inherently differ from those of viewers searching for target objects in natural images. Natural objects are usually concrete and well-segmented. This is not the case in medical images (e.g., OCT reports) which may contain ambiguous sub-regions that are critical for making diagnoses.As shown in Fig 6 , we used Unity to develop an interactive UI that overlays a sequence of bounding boxes on OCT reports. Our model provides temporal information in the form of a sequence of bounding boxes based on a 7 by 7 grid. This temporal information is combined with a heatmap generated by a Vision Transformer (ViT) through attention rollout [7] . By intersecting the heatmap with the temporal bounding boxes, we merge the temporal insights from our model with the attention mechanism of the ViT, enhancing the precision and informativeness of the annotated regions. Our results indicate that this approach effectively highlights key areas within the OCT reports, facilitating detailed analysis. Our preliminary user studies also show that the guidance can enhance efficiency of novices during inspection of OCT reports by reducing the variability of their scanpaths without compromising their diagnostic accuracy. This development underscores the potential of combining temporal and attention mechanisms to enhance the interpretability of complex medical data, providing a robust tool for clinical diagnostics and education.We observe that some of the predicted bounding boxes covered less informative white regions. We suspect that this behavior partially stems from fixations located in white space that are part of clinician saccades during data collection. In future work, we plan to perform segmentation on OCT images and use segmentation masks to eliminate fixation points that fall into white space.When clinicians repeatedly check the same region of OCT scans, it indicates the critical nature of the sub-regions. Our current LSTM model struggles to capture this recurrent inspection behavior effectively. To address this in future work, we will synthesize a heatmap for each OCT report of each clinician to indicate the importance of sub-regions based on the frequency of inspections. This heatmap would then serve as a supervised signal, allowing the LSTM to predict fixation coordinates and assign an importance score to each sub-region at every timestep. This enhancement would be especially useful for training novices, directing their attention to areas of significance and enhancing the model's interpretability.This project presents a novel translation of the "Show, Attend, and Tell" (SAT) framework for predicting ophthalmologist gaze patterns on OCT reports, through the innovative use of a Masked Autoencoder (MAE) as the encoder. This adaptation leverages self-supervised learning, allowing for a more nuanced interpretation of complex OCT images without the need for extensive labeled data.Initial testing within the SAT framework, utilizing regression to anticipate gaze patterns, demonstrates the model's proficiency in identifying the sequence of an ophthalmologist's examination. The attention network trained along with the LSTM is also able to identify critical regions in the OCT report. However, the practical application of the autoregressive LSTM is constrained byscanpath predictions which do not adequately reflect the actual variability in human observation. Higher variance in gaze patterns is introduced by training the LSTM as a classification problem. Post-processing of predicted gaze patterns using beam search yields patterns that better resemble the ground truth and achieve quantitatively superior Pearson coefficients and ScanMatch metrics. Our model shows strong correlation between predicted and groundtruth gaze patterns, offering potential for conducting user studies with our Unity-based guidance interface to determine its effectiveness in guiding ophthalmologists during OCT examination. Future directions include integrating the temporal aspect of our work into a Vision Language Model (with clinician speech/dictations) to refer and ground features in medical images. 
 
 Appendix:  



Reviews: review: title: title: Good ideas but need to describe clinical relevance with experiments
paper_summary: paper_summary: The paper introduces a novel approach for predicting ophthalmologist gaze patterns on Optical Coherence Tomography (OCT) reports. The authors adapt the "Show, Attend and Tell" framework, replacing the traditional CNN encoder with a Masked Autoencoder (MAE) trained on OCT images. This allows for better feature extraction from medical images with limited labeled data. The model uses an LSTM to predict gaze sequences, trained both as an autoregressive model and a classification problem on a 7x7 grid. Results show strong correlations between predicted and ground truth gaze patterns, with beam search improving performance. The authors also developed a Unity-based interface to visualize the predicted gaze patterns on OCT reports. This work has potential applications in training novice ophthalmologists and providing second opinions for experts in ambiguous cases.
strength_weakness: strength_weakness: 1. The study uses a relatively small dataset of only 20 gaze patterns per clinician, with 15 used for training and 5 for testing. This small sample size may limit the generalizability of the results. With such a small dataset and a complex model, there's a risk of overfitting. The paper doesn't discuss measures taken to prevent this, such as regularization techniques.

2. The paper doesn't compare their novel approach to existing methods for gaze prediction or medical image analysis, making it difficult to assess the relative improvement offered by this technique.

3. While the authors mention potential applications in training and second opinions, there's limited discussion on how the predicted gaze patterns translate to clinical outcomes or decision-making.

4. The autoregressive version of the model produces low-variance predictions that don't accurately reflect the variability in human gaze patterns. This limitation could reduce its practical usefulness. The paper doesn't provide a detailed analysis of where and why the model makes errors, which could provide insights for future improvements.
questions: questions: 1. Dataset details: How were the 20 OCT reports per clinician selected? Were they chosen randomly or based on specific criteria?
Were the same 20 reports used for all clinicians, or did each clinician examine different reports? How representative are these 20 reports of the typical variety of cases an ophthalmologist might encounter?

2. What was the rationale behind choosing a 7x7 grid for the classification problem? How might different grid sizes affect performance?
How was the LSTM architecture (number of layers, hidden units, etc.) determined?

3. Why was Pearson correlation chosen as the primary evaluation metric? Were other metrics considered? How does the model's performance compare to human-level agreement between different expert ophthalmologists?

4. How well does the model generalize to OCT reports from different machines or institutions?
Can this approach be extended to other types of medical imaging beyond OCT?
title: title: Predicting Ophthalmologist Gaze Patterns on OCT Data with Masked Autoencoders and Long Short-Term Memory Networks
paper_summary: paper_summary: The paper presents a methodology for predicting opthalmologist gaze patterns on OCT data. The method is based on the use of masked autoencoders and LSTM networks. Overall, it looks like the approach worked well.
strength_weakness: strength_weakness: - The reconstructed images do not look very good. However, this does not matter as long as the overall results on the gaze patterns are good.
questions: questions: The authors claim excellent results. However, the correlations can be low. Can you please expand on this point? 

Also, do you think that Pearson correlations capture the need for accuracy in this problem?
title: title: Review of "Predicting Ophthalmologist Gaze Patterns on OCT Data with Masked Autoencoders and Long Short-Term Memory Networks"
paper_summary: paper_summary: This paper presents a novel AI-based approach to predict the gaze patterns of experienced ophthalmologists on optical coherence tomography (OCT) reports. Specifically, it adapts the "show, attend, tell" framework by leveraging a masked auto-encoder for reconstructing regions of interest, and a LSTM model to perform gaze prediction based on the patterns of each clinician. In addition, the paper borrows the use of beam search decoding from natural language processing to filter erroneous gaze pathways and improve the accuracy of the LSTM model in inference.
strength_weakness: strength_weakness: 1. Lack of baselines. The paper mainly focuses on comparing different configurations of its own LSTM model, while it does not compare it with more recent transformer-based autoregressive models. The results would be more compelling if the performance of LSTM is comparable to that of state-of-the-art models.
2. Clarity of figures. Figure 4, for instance, should be zoomed in to better show the differences between each heat map, or show less heat maps. The current display makes them look the same.
questions: questions: What are some advantages of choosing a LSTM-based model over a transformer-based one? Since the encoder already integrates self-supervised learning, why not include it in the decoding as well? Is the choice motivated by data availability, compute, or the clinical demand for interpretability?