<figure><img src="image_1.png" /></figure>

HAL
open science

# L-MAE: Longitudinal masked auto-encoder with time and severity-aware encoding for diabetic retinopathy progression prediction

Rachid Zeghlache, Pierre-Henri Conze, Mostafa El Habib Daho, Yihao Li, Alireza Rezaei, Hugo Le Boité, Ramin Tadayoni, Pascale Massin, Béatrice Cochener, Ikram Brahim, et al.

<figure><img src="image_2.png" /></figure>

## To cite this version:

Rachid Zeghlache, Pierre-Henri Conze, Mostafa El Habib Daho, Yihao Li, Alireza Rezaei, et al. L-MAE: Longitudinal masked auto-encoder with time and severity-aware encoding for diabetic retinopathy progression prediction. Computers in Biology and Medicine, 2024, 185 (February 2025), pp.109508. 10.1016/j.compbiomed.2024.109508. hal-04815207

HAL Id: hal-04815207

https://imt-atlantique.hal.science/hal-04815207v1

Submitted on 12 Aug 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

# L-MAE: Longitudinal masked auto-encoder with time and severity-aware encoding for diabetic retinopathy progression prediction

Rachid Zeghlache a,b,*, Pierre-Henri Conze a,c,†, Mostafa El Habib Daho a,b,†, Yihao Li a,b,
Alireza Rezaei a,b, Hugo Le Boité d,e,†, Ramin Tadayoni f,g, Pascal Massin e,f,
Béatrice Cochener a,b,g, Ikram Brahim a,h, Gwenolé Quellec a,†, Mathieu Lamard a,b

## ARTICLE INFO

**Keywords:**

Diabetic retinopathy

Self-supervised learning

Longitudinal analysis

Pretext task

Disease progression

## ABSTRACT

Pre-training strategies based on self-supervised learning (SSL) have demonstrated success as pretext tasks for downstream tasks in computer vision. However, while SSL methods are often domain-agnostic, their direct application to medical imaging is challenging due to the distinct nature of medical images, including specific anatomical and temporal patterns relevant to disease progression. Additionally, traditional SSL pretext tasks often lack the contextual knowledge that is essential for clinical decision support. In this paper, we developed a longitudinal masked auto-encoder (MAE) that builds on the Transformer-based MAE architecture, specifically introducing a time-aware position embedding and a disease progression-aware masking strategy. Unlike traditional sequential approaches, our method incorporates the actual time intervals between examinations, allowing for better capture of temporal trends. Furthermore, the masking strategy evolves in alignment with disease progression during follow-up exams to capture pathological changes, improving disease progression assessments. Using the OPHDIAT dataset, a large-scale longitudinal screening dataset for diabetic retinopathy (DR), we evaluated our pre-trained model by predicting the severity level at the next visit within three years, based on past examination series. Our findings demonstrate that both the time-aware position embedding and the disease progression-informed masking significantly enhance predictive accuracy. Compared to conventional baseline models and standard longitudinal Transformers, these simple yet effective adaptations substantially improve the predictive power of deep classification models in this domain.

## 1. Introduction

Diabetic retinopathy (DR), a leading cause of vision loss world-
wide, affects millions of people with diabetes [1]. DR is a common
condition in individuals with diabetes and is a significant threat to
vision worldwide. According to the International Diabetes Federation,
DR will affect more than 700 million people by 2045 [2]. DR occurs
when retinal blood vessels are damaged by long-term high blood sugar
levels, leading to leakage and swelling. Detecting DR early through
retinal imaging, especially by identifying red and bright lesions, is
crucial for preventing vision loss. Color fundus photographs (CFP) are
commonly used to assess the structure of the retina and identify lesions

such as microaneurysms, hemorrhages, and exudates. The International
Clinical DR (ICRD) severity scale divides DR into five grades, ranging
from no DR (grade 0) to severe proliferative DR (grade 4). Timely
interventions can effectively manage the early-to-middle stages of DR
(grades 1, 2, 3), which are characterized by progressive damage to the
small blood vessels and blockages. Proliferative DR (grade 4) carries
a high risk of vision loss due to significant hemorrhages (Fig. 1).
Detecting DR early and providing appropriate treatment, especially
during the mild to moderate stages of non-proliferative DR (NPDR), can
significantly delay the progression of DR and prevent vision problems

<footnote>a LaTIM UMR 1101, Inserm, Brest, France</footnote>

<footnote>b University of Western Brittany, Brest, France</footnote>

<footnote>c IMT Atlantique, Brest, France</footnote>

<footnote>d Sorbonne University, Paris, France</footnote>

<footnote>e Ophthalmology Department, Lariboisière Hospital, AP-HP, Paris, France</footnote>

<footnote>f Paris Cité University, Paris, France</footnote>

<footnote>g Ophthalmology Department, CHRU Brest, Brest, France</footnote>

<footnote>h INSERM U1227 Lymphocytes B et Autoimmunite (LBAI), Brest, France</footnote>

<footnote>* Corresponding author at: University of Western Brittany, Brest, France.
E-mail address: rachid.zeghlache@univ-brest.fr (R. Zeghlache).</footnote>
<figure><img src="image_6.png" /><figcaption>**Fig. 1.** Diabetic retinopathy progression stages. The number and type of anomalies increase with severity. As severity increases, we observe more anomalies in peripheral zones.</figcaption></figure>

related to diabetes. The existing guidelines for DR screening suggest that everyone with diabetes should undergo annual exams, regardless of their specific risk factors. However, this procedure could result in missed diagnoses because it does not take into account the variation in individual risk. Utilizing artificial intelligence (AI) shows potential for enhancing DR screening as it allows the creation of models that can detect [3–5] or predict an individual’s likelihood of developing or advancing DR [6,7].

Despite the potential for AI to bring a revolutionary change in DR screening, there are still several important areas of research that need to be addressed. One such area is the limited scope of existing AI models, which mainly concentrate on predicting the risk of DR within a two-year timeframe based solely on a single color fundus photograph [8–10]. However, DR is a chronic disease that can progress gradually over time, emphasizing the need for models that can predict risk over extended periods, as proposed in [11]. This paper presents a methodology where a pre-training step on a large dataset is combined with a deep survival analysis model. The goal is to predict the progression of DR over a period of 1 to 5 years using only a single CFP.

Additionally, previous methods have focused solely on single-color fundus photography and ignored previous examination information in their deep learning process. The reliance on a single-CFP and the omission of prior examination data in DR prediction raise several specific concerns. Relying solely on a single CFP can limit the field of view, potentially missing early signs of DR that may be more apparent across a broader retinal area. Additionally, ignoring previous examination history means that important information on disease progression is lost, which could lead to less accurate risk assessments. Such limitations might result in inadequate screening intervals and insufficient personalization of care. Integrating both CFP and longitudinal examination data into DR risk prediction models could provide a more comprehensive view of disease dynamics, enhancing the accuracy and reliability of predictions for individualized DR screening.

Recent research suggests that self-supervised learning (SSL) can learn efficient representations by training on pretext tasks [12] before resolving downstream supervised tasks (e.g., classification tasks) and may tackle the challenge of limited label availability.

Current SSL models primarily employ contrastive learning to develop representations by training models to distinguish between similar and dissimilar samples. In this approach, previous studies have leveraged sample pairs, classified as positive (similar) and negative (dissimilar), generated through data augmentation techniques [13] or organized within lookup dictionaries [14]. However, direct application of standard SSL methods to medical imaging is challenging due to the substantial differences between medical and natural images. Medical images diverge from natural images across several dimensions,

including content, data format, contrast, noise levels, dimensionality, semantic relevance, and practical applications. Medical images are crucial for identifying diseases, monitoring treatment responses, and supporting surgical planning and guidance. Additionally, medical images often exhibit significant noise and artifacts, whereas natural images are typically less affected by such distortions. Importantly, the semantic content of medical images is more precise and clinically relevant, while natural images may possess more ambiguous or generalized semantic interpretations. Another major challenge in applying SSL to medical imaging is the absence of contextual information crucial for computer-aided clinical decision support. To address this, recent studies have attempted to incorporate context into SSL frameworks specifically designed for medical applications, such as DR detection. For instance, Huang et al. [15] employed the SimCLR framework [13] on sub-regions of color fundus photographs, focusing on areas containing pathological tissue to enhance the context in representation learning. Similarly, Zeghlache et al. [16] explored longitudinal pretext tasks to capture temporal changes between consecutive image pairs, thus enriching the contextual understanding in learned representations. By integrating contextual relevance into pretext task design, methodologies such as those developed by Zeghlache et al. [16] and Huang et al. [15] achieve enhanced representations, better aligning with the demands of target classification tasks in clinical contexts. These advances suggest that incorporating domain-specific context into SSL frameworks is a promising strategy for improving representation learning in medical imaging.

Transformers [17] are overpowering the old convolutional neural network (CNN) paradigm in medical imaging [18]. In particular, masked auto-encoder (MAE), a Transformer-based method [19], is another type of promising self-supervised learning approach. Introduced in the context of natural language processing (NLP) with BERT [19], it aims to reconstruct a masked version of its input using an auto-encoder. MAE are well-regarded for their ability to capture global context and semantic relationships through self-supervised learning, particularly by predicting masked portions of input data. Unlike contrastive learning methods that prioritize representational learning and perform well in linear evaluation setups, MAEs generally require fine-tuning to achieve optimal performance on downstream tasks as presented in [20]. These characteristics make MAEs advantageous for pre-training deep learning models and improving their execution in machine learning applications. Later, the same idea was extended to images in [20] with vision Transformers (ViT) encoder [21]. For action recognition from videos [22–25], different extensions of the MAE framework applied to spatial-temporal data have been proposed [26].

In this particular situation, the author intends to fully leverage an extensive dataset of retinal images through the application of SSL as outlined in Zhou et al.’s work on foundation models [27]. Specifically, the proposed model, RETFound, which is based on the MAE approach, demonstrates superior performance in disease detection and prognosis tasks compared to conventional models. The unique approach of RETFound involves utilizing SSL on a vast dataset of unlabeled retinal images to extract representations that can be applied in various contexts. Foundation models are specifically designed to acquire general-purpose representations of data. These models are trained on extensive amounts of unlabeled data and can be employed to address a wide range of subsequent tasks. When there is limited labeled data available, foundation models can still be effective because they are pre-trained on extensive datasets, allowing them to learn broadly applicable features that can generalize across different tasks [28]. In contrast, specialized pretext tasks are designed to focus on learning representations that are tailored to a specific task. These specialized tasks are generally trained on more specific datasets, and they become particularly effective when a large amount of labeled data is available to refine their task-specific representations. In this paper, we choose to utilize a specialized pretext task to extract features that are specifically relevant to the progression of DR. This decision is motivated by the fact
that specialized pretext tasks can be more efficient to train compared to foundation models.

Since their introduction, one approach to enhance the capacity of masked autoencoders (MAE) has involved designing specific masking strategies, using either handcrafted methods or strategies learned during training [29,30]. In the early stages of DR, ophthalmologists prioritize the central region of the CFP to detect lesions such as microaneurysms, whereas in more advanced stages, the peripheral areas become increasingly important for DR severity assessment [31]. At these later stages, while the center of the image remains relevant, peripheral lesions provide more discriminatory information for DR severity classification [32]. In cases of severe DR, lesions are typically present across all four quadrants. Furthermore, the size of DR-induced lesions varies depending on the disease stage [33] (Fig. 1). Based on these observations, we developed a masking strategy that selectively masks regions according to the severity progression. This approach is designed to guide the model’s attention to regions that are most likely to contain diagnostic information for each stage of DR severity. Transformers utilize positional encoding to integrate sequence order information. However, standard positional encoding methods can only represent simple sequential orders and lack a time-dependent context between consecutive examinations—an important factor in understanding disease progression. The frequency of examinations, for example, provides critical information on the rate of disease advancement. Time-aware models address this limitation by incorporating time intervals directly into their structure, an approach first introduced with the time-LSTM (T-LSTM) [34]. Using time-aware models to analyze disease progression with irregularly sampled time-series images has shown significant benefits. Zeghlache et al. [35] demonstrated that time-aware LSTM models outperform standard LSTM models in predicting DR severity based on sequences of retinal images over time. Building on this, we propose a simple yet effective modification of the ViVit transformer architecture, introducing a Time-ViVit model by incorporating a time-aware positional encoding along the temporal dimension after the embeddings layer. This adaptation enables the model to account for temporal dependencies, enhancing its capability to capture disease progression trends over time.

In summary, our contributions are as follows:

1. We introduce a longitudinal masked auto-encoder (L-MAE) to overcome the aforementioned limitations while still getting benefits from the MAE framework. More specifically, we add a time-aware embedding to make the embedding aware not only of the sequence order but also of its duration and relative position in time.

2. We also integrate a new masking strategy to exploit and inject domain knowledge with the expectation of producing a better representation for longitudinal downstream tasks.

3. We obtain superior performance on a longitudinal DR progression task when using our pre-trained weights compared to baseline methods.

4. To the best of our knowledge, this work is the first to use a L-MAE with an emphasis on label-aware masking and time-aware embedding for disease progression analysis.

## 2. Materials and methods

This section details the proposed method, outlining its components and the dataset used for the experiments.

### 2.1. Temporal position embedding

Sequential models, like Recurrent Neural Networks (RNN) or Transformers, do not consider the time span between events and thus capture sequential signals rather than temporal patterns. By construction,

<figure><img src="image_7.png" /><figcaption>Fig. 2. An example of position encoding in the time dimension is provided. The conventional approach to position encoding only encodes the order of positions, disregarding the time intervals between elements. Our innovative masking strategies, which are sensitive to disease progression dynamics, incorporate time-dependent information to allow the model to capture the dynamics of disease progression.</figcaption></figure>

recurrent networks structure their inputs by sequences. However, self-attention recognizes event orderings via positional encoding. The positional encoding (PE) was introduced in [17] for Transformers. The PE is a vector of $d$ dimensions that provide a relative position about a certain element in the sequence. Let $k$ be the desired position in the desired input sequence, $p_k$ its related encoding in $\mathbb{R}^d$, and $d$ the encoding dimension. Let $f: \mathbb{N} \to \mathbb{R}^d$ be the mapping that constructs the output vector $p_k$, as follows:

$$p_k^{(i)} = f(k)^{(i)} = \begin{cases} \sin(\omega_l \cdot k), & \text{if } i = 2l \\ \cos(\omega_l \cdot k), & \text{if } i = 2l + 1 \end{cases} \quad (1)$$

This vector is then added to all tokens in the sequence, depending on their relative position $k$ and the dimension of the vector $i$. The value of $\omega_l$ is $\frac{1}{10000^{2l/d}}$.

In order to induce temporal knowledge into sequence tokens and inspired by Lin et al. [36] and Li et al. [37], we present a functional feature $f: \mathbb{R}^+ \to \mathbb{R}^d$ map that embeds time spans into high-dimensional spaces. Named time-aware PE, it is defined with the following expression:

$$p_{\text{time-aware}}^{(i)} = f(t_i)^{(i)} := \cos(\omega_k \cdot (t_i - t_0) + \tau) \quad (2)$$

with $\omega_k$ and $\tau$ are the learnable parameters of the time-aware embedding, with $t_0$ as the starting date of the sequence of examination and $t_i$ represent the absolute time (in years in our case) for sequence tokens. A sequence is composed of multiple tokens. For a given sequence, we add this temporal encoding embedding to all tokens that share the same absolute time for a given sequence. Fig. 2 highlights the difference between the two position encodings. Non-integer values were handled consistently by normalizing time intervals to a scale of $2 \times 365.25$ days, representing an average two years length to account for leap years.

### 2.2. Vision transformer

The traditional Transformer model works with a sequence of token embeddings in one dimension. In order to handle two-dimensional images, the input image $x \in \mathbb{R}^{H \times W \times C}$ is reshaped into a sequence of flattened two-dimensional patches $x_p \in \mathbb{R}^{N \times (P^2 \cdot C)}$. Here, $(H, W)$ represents the spatial dimensions of the original image, $C$ represents the number of channels, $(P, P)$ represents the spatial resolution of each image patch, and $N = HW/P^2$ represents the resulting number of patches. This number of patches also serves as the effective input
<figure><img src="image_8.png" /><figcaption>Fig. 3. Illustration of our proposed progression-aware masking strategies.</figcaption></figure>

sequence length for the Transformer. The Transformer uses a fixed latent vector size *D* for all its layers. Therefore, the patches are flattened and transformed into *D* dimensions using a trainable linear projection (Eq. (3)).

As performed in BERT [19], a trainable embedding is added to the sequence of embedded patches ($z_0^0 = x_{\text{class}}$), whose state at the output of the Transformer encoder ($z_L^0$) serves as the image representation (Eq. (5)).

A classification head is attached to $z_L^0$ to perform the prediction. This head is implemented by a multi-layer perceptron (MLP) with one hidden layer. Position embeddings (Eq. (1)) are added to the patch embeddings to retain positional information. The Transformer encoder [17] consists of alternating layers of Multiheaded Self-Attention (MSA) and MLP blocks (Eqs. (4), (5)). Layer normalization (LN) is applied before every block, and residual connections after every block. The MLP contains two layers with a Gaussian Error Linear Unit (GELU) non-linearity. A schematic representation of the Transformer architecture can be seen in Fig. 4.

$$z_0 = [x_{\text{class}}; x_p^1 E; x_p^2 E; \dots; x_p^N E] + E_{\text{pos}}, \quad (3)$$

with $E \in \mathbb{R}^{(P^2 \cdot C) \times D}$ and $E_{\text{pos}} \in \mathbb{R}^{(N+1) \times D}$.

$$z'_{\ell} = \text{MSA}(\text{LN}(z_{\ell-1})) + z_{\ell-1}, \quad \ell = 1 \dots L \quad (4)$$

$$z_{\ell} = MLP(\text{LN}(z'_{\ell})) + z'_{\ell}, \quad \ell = 1 \dots L \quad (5)$$

## 2.3. Progression-aware masking strategies

Our proposed masking strategies are based on the computation of a spatial Gaussian kernel ($a_{ij}$) and random uniform noise ($b_{ij}$). The circular form is produced by the spatial Gaussian kernel, while the random noise is present to propagate the mask aligned with the increase of severity (Fig. 3). Firstly, based on the size of the image, we note $L = H = W$, and we create a grid based on the $q = \sqrt{L}$. We then define a point $c = (c_x, c_y)$ as the center of this grid size that will be the center of the 2D Gaussian. When creating to increase the diversity of the masking strategy, an offset of $\pm 1$ in each direction is randomly applied to $c$.

$$a_{ij} = \exp\left(\frac{-\pi}{rq^2} \frac{(i - c_x)^2 - (j - c_y)^2}{2}\right) \quad (6)$$

$$b_{ij} \stackrel{iid}{\sim} U[0, 1], \text{ where } i, j \in [1:q] \quad (7)$$

To create our mask, we used $a_{ij}$ (Eq. (6)) and $b_{ij}$ (Eq. (7)) and bina- rized them using the mentioned thresholds. With $r \in [0, 1]$ defining the intensity of the Gaussian kernel. The Gaussian spatial kernel smoothly

goes down when the pixel distance becomes larger with the defined center $c$. In our experiments, the value of $r$ is 0.75. To mask more regions and propagate the mask according to the severity progression, we define the following thresholds: $T_{\text{severity}} = 1 - 0.1 \times s_{DR}$ with $s_{DR} \in [0, 1, 2, 3, 4]$ referring to the DR grades. The first mask is defined with $m_{a_{ij}} = (a_{ij} < r)$ and the second mask is defined by $m_{b_{ij}} = (b_{ij} < T_{\text{severity}})$. Finally, we obtain our proposed mask by multiplying the two masks. Fig. 3 illustrates the masking scheme based on the value of $r$ and $T_{\text{severity}}$.

$$m_{ij} = m_{a_{ij}} m_{b_{ij}} \quad (8)$$

## 2.4. Vision Transformer for video (ViViT)

The Vision Transformer for Video (ViViT) extends the ViT framework to handle spatiotemporal data by embedding video frames into tokens and processing them with a Transformer architecture. The primary components of ViViT include patch embedding, multi-headed self-attention, and Transformer encoder layers, which are detailed below.

### 2.4.1. Patch embedding mechanism

The PatchEmbed class converts a sequence of video frames into a lower-dimensional embedded representation. This is achieved by dividing the frames into smaller, non-overlapping patches, and then embedding each patch into a higher-dimensional space. The detailed formulation of this process is as follows: Let $X \in \mathbb{R}^{B \times C \times T \times H \times W}$ be the input video sequence, where $B$ is the batch size, $C$ is the number of input channels (e.g., 3 for RGB), $T$ is the number of frames, and $H$ and $W$ are the height and width of each frame, respectively.

$$\text{num\_patches} = \left(\frac{H}{P_h}\right) \times \left(\frac{W}{P_w}\right) \times \left(\frac{T}{P_t}\right) \quad (9)$$

The input sequence is then divided into patches of size ($P_t, P_h, P_w$), resulting in an intermediate tensor $X' \in \mathbb{R}^{B \times C \times T \times H \times W}$. A 3D convolutional layer, parameterized by a kernel of size ($P_t, P_h, P_w$), is applied to $X'$ to project each patch into the embedding space $\mathbb{R}^D$:

$$E = \text{Conv3d}(X', W) \in \mathbb{R}^{B \times D \times \frac{T}{P_t} \times \frac{H}{P_h} \times \frac{W}{P_w}} \quad (10)$$

where $W$ denotes the weights of the convolutional layer. The resulting tensor $E$ is then flattened and permuted to yield the final patch embeddings:

$$Z = \text{Flatten}(E) \in \mathbb{R}^{B \times \frac{T}{P_t} \times \left(\frac{H}{P_h} \times \frac{W}{P_w}\right) \times D} \quad (11)$$

$$Z = \text{Permute}(Z, (0, 2, 1, 3)) \in \mathbb{R}^{B \times \frac{T}{P_t} \times \frac{H \cdot W}{P_h \cdot P_w} \times D} \quad (12)$$

This final tensor $Z$, containing the embedded patches, is now suitable for subsequent processing by transformer models or other architectures designed to handle sequential data. Positional encodings are added to the embedded tokens to retain information about the original positions of the patches. These encodings are learnable parameters:

$$z_{i,j,k} = z_{i,j,k} + \mathbf{p}_{i,j,k}, \quad (13)$$

where $\mathbf{p}_{i,j,k} \in \mathbb{R}^D$ is the positional encoding for the $(i, j, k)$th patch.

### 2.4.2. Temporal and spatial consistency

To ensure the temporal and spatial consistency of the patches, the method verifies that the input dimensions are divisible by the patch sizes:

$$H \equiv 0 \pmod{P_h} \quad (14)$$

$$W \equiv 0 \pmod{P_w} \quad (15)$$

$$T \equiv 0 \pmod{P_t} \quad (16)$$
<figure><img src="image_9.png" /><figcaption>Fig. 4. Illustration of our longitudinal masked auto-encoder. Our proposed longitudinal masked autoencoder differs from the classic Video-MAE from two perspectives. The first one is the fact that, in the embedding layers, we add our proposed time-aware position embedding to the classical position embeddings used in the transformer. The second difference lies in the masking strategies.</figcaption></figure>

These conditions guarantee that the input frames can be evenly divided into patches without any residuals, thereby preserving the integrity of the spatiotemporal structure during the embedding process. Thus, the embedding layer provides a robust and efficient mechanism for transforming video sequences into a structured format.

### 2.4.3. Multi-headed self-attention (MSA)

Each transformer layer includes a multi-headed self-attention mechanism and a feed-forward network. The self-attention mechanism computes pairwise interactions between all tokens. Given an input sequence of tokens $Z \in \mathbb{R}^{N \times D}$, where N is the number of tokens, the attention mechanism is defined as:

$$ \text{Attention}(Q, K, V) = \text{softmax} \left( \frac{QK^T}{\sqrt{d_k}} \right) V, \qquad (17) $$

where $Q = ZW_Q$, $K = ZW_K$, and $V = ZW_V$ are the query, key, and value matrices, respectively, and $d_k$ is the dimensionality of the key vectors. The multi-headed self-attention mechanism allows the model to jointly attend to information from different representation subspaces:

$$ \text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \text{head}_2, \dots, \text{head}_h)W_O, \qquad (18) $$

where each head$_i$ = Attention(Q$_i$, K$_i$, V$_i$) represents an independent self-attention mechanism.

### 2.4.4. Transformer encoder layers

Each transformer encoder layer consists of a multi-headed self-attention module followed by a position-wise feed-forward network, which comprises two linear transformations with a ReLU activation in between:

$$ \text{FFN}(x) = \max(0, xW_1 + b_1)W_2 + b_2. \qquad (19) $$

The output of each transformer encoder layer is given by:

$$ Z' = \text{LayerNorm}(Z + \text{MultiHead}(Q, K, V)), \qquad (20) $$

$$ Z'' = \text{LayerNorm}(Z' + \text{FFN}(Z')). \qquad (21) $$

### 2.4.5. Factorized encoder for efficient processing

The spatial encoder processes tokens within each frame independently, while the temporal encoder processes the frame-level representations. Given the spatial representations $H \in \mathbb{R}^{n_t \times d}$, we introduce a class token $h_{cls} \in \mathbb{R}^d$. This class token is prepended to the sequence of spatial representations, resulting in an augmented sequence:

$$ \text{Haug} = [h_{cls}; H] \in \mathbb{R}^{(n_t+1) \times d}. \qquad (22) $$

The temporal encoder then processes this augmented sequence to model interactions across different frames and the class token:

$$ H' = \text{TemporalEncoder}(H_{\text{aug}}). \qquad (23) $$

The updated representation of the class token $h'_{cls}$ encapsulates the aggregated spatiotemporal information from the entire video sequence, serving as a comprehensive summary used for classification.

### 2.5. Time video vision Transformer (T-ViViT)

We propose the Time Video Vision Transformer (T-ViViT), which is built upon the Video Vision Transformer (ViViT) by incorporating temporal embeddings to better capture the dynamics of video sequences. T-Vivit is more versatile and better suited to our problem as it can accommodate variable time steps between images. In our approach, each video frame is first divided into non-overlapping patches, which are then linearly embedded into patch vectors using the patch embeddings described by Eq. (12). Specifically, given a set of images with T frames, each of spatial resolution $H \times W$, and a patch size of $P \times P$, the number of patches per frame is $\frac{H \times W}{P \times P}$. Each patch is flattened and mapped to a vector of dimension D through a learnable linear projection. To capture the temporal dimension, we introduce
<figure><img src="image_10.png" /><figcaption>Fig. 5. Illustration of the ViVit. In the representation, we include our time-aware position encoding instead of a regular position encoding along the temporal dimension.</figcaption></figure>

time embeddings. A temporal embedding $p_{time-aware}^{(i)}$ is assigned to each frame $i$, where $i \in \{1, ..., T\}$. These time embeddings are learnable parameters that encode the position of each frame within the sequence. The final input to the transformer model is a combination of patch embeddings and time embeddings. Formally, for a patch embedding $p_{ij}$ corresponding to the $j$th patch of the $i$th frame, the input embedding $e_{ij}$ is given by:

$$e_{i,j} = p_{ij} + p_{time-aware}^{(i)} \quad (24)$$

where $p_{time-aware}^{(i)}$ is the time embedding for the $i$th frame defined in Eq. (2). These combined embeddings are then processed by a series of transformer layers, which consist of multi-head self-attention mechanisms and feed-forward neural networks. The self-attention mechanism enables the model to attend to different patches across both spatial and temporal dimensions, capturing intricate dependencies in the video data. Each transformer layer applies layer normalization, multi-head self-attention, and position-wise feed-forward networks in a sequential manner. The output from the final transformer layer is then used for downstream tasks. By incorporating time embeddings, the T-ViVIT effectively models the temporal aspect of image sequences, enhancing its capability to understand and predict complex spatiotemporal patterns in image sequence data.

## 2.6. Longitudinal masked auto-encoder (L-MAE)

Our encoder is a basic ViVIT [38] that only operates on the visible set of embedded patches (i.e., not on the masked patches), inspired by standard ViT [21]. An illustration of the ViVIT is presented in Fig. 5. This design significantly reduces computational and memory demands, resulting in a more practical solution.

Our decoder is another standard ViT that processes the combined set of encoded patches and mask tokens. It also incorporates decoder-specific positional embeddings and temporal embeddings. The decoder is constructed to be more compact than the encoder. Even though it processes the entire set, its complexity is less than that of the encoder. As performed in [26], the decoder predicts only the patches in the pixel space. The encoder and decoder are independent of the problem's space-time structure. There is no hierarchy or spacetime factorization. This technique utilizes global self-attention rather than the usual approaches to acquire valuable information from data. Our L-MAE takes a sequence of Time-Series Images (TSI), $S \in T \times R^3 \times H \times W$, as input.

As used for the ViVIT, a 3D convolution layer is used to construct the patch embedding. Following the implementation of He et al. [20], our MAE is a straightforward AE approach that reconstructs the supplied

<figure><img src="image_11.png" /><figcaption>Fig. 6. Distributions of patient demographics, diabetes types, visit intervals, and disease progression.</figcaption></figure>

TSI based on the masked TSI. The decoder of the MAE is fed with the full set of tokens, which includes encoded visible patches and mask tokens (Fig. 3). Each mask token is a shared, learned vector that indicates the presence of a patch that needs to be predicted. We incorporate positional embeddings (for the spatial order) and temporal embeddings into all tokens in this complete set. The loss function is analogous to the one used in a regular MAE:

$$L = \sum_{p \in \chi} \|S(p) - \tilde{S}(p)\|_2^2 \quad (25)$$

where $p$ is the token index, $\chi$ the set of masked tokens (random, full frame masked or progression-aware as in Fig. 3), $S$ the image sequence and $\tilde{S}$ the reconstructed image sequence.

## 2.7. Dataset

The proposed models were trained and evaluated on OPHDIAT [39] a proprietary dataset owned by the Assistance Publique-Hôpitaux de Paris (AP-HP), a comprehensive CFP database collected from the Ophthalmology Diabetes Telemedicine network. This database comprises examinations from 101,383 patients between 2004 and 2017. Out of over 763,848 interpreted CFP images, approximately 673,000 were assigned a DR severity grade, while the remainder were deemed ungradable. The ages of the patients ranged from 9 to 91 years. Each examination included at least two images per eye, and most patients had multiple images with different fields of view for both eyes. For consistency, one image per eye was randomly selected from each examination. The dataset was annotated according to the ICDR severity scale [31].

For our experiments, we selected patients from the OPHDIAT database who had at least four visits, resulting in a subset of 7244 patients and 13,997 sequences. This subset was further divided into training (60%), validation (20%), and test (20%) sets, ensuring no overlap between patients in different sets.

Here are the various distributions within the dataset as shown in Fig. 6:

1. Gender Distribution (Fig. 6.a): This histogram illustrates a nearly equal split between genders, with 6815 females (F) and 7182 males (M).

2. Diabetes Type Distribution (Fig. 6.b): The majority of patients have type 2 diabetes (9320), followed by type 1 diabetes (4415), and a smaller group categorized as 'other' (262).
3. Time Between Visits (Fig. 6.c): This boxplot shows the distribution of time intervals between patient visits, with a median of about 500 days and an interquartile range from roughly 200 to 1000 days. Some intervals are much longer, appearing as outliers.

4. Age at Baseline (Fig. 6.d): The combined KDE plot and histogram present the age distribution of patients at baseline, peaking in the 50 to 60-year range, indicating most patients are middle-aged.

5. Follow-Up Duration (Fig. 6.e): This KDE plot and histogram illustrate the duration from the first visit to the baseline in days, with a peak around 1000 days and a long tail extending up to 3500 days.

6. DR Labels Distribution (Fig. 6.f): Using a log scale, this histogram compares the counts of different DR labels at baseline and within three years. 'No DR' and 'mild DR' categories are much more common, while 'moderate DR', 'severe DR', and proliferative diabetic retinopathy (PDR) are less frequent.

## 2.8. Pre-processing

### 2.8.1. Pre-processing pipeline

This section describes the pre-processing pipeline implemented for retinal images intended for deep learning algorithms. This pipeline aims to transform raw retinal images into a standardized format suitable for deep learning models.

**Resizing and Region of Interest (ROI) extraction.** The pipeline begins by resizing the image to a desired radius around the center of the image. This is achieved by first calculating the average intensity across each row to identify the region containing the retina. Subsequently, the image is scaled proportionally so this region falls within a pre-defined radius.

**Empty row detection.** Empty rows, which may appear in some retinal images due to capturing artifacts, are identified and excluded from further processing. This is accomplished by calculating the sum of pixel intensities across each row and removing rows with intensities significantly lower than the average.

**Intensity normalization.** Intensity normalization is applied to ensure that pixel values across retinal images are standardized, enhancing the model's ability to generalize across varying lighting conditions and camera settings. This process begins by applying a Gaussian blur filter across the image's color channels (BGR channels) to smooth out minor intensity fluctuations and suppress noise. Following this, a constant value is subtracted from each pixel intensity, with an additional weighted constant added to maintain the overall brightness balance. This normalization technique ensures a more uniform intensity distribution within each image, making essential features – such as blood vessels and lesions – more consistently visible across the dataset.

**CLAHE (Contrast Limited Adaptive Histogram Equalization).** For even greater local contrast and improved image quality, Contrast Limited Adaptive Histogram Equalization (CLAHE) can be applied to the pre-processed image. This step specifically targets illumination variations within the image. This is particularly relevant for retinal images, which often exhibit uneven lighting due to factors like depth and shadows. CLAHE effectively normalizes these variations, enhancing the definition of critical features like blood vessels and lesions.

**Mask generation & image masking and padding.** A circular mask is created centered on the image with a radius corresponding to the pre-defined value. If necessary, this mask is then padded with zeros to match the final image size. Subsequently, the mask is eroded by a small kernel to refine the edges and exclude potential noise from the periphery. The pre-processed image is masked using the generated

mask. Pixels outside the masked region are set to a constant value (typically, the mean intensity). Padding with zeros is applied if the image dimensions are smaller than the desired output size. Finally, the masked and padded image is cropped to a fixed size (specified by the output_size parameter) centered on the original retinal region. These pre-processing steps prepare the retinal images for further analysis by deep learning models.

**Data-augmentation.** Concerning the pre-processing inside the dataloader, we use common data augmentation techniques involving the following steps that aim to enhance the variability and robustness of the training dataset for deep learning models. The pipeline includes three primary augmentation techniques applied sequentially. First, a random rotation up to 15 degrees is applied to each image to simulate various orientations of the retina. Second, random horizontal flipping is performed to introduce left-right variations. Lastly, the images undergo random resized cropping, where the crop size is randomly adjusted within a scale range of 96% to 100% and an aspect ratio range of 0.95 to 1.05, followed by resizing to a fixed target size. We independently apply data augmentation to the entire sequence. This combination of augmentations increases the diversity of the training data, helping the model generalize better to unseen images.

## 2.9. Target task

The prediction task was to predict the onset of diabetic retinopathy within three years. We utilize the proposed ViViT model to process the last three frames of a patient's medical history. Conventional screening guidelines, advocating annual retinal examinations for all individuals with diabetes [40], fail to consider individual risk profiles, resulting in unnecessary examinations and potential missed diagnoses. This approach is inefficient as most patients with DR do not exhibit significant disease progression within two years, highlighting the protracted nature of DR development [1]. Moreover, the three-year span was also chosen because it was the interval that contained the largest number of patients with a known development of diabetic retinopathy within a period of time. The input data consists of image sequences $X \in \mathbb{R}^{B \times C \times T \times H \times W}$, where $T = 3$ corresponds to the last three frames of the patient's history. The ViViT model processes the image sequence and extracts a high-dimensional feature representation for each frame. The frame-level representations are then concatenated to form a comprehensive representation of the patient's medical history. The final output token from the temporal encoder is used to predict the presence of diabetic retinopathy. A classification layer is applied to the encoded representation:

$$ \hat{y} = \mathrm{softmax}(\mathbf{W}_c \mathbf{H}_{cls} + \mathbf{b}_c), \qquad (26) $$

where $\mathbf{W}_c$ and $\mathbf{b}_c$ are the weights and bias of the classification layer, respectively, and $\hat{y}$ is the predicted probability distribution over the classes. The model is trained using the cross-entropy loss function.

Which measures the discrepancy between the predicted probability distribution $\hat{y}$ and the true labels $y$: Where $N$ is the number of samples in the batch, $y_i$ is the true label for the $i$th sample, and $\hat{y}_i$ is the predicted probability for the $i$th sample. $y_i$ corresponds to the different stages of advancement of DR within 3 years.

$$ \mathcal{L} = -\frac{1}{N} \sum_{i=1}^{N} y_i \log(\hat{y}_i), \qquad (27) $$

## 2.10. Implementation details

Our Time-ViViT encoder is based on the implementation introduced in [38]. Our encoder and decoder follow the vanilla ViViT architectures presented in the paper [38]. Specifically, the architecture is extended to work with multiple frames based on the ViT-B [21] for simplicity. We experimented with three different image sizes: 224 × 224, 256 × 256,
<caption>**Table 1**</caption>

Comparison of approaches in terms of the different AUC scores for the predict next task, on the test set. We only report the best performance of the presented methods for the different hyper-parameters. Results in bold show the best results.

<table><thead><tr><th>Method</th><th>Masking parameter</th><th>Temporal embedding</th><th>Masking strategies</th><th>Pretrain weights</th><th>Backbone</th><th>AUC Mild+</th><th>AUC Moder+</th><th>AUC Severe+</th><th>PRAUC Severe+</th></tr></thead><tbody><tr><td>2D CNN+pooling</td><td>-</td><td>-</td><td>-</td><td>ImageNet</td><td>ResNet50</td><td>0.539 ± 0.036</td><td>0.593 ± 0.051</td><td>0.672 ± 0.0631</td><td>0.174 ± 0.024</td></tr><tr><td>CNN+LSTM</td><td>-</td><td>-</td><td>-</td><td>ImageNet</td><td>ResNet50</td><td>0.563 ± 0.028</td><td><b>0.620</b> ± 0.054</td><td>0.739 ± 0.065</td><td>0.187 ± 0.045</td></tr><tr><td>CNN+T-LSTM</td><td>-</td><td>-</td><td>-</td><td>ImageNet</td><td>ResNet50</td><td><b>0.598</b> ± 0.066</td><td>0.623 ± 0.056</td><td>0.775 ± 0.068</td><td>0.196 ± 0.021</td></tr><tr><td>ViViT</td><td>-</td><td>-</td><td>-</td><td>-</td><td>ViT</td><td>0.474 ± 0.020</td><td>0.501 ± 0.0500</td><td>0.566 ± 0.075</td><td>0.199 ± 0.008</td></tr><tr><td>Video-Vit</td><td>-</td><td>-</td><td>-</td><td>-</td><td>ViT</td><td>0.507 ± 0.014</td><td>0.510 ± 0.038</td><td>0.567 ± 0.093</td><td>0.210 ± 0.012</td></tr><tr><td>ViVit</td><td>-</td><td>-</td><td>Random</td><td>MAE</td><td>ViT</td><td>0.530 ± 0.014</td><td>0.548 ± 0.031</td><td>0.691 ± 0.077</td><td>0.235 ± 0.009</td></tr><tr><td>ViVit</td><td>-</td><td>-</td><td>Random</td><td>RetFound</td><td>ViT</td><td>0.539 ± 0.013</td><td>0.562 ± 0.018</td><td>0.734 ± 0.077</td><td>0.244 ± 0.010</td></tr><tr><td>ViViT</td><td>r = 0.75</td><td>-</td><td>Prog-aware</td><td>L-MAE</td><td>ViT</td><td>0.574 ± 0.020</td><td>0.605 ± 0.031</td><td>0.743 ± 0.059</td><td>0.263 ± 0.017</td></tr><tr><td>L-ViViT</td><td>r = 0.75</td><td>Time-aware</td><td>Prog.-aware</td><td>L-MAE</td><td>ViT</td><td>0.563 ± 0.025</td><td>0.577 ± 0.028</td><td><b>0.811</b> ± 0.062</td><td>0.250 ± 0.019</td></tr></tbody></table>

and 448 × 448, using patch sizes of 16 × 16 and 32 × 32. For the temporal dimension, we opted for a temporal patch size of 1, as the images in our dataset, although sourced from the same patient, have substantial time intervals between them and differ in field of view. Indeed, while video classification typically benefits from a small time lapse between frames to preserve continuity, our case involves non-contiguous images, making a temporal patch size of 1 more suitable for handling this dissimilarity in temporal alignment. Three different masking ratios (denoted $r_{mask} \in \{25\%,50\%,75\%\}$) were used when the random masking strategy was performed. The different networks were trained for 400 epochs by the AdamW optimizer with a learning rate of 5 × 10⁻³, OneCycleLR as a scheduler, and a weight decay of 10⁻⁵, using an NVIDIA A6000 GPU with the PyTorch framework. For pre-training the L-MAE, we used three different types of PE: an empty encoding equivalent to no position embedding added for the temporal dimension, base encoding equivalent to the PE in Eq. (1) and time-aware embedding based on Eq. (2).

## 2.11. Evaluation of the feature extractor

To assess the effectiveness of the features extracted by the different pre-training strategies, we only used *fine-tuning* on the longitudinal downstream task, which was to predict the severity of the next visit within a 3-year period based on previous examinations. The model was trained using cross-entropy loss and the Area Under the receiver operating characteristic Curve (AUC) was used to evaluate the models with three binary tasks: predicting at least non-proliferative diabetic retinopathy (NPDR) in the next visit (AUC Mild+), at least moderate NPDR in the next visit (AUC Moderate+) and finally at least severe NPDR in the next visit (AUC Severe+). We also use Area Under the Precision-Recall Curve (PR AUC) for the detection of at least severe NPDR (denoted PRAUC severe+). We report the mean and standard deviation for each metric, calculated across repeated experiments with different seed values for the different tables. The fine-tuning scenario consisted of initializing the Time-ViViT (encoder) weights using the weights obtained by training the L-MAE and the time-aware embeddings. These embedding layers were learned during the pretraining. After the encoder, we added an MLP layer to predict the label of the next visit. In the *fine tuning* scenario, the model was trained during 200 epochs, with a learning rate of 0.001 and weight decay of 0.0001.

## 2.12. Methods comparison

We compared our approach to classic and state-of-the-art techniques that process sequence images and the potential use of time. We trained each model using the cross-entropy loss and monitored the model's performance on the validation set by tracking the loss value. The best model was then retained.

**2D CNN+pooling** [41] where each frame was fed to the CNN, thus producing latent vectors. These vectors were concatenated, and a global average pooling was applied on top of the formed representation, then injected in a classification head to predict the label of the next visit.
**CNN+LSTM** [42] where each frame was fed to the CNN and where each frame's latent vector was then sequentially fed to an LSTM layer. The last representation produced by the LSTM was given to the classification head to predict the label of the next visit.

**CNN+T-LSTM** was employed similar to the last method, but instead of the classic LSTM, we used time-LSTM introduced by Baytas et al. [34]. We used the Pytorch implementation of Zeghlache et al. [35] in order to have a model that is coupled with CNN and the use of time.

**Video ViT (ViViT)** [38,43] was used to classify image sequences. In their paper, they proposed three variants. In our paper, we implement the first version of ViVit, which extends ViT to include the capability to model the relationships between all spatiotemporal tokens in pairs.

**Time-ViViT.** We extend the previous model with our time-aware embeddings, motivated by the fact that previous study [36,44] have demonstrated the effectiveness of including time-related embeddings instead of just token position along the temporal axis.

## 3. Results

According to our experiments, we observed that the image size was not important for the Transformer. But instead, what was more discriminant was the number of total tokens (i.e., the number of patches). The opposite was observed for the CNN-based method. The first three rows in Table 1 report the baselines. The next three lines present the Time-ViViT trained from scratch but with different temporal embedding. We also observe that the classical PE is an important component for the Transformer-based approach because the model with empty encoding has poor predictive ability compared to the baseline or Time-ViViT with other embeddings, the same being observed across the different masking strategies.

We report that, although Precision-Recall Area Under the Curve (PRAUC) scores are relatively low, they exhibit markedly improved consistency when our proposed stacking method is applied across multiple data tables. This stacking approach, specifically designed to address the challenges posed by highly imbalanced datasets, enhances both the stability and robustness of performance metrics. Notably, our results are in line with previous studies on disease progression prediction, which similarly report low PRAUC values—reflecting the inherent difficulty in distinguishing relevant signal from noise within highly skewed datasets [45].

In addition, using the temporal positional embedding compared to empty or base embedding or baselines exhibits the best performance for the AUC Severe+. This suggests that our temporal PE provides
<caption>**Table 2**</caption>

Training time and trainable parameters for each model on the test set. This table presents a fair comparison of computational costs, including training time in hours (rounded to the nearest hour) and the total number of trainable parameters (in millions).

<table><thead><tr><th>Method</th><th>Trainable parameters (M)</th><th>Training time (h) 100 epochs</th><th>Backbone</th></tr></thead><tbody><tr><td>2D CNN+ pooling</td><td>23.4</td><td>10</td><td>ResNet50</td></tr><tr><td>CNN+ LSTM</td><td>27.8</td><td>13</td><td>ResNet50</td></tr><tr><td>CNN+ T-LSTM</td><td>28.1</td><td>15</td><td>ResNet50</td></tr><tr><td>ViViT</td><td>85.7</td><td>20</td><td>ViT</td></tr><tr><td>ViViT</td><td>86.1</td><td>22</td><td>ViViT</td></tr><tr><td>L-ViViT</td><td>87.5</td><td>24</td><td>ViViT</td></tr></tbody></table>

key information to the model. Similarly, we observed that the CNN+T-LSTM demonstrates a better ability to predict disease progression than the standard CNN+LSTM. This is aligned with the fact that adding temporal information provides better context for predicting the disease progression [35,46]. Except for the CNN baselines, the best performance for AUC Mild+ and AUC Moderate+ is obtained using our masking strategy and the time-aware embedding.

The rest of the results presented in Table 3 concern using different masking strategies in conjunction with a specific position embedding in the temporal dimension. Concerning the masking strategy, we report that masking more areas enables the MAE to capture a better representation, yielding a better performance across the different binary tasks.

To facilitate a comprehensive comparison, we also report the training time and the number of trainable parameters for each model. This data offers insight into the computational efficiency of each approach and assists in evaluating the trade-off between model complexity and predictive performance. While the models with advanced temporal embeddings, like CNN+T-LSTM and L-ViViT, generally achieve higher AUC scores, they may also demand greater computational resources. By examining training time and parameter counts, we can assess the practicality of each approach for deployment scenarios where resources or time constraints are a concern.

Table 2 indicates that the ViViT-based models, including L-ViViT, have the highest parameter counts and longest training times. In contrast, the 2D CNN with pooling is the most computationally efficient but exhibits lower predictive performance.

## 3.1. Ablation study

In this Section 3.1, we further study the impact of the specific hyperparameters and components presented in our proposed approach. We tested the efficacy of the proposed method by comparing it to the baseline method and different variants.

### 3.1.1. Importance of pre-trained model initialization

One of the motivations for our paper is the demonstrated effectiveness of pre-training initialization in ViT models, particularly when trained on large datasets, as Transformers lack certain inductive biases inherent in convolutional networks [21]. Consequently, training large models from scratch to achieve high accuracy is challenging. Our work highlights the essential nature of pre-trained weights for achieving high model precision.

Specifically, we illustrate the significance of three pre-trained weight components in our model: the layer embedding, time-aware embedding, and the encoder. We experimented with various combinations of pre-trained models, including random weights and weights from the best-performing model. The results, summarized in Table 4,

indicate that weight initialization is crucial for enhancing model performance. While all weight components contribute relevant information, our observations suggest that the temporal embedding and encoder weights are particularly discriminative for predicting the development of Severe+ RD. Conversely, the embedding layer appears to play a minor role across different tasks.

### 3.1.2. Progressive masking-aware strategies

Among the hyperparameters that change the nature of the masking, there is the value ratio of *r*. We tried three values of *r*. The effect of the change in value can be seen in Fig. 3. The more *r* increases, the larger the central region is masked. An increasing random noise is added on top of the masking depending on the severity grade of the image; a higher severity grade results in more regions masked in the periphery. Based on the Table 3 result in terms of AUC indicates that the best value of *r* is 0.75. Moreover, we also observe that the more we increase the value of the masking parameter for both random and progression-aware schemes, the better the results.

### 3.1.3. Importance of the component of our proposed method

Table 5 presents the performance of two variants of ViViT, a vision transformer architecture, for the task of classifying medical images into three severity levels three years later: mild, moderate, and severe. The two variants differ in their temporal embedding and masking strategies. The temporal embedding encodes the temporal information in the image sequence, while the masking strategy determines which frames to use for training and evaluation. The results show that both temporal embedding and masking strategies have a significant impact on the model performance. The Time-ViViT variant, which uses a time-aware temporal embedding and a progressive-aware masking strategy, consistently outperforms the ViViT variant, which does not use temporal embedding or a masking strategy. This suggests that explicitly modeling temporal information and dynamically masking zones that are correlated with the progression of DR are crucial for improving the performance of ViViT for this task. More specifically, the Time-ViViT variant achieves an AUC of 0.563 for mild+, 0.577 for moderate+, and 0.811 for severe+, which is significantly higher than the AUCs of the ViViT variant, respectively. This indicates that Time-ViViT is better at distinguishing between mild and moderate cases, and it is also significantly better at identifying severe cases. The improved performance of Time-ViViT can be attributed to its ability to capture temporal dependencies in the image sequence. The time-aware temporal embedding allows the model to learn the temporal relationships between frames, while the progressive-aware masking strategy ensures that the model pays attention to the most relevant frames for the task. Overall, the results demonstrate that Time-ViViT is a more effective architecture for classifying medical images into severity levels than ViViT. The use of temporal embedding and masking strategies can significantly improve the model performance, especially for identifying severe cases. These findings have implications for the development of automated systems for medical image analysis.

### 3.1.4. Masking strategies

We tried two other masking strategies. The first is the proposed masking strategy, but instead of masking based on the value of the image label, we randomly assigned a severity grade to an image and masked it according to Fig. 4. In order to demonstrate whether using at random label, the masking strategy was better than our masked strategy guided by the severity grade of DR progression.

The second was to mask an entire frame at random using a uniform distribution between the set of available images in the sequence in the time dimension. Motivated by the fact that removing a visit would force the model to take advantage of the remaining visit and capture the overall progression of the patient's trajectory with the global self-attention layers across all tokens. Conformable to Table 6, the visit masking strategies perform worse than the baseline random masking
<caption>**Table 3**</caption>

Comparison of approaches in terms of the different AUC scores for the predict next task, on the test set regarding the importance of masking hyper-parameters.

<table><caption>**Table 4**</caption><thead><tr><th>Method</th><th>Masking parameter</th><th>Temporal embedding</th><th>Masking strategies</th><th>AUC Mild+</th><th>AUC Moder+</th><th>AUC Severe+</th><th>PRAUC Severe+</th></tr></thead><tbody><tr><td>ViViT</td><td>r<sub>mask</sub> = 0.25</td><td>Empty</td><td>Random</td><td>0.484 ± 0.015</td><td>0.469 ± 0.034</td><td>0.444 ± 0.087</td><td>0.190 ± 0.006</td></tr><tr><td>ViViT</td><td>r<sub>mask</sub> = 0.5</td><td>Empty</td><td>Random</td><td>0.510 ± 0.018</td><td>0.522 ± 0.043</td><td>0.469 ± 0.074</td><td>0.210 ± 0.011</td></tr><tr><td>ViViT</td><td>r<sub>mask</sub> = 0.75</td><td>Empty</td><td>Random</td><td>0.492 ± 0.018</td><td>0.478 ± 0.048</td><td>0.481 ± 0.090</td><td>0.194 ± 0.009</td></tr><tr><td>ViViT</td><td>r = 0.25</td><td>Empty</td><td>Prog.-aware</td><td>0.494 ± 0.012</td><td>0.512 ± 0.019</td><td>0.462 ± 0.129</td><td>0.201 ± 0.010</td></tr><tr><td>ViViT</td><td>r = 0.5</td><td>Empty</td><td>Prog.-aware</td><td>0.480 ± 0.012</td><td>0.533 ± 0.022</td><td>0.614 ± 0.077</td><td>0.204 ± 0.010</td></tr><tr><td>ViViT</td><td>r = 0.75</td><td>Empty</td><td>Prog.-aware</td><td>0.514 ± 0.018</td><td>0.537 ± 0.040</td><td>0.620 ± 0.098</td><td>0.212 ± 0.010</td></tr><tr><td>ViViT</td><td>r<sub>mask</sub> = 0.25</td><td>Base</td><td>Random</td><td>0.504 ± 0.012</td><td>0.505 ± 0.023</td><td>0.512 ± 0.094</td><td>0.204 ± 0.007</td></tr><tr><td>ViViT</td><td>r<sub>mask</sub> = 0.5</td><td>Base</td><td>Random</td><td>0.490 ± 0.012</td><td>0.471 ± 0.026</td><td>0.492 ± 0.089</td><td>0.193 ± 0.004</td></tr><tr><td>ViViT</td><td>r<sub>mask</sub> = 0.75</td><td>Base</td><td>Random</td><td>0.504 ± 0.013</td><td>0.504 ± 0.034</td><td>0.587 ± 0.065</td><td>0.202 ± 0.054</td></tr><tr><td>ViViT</td><td>r = 0.25</td><td>Base</td><td>Prog.-aware</td><td>0.496 ± 0.011</td><td>0.487 ± 0.030</td><td>0.624 ± 0.108</td><td>0.204 ± 0.005</td></tr><tr><td>ViViT</td><td>r = 0.5</td><td>Base</td><td>Prog.-aware</td><td>0.502 ± 0.015</td><td>0.511 ± 0.034</td><td>0.614 ± 0.092</td><td>0.206 ± 0.009</td></tr><tr><td>ViViT</td><td>r = 0.75</td><td>Base</td><td>Prog.-aware</td><td>0.506 ± 0.027</td><td>0.525 ± 0.030</td><td>0.637 ± 0.066</td><td>0.209 ± 0.010</td></tr><tr><td>Time-ViViT</td><td>r<sub>mask</sub> = 0.25</td><td>Time-aware</td><td>Random</td><td>0.502 ± 0.009</td><td>0.539 ± 0.027</td><td>0.656 ± 0.070</td><td>0.219 ± 0.007</td></tr><tr><td>Time-ViViT</td><td>r<sub>mask</sub> = 0.5</td><td>Time-aware</td><td>Random</td><td>0.488 ± 0.018</td><td>0.463 ± 0.037</td><td>0.712 ± 0.047</td><td>0.191 ± 0.009</td></tr><tr><td>Time-ViViT</td><td>r<sub>mask</sub> = 0.75</td><td>Time-aware</td><td>Random</td><td>0.532 ± 0.016</td><td>0.539 ± 0.034</td><td>0.759 ± 0.021</td><td>0.233 ± 0.015</td></tr><tr><td>Time-ViViT</td><td>r = 0.25</td><td>Time-aware</td><td>Prog.-aware</td><td>0.504 ± 0.019</td><td>0.511 ± 0.039</td><td>0.700 ± 0.097</td><td>0.209 ± 0.011</td></tr><tr><td>Time-ViViT</td><td>r = 0.5</td><td>Time-aware</td><td>Prog.-aware</td><td>0.539 ± 0.013</td><td>0.562 ± 0.018</td><td>0.734 ± 0.077</td><td>0.244 ± 0.010</td></tr><tr><td>Time-ViViT</td><td>r = 0.75</td><td>Time-aware</td><td>Prog.-aware</td><td>0.563 ± 0.025</td><td>0.577 ± 0.028</td><td>0.811 ± 0.062</td><td>0.250 ± 0.019</td></tr></tbody></table>

Comparison of approaches with different weight initialization. With ✓ and ✗ denoting with and without the weight initialization, respectively.

<table><caption>**Table 5**</caption><thead><tr><th>Method</th><th>Embeddings layer</th><th>Temporal embedding</th><th>Encoder weights</th><th>AUC Mild+</th><th>AUC Moder+</th><th>AUC Severe+</th><th>PRAUC Severe+</th></tr></thead><tbody><tr><td>Time-ViViT</td><td>✗</td><td>✗</td><td>✗</td><td>0.490 ± 0.012</td><td>0.471 ± 0.026</td><td>0.492 ± 0.089</td><td>0.193 ± 0.004</td></tr><tr><td>Time-ViViT</td><td>✓</td><td>✗</td><td>✗</td><td>0.507 ± 0.018</td><td>0.522 ± 0.037</td><td>0.601 ± 0.104</td><td>0.208 ± 0.010</td></tr><tr><td>Time-ViViT</td><td>✗</td><td>✓</td><td>✗</td><td>0.488 ± 0.018</td><td>0.463 ± 0.037</td><td>0.712 ± 0.047</td><td>0.191 ± 0.009</td></tr><tr><td>Time-ViViT</td><td>✗</td><td>✗</td><td>✓</td><td>0.534 ± 0.016</td><td>0.534 ± 0.029</td><td>0.707 ± 0.069</td><td>0.218 ± 0.010</td></tr><tr><td>Time-ViViT</td><td>✓</td><td>✓</td><td>✗</td><td>0.532 ± 0.016</td><td>0.539 ± 0.034</td><td>0.759 ± 0.021</td><td>0.233 ± 0.015</td></tr><tr><td>Time-ViViT</td><td>✓</td><td>✗</td><td>✓</td><td>0.544 ± 0.012</td><td>0.549 ± 0.024</td><td>0.748 ± 0.067</td><td>0.248 ± 0.008</td></tr><tr><td>Time-ViViT</td><td>✗</td><td>✓</td><td>✓</td><td>0.529 ± 0.021</td><td>0.515 ± 0.035</td><td>0.764 ± 0.041</td><td>0.219 ± 0.011</td></tr><tr><td>Time-ViViT</td><td>✓</td><td>✓</td><td>✓</td><td>0.563 ± 0.025</td><td>0.577 ± 0.028</td><td>0.811 ± 0.062</td><td>0.250 ± 0.019</td></tr></tbody></table>

Comparison of approaches in terms of the different AUC scores for the predict next task, on the test set regarding the importance of each component of our proposed method.

<table><caption>**Table 6**</caption><thead><tr><th>Method</th><th>Temporal embedding</th><th>Masking strategies</th><th>AUC Mild+</th><th>AUC Moder+</th><th>AUC Severe+</th><th>PRAUC Mild+</th></tr></thead><tbody><tr><td>ViViT</td><td>No</td><td>Random</td><td>0.501 ± 0.020</td><td>0.493 ± 0.046</td><td>0.519 ± 0.106</td><td>0.205 ± 0.009</td></tr><tr><td>ViViT</td><td>No</td><td>Prog.-aware</td><td>0.504 ± 0.013</td><td>0.504 ± 0.034</td><td>0.587 ± 0.065</td><td>0.202 ± 0.005</td></tr><tr><td>ViViT</td><td>Base</td><td>Random</td><td>0.507 ± 0.018</td><td>0.522 ± 0.037</td><td>0.601 ± 0.104</td><td>0.208 ± 0.010</td></tr><tr><td>ViViT</td><td>Base</td><td>Prog.-aware</td><td>0.499 ± 0.015</td><td>0.559 ± 0.026</td><td>0.613 ± 0.080</td><td>0.207 ± 0.010</td></tr><tr><td>Time-ViViT</td><td>Time-aware</td><td>Random</td><td>0.544 ± 0.012</td><td>0.549 ± 0.024</td><td>0.748 ± 0.067</td><td>0.248 ± 0.008</td></tr><tr><td>Time-ViViT</td><td>Time-aware</td><td>Prog.-aware</td><td>0.563 ± 0.025</td><td>0.577 ± 0.028</td><td>0.811 ± 0.062</td><td>0.250 ± 0.019</td></tr></tbody></table>

Comparison of approaches in terms of masking strategies during the pretraining phase for the different AUC scores for the predict next task, on the test set. Results in bold show the best results.

<table><thead><tr><th>Method</th><th>Masking strategies</th><th>AUC Mild+</th><th>AUC Moder+</th><th>AUC Severe+</th><th>PRAUC Severe+</th></tr></thead><tbody><tr><td>Time-ViViT</td><td>Random</td><td>0.484 ± 0.011</td><td>0.504 ± 0.039</td><td>0.704 ± 0.069</td><td>0.203 ± 0.007</td></tr><tr><td>Time-ViViT</td><td>Visit</td><td>0.560 ± 0.018</td><td>0.562 ± 0.026</td><td>0.755 ± 0.076</td><td>0.248 ± 0.014</td></tr><tr><td>Time-ViViT</td><td>Prog.-aware random</td><td>0.567 ± 0.016</td><td>0.585 ± 0.033</td><td>0.799 ± 0.112</td><td>0.254 ± 0.014</td></tr><tr><td>Time-ViViT</td><td>Prog.-aware</td><td><strong>0.563</strong> ± 0.025</td><td><strong>0.577</strong> ± 0.028</td><td><strong>0.811</strong> ± 0.062</td><td>0.250 ± 0.019</td></tr></tbody></table>

for the different tasks. Surprisingly, the progression-aware masking with a random choice performs better than the baseline. This indicates that the form of the masking strategies is more relevant for predicting the DR's development than a random form. However, it performs less than the standard progression-aware masking. This supports our argument that masking strategies aligned with disease progression characteristics are a more informative masking strategy for the MAE model for predicting the development of diabetic retinopathy [32,47].

### 3.1.5. Type of time aware embeddings

The Table 7 presents an ablation study for temporal position embedding variants, across all the masking strategies. We examine the effects of different components, including the cosine function, MLP, and learnable parameters (denoted as $\omega_k$ and $\tau$), on four evaluation metrics: AUC Mild+, AUC Moder+, and AUC Severe+, PRAUC severe+. The

"Baseline" configuration represents the full embedding setup, while other rows show the impact of removing or altering specific components. Results are shown as AUC scores for each variant. The results in Table 7 reveal distinct performance variations across configurations, highlighting the impact of both the cosine function and the learnable parameters ($\omega_k$, $\tau$) within the MLP on temporal position embedding. Each row reflects a specific variant of the embedding setup, with modifications in either the cosine function or MLP. The values reported include the average AUC and PRAUC scores with their respective standard deviations for mild, moderate, and severe cases. The configuration employing both the cosine function and the MLP (full setup) achieves the highest AUC scores across all categories, particularly for severe cases (AUC Severe+ of 0.799), suggesting that this combination captures temporal dependencies most effectively. Removing the cosine function but retaining the MLP leads to a slight reduction in AUC scores,
<table><caption>**Table 7**
Ablation study of temporal position embedding variants.</caption><thead><tr><th>Cosine function</th><th>Learnable parameters (ω<sub>k</sub>, τ)</th><th>AUC Mild+</th><th>AUC Moder+</th><th>AUC Severe+</th><th>PRAUC Mild+</th></tr></thead><tbody><tr><td>Yes</td><td>With MLP</td><td>0.567 ± 0.060</td><td>0.585 ± 0.0169</td><td>0.799 ± 0.033</td><td>0.254 ± 0.112</td></tr><tr><td>No</td><td>With MLP</td><td>0.560 ± 0.050</td><td>0.562 ± 0.018</td><td>0.755 ± 0.026</td><td>0.248 ± 0.076</td></tr><tr><td>Yes</td><td>No MLP</td><td>0.488 ± 0.011</td><td>0.463 ± 0.018</td><td>0.712 ± 0.037</td><td>0.191 ± 0.047</td></tr><tr><td>No</td><td>No MLP</td><td>0.507 ± 0.010</td><td>0.522 ± 0.018</td><td>0.601 ± 0.037</td><td>0.208 ± 0.104</td></tr></tbody></table>

especially notable in moderate (AUC Moder+ of 0.562) and severe cases (AUC Severe+ of 0.755), implying that the cosine function contributes additional nuanced temporal information beyond what the MLP provides alone. Interestingly, configurations without the MLP exhibit a marked performance drop across metrics, especially for mild cases (AUC Mild+ of 0.488 with cosine, 0.507 without), indicating that the learnable parameters (ω<sub>k</sub>, τ) embedded within the MLP play a crucial role in fine-tuning the temporal encoding. In the absence of both the cosine function and MLP, a further decrease is observed, particularly in severe cases (AUC Severe+ of 0.601), reinforcing that both the MLP and the cosine function are vital for capturing comprehensive temporal patterns.

### 3.1.6. Comparison with RetFound

We compared our pretraining strategy with the ophthalmic foundation model RetFound. The results, presented in Table 1, indicate that our model outperforms RetFound. This outcome was anticipated, as our pretraining strategy is tailored specifically to our problem, whereas RetFound provides a more general representation. These findings are consistent with those reported by Holland et al. [48], who demonstrated that pretraining on task-specific contexts in ophthalmology can yield better performance than using RetFound's generalized weights.

## 4. Discussion

In this study, we tackle the task of predicting the progression of diabetic retinopathy, and we observed that the models based on using time perform better than those not time-aware. The use of a model not based on time to solve a task that tackles irregularly sampled time series is not optimal like it was mentioned in previous studies [49–51], which is also reflected by our results. One interesting aspect that emerged from the analysis is that the masking strategies for the MAE are clearly important. Adding medical knowledge to the masking strategy brings out a better representation according to the presented results.

However, we note that our masking strategy performs the best compared to the different random masking strategies, providing a better representation for the late stage according to the AUC Moderate+ and AUC Severe+. We suspect that it is due to the design of our mask that forces the model to reconstruct highly discriminating zones, resulting in an enhanced representation for predicting the progression of DR. This finding is notably observed for the experiments where an empty embedding was compared for the different masking strategies. This result is aligned with the following study [30], which performs masking auto encoding pretraining and observed that the masking area is the crucial element in the success of getting a good feature extractor that performs well on the downstream task.

Our findings support [20,26] hypothesis that masking a relatively large region is more effective in creating a more robust feature extractor.

Yet some limitations should be pointed out. There is a large number of hyper-parameters in the proposed masking strategy, and the creation of masks is less efficient in terms of computation power than classical random masking. In addition, we also think that our proposed masking strategy could shine better using larger fields of view images based either on a mosaic of CFP or ultra-widefield color fundus photographs (UWF-CFP) in order to fully take advantage of our contributions. As an extension and to overcome the mentioned limitation, we would like to

learn an optimal mask using a semi-supervised technique through attention maps with a Transformer-based method [52,53] while keeping the idea of a progressive mask according to the increase of DR severity applied to UWF-CFP or CFP mosaics.

It should be noted that neither the baselines nor any configuration of the Time-ViViT can produce satisfactory results in terms of AUC, for the prediction of the development of DR in the early stages. These different results can be partly explained by the fact that the images we used are relatively small compared to their original size and that the most critical characteristic indicates a small lesion or that the image alone is insufficient to establish the future progression of diabetic retinopathy, especially for early grades.

It should also be noted that the task of predicting the future development of diabetic retinopathy is challenging due to the following two problems.

The task of predicting disease progression poses unique challenges, especially in the context of irregularly sampled data with variable time intervals between events. Although deep learning models—particularly sequential architectures such as RNNs and LSTMs have shown promise in temporal tasks, they can struggle to accurately capture temporal dependencies when event sampling is inconsistent. To address this, time-dependent models that integrate temporal embeddings or decay mechanisms are often employed [50,51,54], as they can better account for these irregular intervals, resulting in improved predictive performance. Without incorporating temporal information, it becomes challenging for these models to reliably forecast disease progression over time. And the second one is the fact most tasks that are defined to predict the disease progression within a certain period of time often remove many patients that do not respect the selection criteria. An alternative could be to use the time to condition the future prediction point as it was performed in [55]. These approaches allow the model to train without restriction, which was not made explicit in the article.

Our method inherits the quadratic complexity of self-attention [56] with respect to the number of tokens of a Transformer [17]. This complexity for longitudinal image data, as the number of tokens grows in a linear fashion with the number of input frames, encourages the creation of more efficient architectures. In this regard, researchers are working to develop new techniques to reduce the memory usage of Transformer-based methods. These techniques include developing new model architectures that are more efficient in terms of memory usage, developing new sparse attention techniques [54,57] that can reduce the number of attention weights that need to be computed, and developing new training and inference techniques that can be used with transformer-based models on devices with limited memory resources [58].

Our results in Tables 1 and 5 and ablation study have demonstrated the complexity of such tasks. To alleviate this problem, one possible way would be to inject even more context into our approach using multimodal data based on other retina modalities [59,60] or on non-imaging data [8,11]. Typical approaches related to disease progression in DR often rely on clinical data, specifically HbA1c [8,11]. Adding this information could further improve the predictive capacity of our framework, especially for early stages.

## 5. Conclusion

In this paper, we proposed to apply MAE to longitudinal data with two simple but effective extensions. The results demonstrate the importance of injecting time in the positional embedding to increase the
predictive ability of the Transformer. Moreover, the proposed masking strategy exhibits a more discriminate nature to detect late stages of DR compared to standard random masking, which is in accordance with clinical studies [47]. In conclusion, our research has laid the groundwork for using Transformers for longitudinal and disease progression tasks. The outcomes are promising, and further exploration could assist medical professionals in selecting the most suitable DR screening intervals. Our framework could be extended to other retinal problems that involve longitudinal studies. It would also be beneficial to examine the effect of our masking approach on other retinal diseases, as many of these conditions are classified by the amount, size, and kind of lesions.

## CRediT authorship contribution statement

**Rachid Zeghlache:** Writing – review & editing, Writing – original draft, Visualization, Validation, Software, Methodology, Investigation, Formal analysis, Data curation, Conceptualization. **Pierre-Henri Conze:** Writing – review & editing, Writing – original draft, Supervision, Resources, Project administration, Methodology. **Mostafa El Habib Daho:** Writing – review & editing, Writing – original draft, Methodology. **Yihao Li:** Writing – review & editing, Writing – original draft, Methodology. **Alireza Rezaei:** Writing – review & editing, Writing – original draft. **Hugo Le Boité:** Methodology, Investigation. **Ramin Tadayoni:** Supervision, Resources, Project administration, Funding acquisition. **Pascal Massin:** Resources, Project administration, Funding acquisition, Data curation. **Béatrice Cochener:** Resources, Project administration, Funding acquisition. **Ikram Brahim:** Writing – review & editing, Writing – original draft, Methodology. **Gwenolé Quellec:** Writing – review & editing, Writing – original draft, Supervision, Resources, Project administration, Methodology, Funding acquisition. **Mathieu Lamard:** Writing – review & editing, Writing – original draft, Supervision, Resources, Project administration, Funding acquisition.

## Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

## Acknowledgments

The work takes place in the framework of Evired, an ANR RHU project. This work benefits from State aid managed by the French National Research Agency under "Investissement d'Avenir" program bearing the reference ANR-18-RHUS-0008.

## References

[1] M. Kropp, O. Golubnitschaja, A. Mazurakova, L. Koklesova, N. Sargheini, T.-T.K.S. Vo, E. de Clerck, J. Polivka, P. Potuznik, J. Polivka, I. Stetkarova, P. Kubatka, G. Thumann, Diabetic retinopathy as the leading cause of blindness and early predictor of cascading complications—risks and mitigation, EPMJA. 14 (1) (2023) 21–42, http://dx.doi.org/10.1007/s13167-023-00314-8.

[2] P. Saeedi, I. Petersohn, P. Salpea, B. Malanda, S. Karuranga, N. Unwin, S. Colagiuri, L. Guariguata, A.A. Motala, K. Ogurtsova, J.E. Shaw, D. Bright, R. Williams, Global and regional diabetes prevalence estimates for 2019 and projections for 2030 and 2045: Results from the international diabetes federation diabetes Atlas, 9th edition, Diabetes Res. Clin. Pract. 157 (2019) 107843, http://dx.doi.org/10.1016/j.diabres.2019.107843.

[3] M. Oulhadj, J. Riffi, C. Khodriss, A.M. Mahraz, A. Yahyaouy, M. Abdellaoui, I.B. Andaloussi, H. Tairi, Diabetic retinopathy prediction based on vision transformer and modified capsule network, Comput. Biol. Med. 175 (2024) 108523, http://dx.doi.org/10.1016/j.compbiomed.2024.108523, URL: https://www.sciencedirect.com/science/article/pii/S0010482524006073.

[4] M. Saini, S. Susan, Diabetic retinopathy screening using deep learning for multi-class imbalanced datasets, Comput. Biol. Med. 149 (2022) 105989, http://dx.doi.org/10.1016/j.compbiomed.2022.105989, URL: https://www.sciencedirect.com/science/article/pii/S0010482522007090.

[5] N. Tsiknakis, D. Theodoropoulos, G. Manikis, E. Ktistakis, O. Boutsora, A. Berto, F. Scarpa, A. Scarpa, D.I. Fotiadis, K. Marias, Deep learning for diabetic retinopathy detection and classification based on fundus images: A review, Comput. Biol. Med. 135 (2021) 104599, http://dx.doi.org/10.1016/j.compbiomed.2021.104599, URL: https://www.sciencedirect.com/science/article/pii/S0010482521003930.

[6] A.E. Rajesh, O.Q. Davidson, C.S. Lee, A.Y. Lee, Artificial intelligence and diabetic retinopathy: AI framework, prospective studies, head-to-head validation, and cost-effectiveness, Diabetes Care 46 (10) (2023) 1728–1739, http://dx.doi.org/10.2337/dci23-0032.

[7] W. Almattar, H. Luqman, F.A. Khan, Diabetic retinopathy grading review: Current techniques and future directions, Image Vis. Comput. 139 (2023) 104821, http://dx.doi.org/10.1016/j.imavis.2023.104821, URL: https://www.sciencedirect.com/science/article/pii/S0262885623001956.

[8] A. Bora, S. Balasubramanian, B. Babenko, S. Virmani, S. Venugopalan, A. Mitani, G. de Oliveira Marinho, J. Cuadros, P. Ruamviboonsuk, G.S. Corrado, L. Peng, D.R. Webster, A.V. Varadarajan, N. Hammel, Y. Liu, P. Bavishi, Predicting the risk of developing diabetic retinopathy using deep learning, Lancet Digit. Health 3 (2021) e10-e19, http://dx.doi.org/10.1016/S2589-7500(20)30250-8.

[9] Y. Rom, R. Aviv, S. Ianchulev, Z. Drey-Aharon, Predicting the future development of diabetic retinopathy using a deep learning algorithm for the analysis of non-invasive retinal imaging, 2022, http://dx.doi.org/10.1101/2022.03.31.22272079, URL: https://www.medrxiv.org/content/early/2022/07/05/2022.03.31.22272079.

[10] P. Nderitu, J. Nunez do Rio, L. Webster, S. Mann, D. Hopkins, J. Cardoso, M. Modat, C. Bergeles, T.L. Jackson, Predicting progression to referable diabetic retinopathy from retinal images and screening data using deep learning, Invest. Ophthalmol. Vis. Sci. 63 (7) (2022) 2087 – F0076–2087 – F0076.

[11] L. Dai, B. Sheng, T. Chen, Q. Wu, R. Liu, C. Cai, L. Wu, D. Yang, H. Hamzah, Y. Liu, X. Wang, Z. Guan, S. Yu, T. Li, Z. Tang, A. Ran, H. Che, H. Chen, Y. Zheng, J. Shu, S. Huang, C. Wu, S. Lin, D. Liu, J. Li, Z. Wang, Z. Meng, J. Shen, X. Hou, C. Deng, L. Ruan, F. Lu, M. Chee, T.C. Quek, R. Srinivasan, R. Raman, X. Sun, Y.X. Wang, J. Wu, H. Jin, R. Dai, D. Shen, X. Yang, M. Guo, C. Zhang, C.Y. Cheung, G.S.W. Tan, Y.-C. Tham, C.-Y. Cheng, H. Li, T.Y. Wong, W. Jia, A deep learning system for predicting time to progression of diabetic retinopathy, Nat. Med. (2024) 1–11, http://dx.doi.org/10.1038/s41591-023-02702-z, URL: https://www.nature.com/articles/s41591-023-02702-z. Publisher: Nature Publishing Group.

[12] J. Gui, T. Chen, J. Zhang, Q. Cao, Z. Sun, H. Luo, D. Tao, A survey on self-supervised learning: Algorithms, applications, and future trends, IEEE Trans. Pattern Anal. Mach. Intell. 46 (12) (2024) 9052–9071, http://dx.doi.org/10.1109/TPAMI.2024.3415112.

[13] T. Chen, S. Kornblith, M. Norouzi, G. Hinton, A simple framework for contrastive learning of visual representations, 2020, http://dx.doi.org/10.48550/ARXIV.2002.05709, URL: https://arxiv.org/abs/2002.05709.

[14] K. He, H. Fan, Y. Wu, S. Xie, R. Girshick, Momentum contrast for unsupervised visual representation learning, 2020, arXiv:1911.05722.

[15] Y. Huang, L. Lin, P. Cheng, J. Lyu, X. Tang, Lesion-based contrastive learning for diabetic retinopathy grading from fundus images, 2021, arXiv:2107.08274.

[16] R. Zeghlache, P.-H. Conze, M.E.H. Daho, R. Tadayoni, P. Massin, B. Cochener, G. Quellec, M. Lamard, Detection of diabetic retinopathy using longitudinal self-supervised learning, in: B. Antony, H. Fu, C.S. Lee, T. MacGillivray, Y. Xu, Y. Zheng (Eds.), Ophthalmic Medical Image Analysis, Springer International Publishing, Cham, 2022, pp. 43–52.

[17] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, 2017, arXiv:1706.03762.

[18] F. Shamshad, S. Khan, S.W. Zamir, M.H. Khan, M. Hayat, F.S. Khan, H. Fu, Transformers in medical imaging: A survey, 2022, arXiv:2201.09873.

[19] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, 2019, arXiv:1810.04805.

[20] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, R. Girshick, Masked autoencoders are scalable vision learners, 2021, arXiv:2111.06377.

[21] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Deghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, N. Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale, 2021, arXiv:2010.11929.

[22] F. He, F. Liu, R. Yao, G. Lin, Local fusion networks with chained residual pooling for video action recognition, Image Vis. Comput. 81 (2019) 34–41, http://dx.doi.org/10.1016/j.imavis.2018.12.002, URL: https://www.sciencedirect.com/science/article/pii/S0262885618301811.

[23] S. Li, Z. Wang, Y. Liu, Y. Zhang, J. Zhu, X. Cui, J. Liu, FSformer: Fast-slow transformer for video action recognition, Image Vis. Comput. 137 (2023) 104740, http://dx.doi.org/10.1016/j.imavis.2023.104740, URL: https://www.sciencedirect.com/science/article/pii/S0262885623001142.

[24] R.-C. Li, X.-J. Wu, C. Wu, T.-Y. Xu, J. Kittler, Dynamic information enhancement for video classification, Image Vis. Comput. 114 (2021) 104244, http://dx.doi.org/10.1016/j.imavis.2021.104244, URL: https://www.sciencedirect.com/science/article/pii/S0262885621001499.
[25] Y. Zhang, T. Zhang, C. Wu, Y. Zheng, Accurate video saliency prediction via hierarchical fusion and temporal recurrence, Image Vis. Comput. 136 (2023) 104744, http://dx.doi.org/10.1016/j.imavis.2023.104744, URL: https://www.sciencedirect.com/science/article/pii/S026288562300118X.

[26] C. Feichtenhofer, H. Fan, Y. Li, K. He, Masked autoencoders as spatiotemporal learners, 2022, arXiv:2205.09113.

[27] Y. Zhou, M.A. Chia, S.K. Wagner, M.S. Ayhan, D.J. Williamson, R.R. Struyven, T. Liu, M. Xu, M.G. Lozano, P. Woodward-Court, Y. Kihara, A. Altmann, A.Y. Lee, E.J. Topol, A.K. Denniston, D.C. Alexander, P.A. Keane, A foundation model for generalizable disease detection from retinal images, Nature 622 (7981) (2023) 156-163, http://dx.doi.org/10.1038/s41586-023-06555-x, URL: https://www.nature.com/articles/s41586-023-06555-x. Number: 7981 Publisher: Nature Publishing Group.

[28] R. Bommasani, D.A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M.S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, E. Brynjolfsson, S. Buch, D. Card, R. Castellon, N. Chatterji, A. Chen, K. Creel, J.Q. Davis, D. Demszky, C. Donahue, M. Doumbouya, E. Durmus, S. Ermon, J. Etchemendy, K. Ethayarajh, L. Fei-Fei, C. Finn, T. Gale, L. Gillespie, K. Goel, N. Goodman, S. Grossman, N. Guha, T. Hashimoto, P. Henderson, J. Hewitt, D.E. Ho, J. Hong, K. Hsu, J. Huang, T. Icard, S. Jain, D. Jurafsky, P. Kalluri, S. Karamcheti, G. Keeling, F. Khani, O. Khattab, P.W. Koh, M. Krass, R. Krishna, R. Kuditipudi, A. Kumar, F. Ladhak, M. Lee, T. Lee, J. Leskovec, I. Levent, X.L. Li, X. Li, T. Ma, A. Malik, C.D. Manning, S. Mirchandani, E. Mitchell, Z. Munyikwa, S. Nair, A. Narayan, D. Narayanan, B. Newman, A. Nie, J.C. Niebles, H. Nilforoshan, J. Nyarko, G. Ogut, L. Orr, I. Papadimitriou, J.S. Park, C. Piech, E. Portelance, C. Potts, A. Raghunathan, R. Reich, H. Ren, F. Rong, Y. Roohani, C. Ruiz, J. Ryan, C. Ré, D. Sadigh, S. Sagawa, K. Santhanam, A. Shih, K. Srinivasan, A. Tamkin, R. Taori, A.W. Thomas, F. Tramér, R.E. Wang, W. Wang, B. Wu, J. Wu, Y. Wu, S.M. Xie, M. Yasunaga, J. You, M. Zaharia, M. Zhang, T. Zhang, X. Zhang, Y. Zhang, L. Zheng, K. Zhou, P. Liang, On the opportunities and risks of foundation models, 2022, arXiv:2108.07258, URL: https://arxiv.org/abs/2108.07258.

[29] H. Chen, W. Zhang, Y. Wang, X. Yang, Improving masked autoencoders by learning where to mask, 2023, arXiv:2303.06583.

[30] J. Mao, S. Guo, Y. Chang, X. Yin, B. Nie, Medical supervised masked autoencoders: Crafting a better masking strategy and efficient fine-tuning schedule for medical image classification, 2023, arXiv:2305.05871.

[31] C. Wilkinson, F.L. Ferris, R.E. Klein, P.P. Lee, C.D. Agardh, M. Davis, D. Dills, A. Kampik, R. Pararajasegaram, J.T. Verdaguer, Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales, Ophthalmology 110 (9) (2003) 1677-1682, http://dx.doi.org/10.1016/S0161-6420(03)00475-5, URL: https://www.sciencedirect.com/science/article/pii/S0161642003004755.

[32] H. Stino, K.L. Huber, M. Niederleithner, N. Mahnert, A. Sedova, T. Schlegl, I. Steiner, S. Sacu, W. Drexler, T. Schmoll, R. Leitgeb, U. Schmidt-Erfurth, A. Pollreisz, Association of diabetic lesions and retinal nonperfusion using widefield multimodal imaging, Ophthalmol. Retina (2023) http://dx.doi.org/10.1016/j.oret.2023.07.020, URL: https://www.sciencedirect.com/science/article/pii/S246865302300355X.

[33] W.L. Alyoubi, M.F. Abulkhair, W.M. Shalash, Diabetic retinopathy fundus image classification and lesions localization system using deep learning, Sensors (Basel, Switzerland) 21 (2021).

[34] I. Baytas, C. Xiao, X. Zhang, F. Wang, A. Jain, J. Zhou, Patient subtyping via time-aware LSTM networks, 2017, pp. 65-74, http://dx.doi.org/10.1145/3097983.3097997.

[35] R. Zeghlache, P.-H. Conze, M. El Habib Daho, Y. Li, I. Brahim, H. Le Boité, P. Massin, R. Tadayoni, B. Cochener, G. Quellec, M. Lamard, Time-aware deep models for predicting diabetic retinopathy progression, Invest. Ophthalmol. Vis. Sci. 64 (8) (2023) 246-246.

[36] Y. Lin, H. Wan, S. Guo, Y. Lin, Pre-training context and time aware location embeddings from spatial-temporal trajectories for user next location prediction, in: AAAI Conference on Artificial Intelligence, 2021.

[37] T.Z. Li, K. Xu, R. Gao, Y. Tang, T.A. Lasko, F. Maldonado, K.L. Sandler, B.A. Landman, Time-distance vision transformers in lung cancer diagnosis from longitudinal computed tomography, Proc. SPIE Int. Soc. Opt. Eng. 12464 (2023) 1246412, http://dx.doi.org/10.1117/12.2653911, URL: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10353776/.

[38] A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lučić, C. Schmid, ViViT: A video vision transformer, 2021, arXiv:2103.15691.

[39] P. Massin, A. Chabouis, A. Erginay, C. Viens-Bitker, A. Lecleire-Collet, T. Meas, P. Guillausseau, G. Choupot, B. André, P. Denormandie, OPHDIAT©: A telemedical network screening system for diabetic retinopathy in the Ile-de-France, Diabetes Metab. 34 (2008) 227-234, http://dx.doi.org/10.1016/j.diabet.2007.12.006.

[40] T. Das, B. Takkar, S. Sivaprasad, T. Thanksphon, H. Taylor, P. Wiedemann, J. Nemeth, P.D. Nayar, P.K. Rani, R. Khandekar, Recently updated global diabetic retinopathy screening guidelines: commonalities, differences, and future possibilities, Eye 35 (10) (2021) 2685-2698, http://dx.doi.org/10.1038/s41433-021-01572-4, URL: https://www.nature.com/articles/s41433-021-01572-4. Number: 10 Publisher: Nature Publishing Group.

[41] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, L. Fei-Fei, Large-scale video classification with convolutional neural networks, in: 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 1725-1732, http://dx.doi.org/10.1109/CVPR.2014.223.

[42] J. Wang, Y. Yang, J. Mao, Z. Huang, C. Huang, W. Xu, CNN-RNN: A unified framework for multi-label image classification, 2016, arXiv:1604.04573.

[43] D. Neimark, O. Bar, M. Zohar, D. Asselmann, Video transformer network, in: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2021, pp. 3163-3172.

[44] T.Z. Li, K. Xu, R. Gao, Y. Tang, T.A. Lasko, F. Maldonado, K.L. Sandler, B.A. Landman, Time-distance vision transformers in lung cancer diagnosis from longitudinal computed tomography, in: O. Colliot, I. Iğum (Eds.), Medical Imaging 2023: Image Processing, vol. 12464, International Society for Optics and Photonics, SPIE, 2023, 1246412, http://dx.doi.org/10.1117/12.2653911.

[45] T. Emre, A. Chakravarty, A. Rivail, D. Lachinov, O. Leingang, S. Riedl, J. Mai, H.P. Scholl, S. Sivaprasad, D. Rueckert, et al., 3DTINC: Time-equivariant non-contrastive learning for predicting disease progression from longitudinal OCTs, IEEE Trans. Med. Imaging (2024).

[46] J. Bridge, S. Harding, Y. Zheng, Development and validation of a novel prognostic model for predicting AMD progression using longitudinal fundus images, BMJ Open Ophthalmol. 5 (1) (2020) http://dx.doi.org/10.1136/bmjophth-2020-000569, URL: https://bmjophth.bmj.com/content/5/1/e000569. arXiv:https://bmjophth.bmj.com/content/5/1/e000569.full.pdf.

[47] N.K. Waheed, R.B. Rosen, Y. Jia, M.R. Munk, D. Huang, A. Fawzi, V. Chong, Q.D. Nguyen, Y. Sepah, E. Pearce, Optical coherence tomography angiography in diabetic retinopathy, Prog. Retin. Eye Res. 97 (2023) 101206, http://dx.doi.org/10.1016/j.preteyeres.2023.101206, URL: https://www.sciencedirect.com/science/article/pii/S1350946223000459.

[48] R. Holland, O. Leingang, H. Bogunović, S. Riedl, L. Fritsche, T. Prevost, H.P. Scholl, U. Schmidt-Erfurth, S. Sivaprasad, A.J. Lotery, et al., Metadata-enhanced contrastive learning from retinal optical coherence tomography images, Med. Image Anal. 97 (2024) 103296.

[49] S.C.-X. Li, B.M. Marlin, Learning from irregularly-sampled time series: A missing data perspective, in: International Conference on Machine Learning, 2020, URL: https://apisemanticscholar.org/CorpusID:221089813.

[50] C. Sun, Linda Qiao, M. Song, H. Li, A review of deep learning methods for irregularly sampled medical time series data, 2020, ArXiv abs/2010.12493. URL: https://api semanticscholar.org/CorpusID:225061931.

[51] Y. Rubanova, R.T.Q. Chen, D. Duvenaud, Latent ODEs for irregularly-sampled time series, in: Proceedings of the 33rd International Conference on Neural Information Processing Systems, Curran Associates Inc., Red Hook, NY, USA, 2019.

[52] R. Sun, Y. Li, T. Zhang, Z. Mao, F. Wu, Y. Zhang, Lesion-aware transformers for diabetic retinopathy grading, in: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2021, pp. 10933-10942, URL: https://api semanticscholar.org/CorpusID:235719110.

[53] G. Li, H. Zheng, D. Liu, C. Wang, B. Su, C. Zheng, SemMAE: Semantic-guided masking for learning masked autoencoders, 2022, arXiv:2206.10207.

[54] L. Liu, Z. Qu, Z. Chen, Y. Ding, Y. Xie, Transformer acceleration with dynamic sparse attention, 2021, arXiv:2110.11299.

[55] R. Zeghlache, P.-H. Conze, M.E.H. Daho, Y. Li, H. Le Boité, R. Tadayoni, P. Massin, B. Cochener, I. Brahim, G. Quellec, M. Lamard, LMT: Longitudinal mixing training, a framework to predict disease progression from a single image, in: X. Cao, X. Xu, I. Rekik, Z. Cui, X. Ouyang (Eds.), Machine Learning in Medical Imaging, Springer Nature Switzerland, Cham, 2023, pp. 22-32.

[56] F.D. Keles, P.M. Wijewardena, C. Hegde, On the computational complexity of self-attention, 2022, arXiv:2209.04881.

[57] Y. Tay, D. Bahri, L. Yang, D. Metzler, D.-C. Juan, Sparse sinkhorn attention, in: International Conference on Machine Learning, 2020, URL: https://api. semanticscholar.org/CorpusID:211505992.

[58] T. Dao, D.Y. Fu, S. Ermon, A. Rudra, C. Ré, FlashAttention: Fast and memory-efficient exact attention with IO-awareness, 2022, arXiv:2205.14135.

[59] Y. Li, M. El Habib Daho, P.-H. Conze, R. Zeghlache, H. Ren, C. Lepicard, P. Deman, A. Le Guilcher, R. Tadayoni, B. Cochener, M. Lamard, G. Quellec, 3-D analysis of multiple OCTA acquisitions for the automatic diagnosis of diabetic retinopathy, in: Proc ARVO, New Orleans, LA, USA, 2023, p. C0116.

[60] M. El Habib Daho, Y. Li, R. Zeghlache, H. Le Boité, P. Deman, L. Borderie, H. Ren, N. Mannivanan, C. Lepicard, B. Cochener, et al., DISCOVER: 2-D multiview summarization of optical coherence tomography angiography for automatic diabetic retinopathy diagnosis, 2024, arXiv e-prints, arXiv:2401.