# Ask and Remember: A Questions-Only Replay Strategy for Continual Visual Question Answering

Imad Eddine Marouf, Enzo Tartaglione, Stéphane Lathuilière, Joost van de Weijer

► To cite this version:

Imad Eddine Marouf, Enzo Tartaglione, Stéphane Lathuilière, Joost van de Weijer. Ask and Remember: A Questions-Only Replay Strategy for Continual Visual Question Answering. ICCV 2025 - International Conference on Computer Vision, Oct 2025, Honolulu, United States. hal-05460430

HAL Id: hal-05460430

https://hal.science/hal-05460430v1

Submitted on 16 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
arXiv:2502.04469v2 [cs.CV] 27 Jul 2025

# Ask and Remember: A Questions-Only Replay Strategy for Continual Visual Question Answering

Imad Eddine Marouf<fnref n="1" />

Enzo Tartaglione<fnref n="1" />

Stéphane Lathuilière<fnref n="1" />,²

Joost van de Weijer³

¹ LTCI, Télécom-Paris, Institut Polytechnique de Paris, France

²Inria, LJK, Univ. Grenoble Alpes, France, ³Universitat Autònoma de Barcelona, Spain

## Abstract

Continual Learning in Visual Question Answering (VQACL) requires models to acquire new visual-linguistic skills (plasticity) while preserving previously learned knowledge (stability). The inherent multimodality of VQACL exacerbates this challenge, as models must balance stability across visual and textual domains while adapting to novel objects and reasoning tasks. Existing methods, primarily designed for unimodal settings, often fall short in addressing this dual requirement. In this work, we present Question-only replay with Attention Distillation (QUAD), a novel approach for VQACL that leverages only past task questions for regularization. By eliminating the need to store visual data, QUAD not only reduces memory overhead, but also alleviates privacy concerns. Our method introduces a Question-only Replay mechanism that selectively reuses prior task questions to counteract overfitting to the answer space of the current task, addressing the problem out of answer set. Complementing this, we propose Attention Consistency Distillation to enforce both intra-modal and inter-modal attention consistency across tasks, preserving essential visual-linguistic associations. Extensive experiments on VQAv2 and NExT-QA demonstrate that QUAD significantly outperforms state-of-the-art methods, achieving robust performance in continual VQA. Code is available at: https://github.com/lemProg/QUAD.<fnref n="1" />

## 1. Introduction

Continual learning (CL) allows models to lean new skills while retaining prior knowledge, effectively mitigating catastrophic forgetting (CF) [57, 58]. This capability is essential in dynamic real-world settings, where models must continuously adapt to evolving data while preserving pre-

<figure><img src="image_2.png" /><figcaption>Figure 1. Comparison of continual learning methods for Visual Question Answering (VQA) in terms of storage, forgetting, and privacy. Memory-free methods ensure privacy but suffer from high forgetting. Memory-rehearsal methods reduce forgetting but raise privacy concerns by storing sensitive images (e.g. people's identities, car plates). Our approach QUAD, only stores questions and avoids image storage, preserving privacy while achieving low forgetting by leveraging question-based regularization to effectively solve the out-of-answer set problem.</figcaption></figure>

viously acquired knowledge [95]. Despite significant advances in CL, most research has focused on unimodal tasks (e.g. image classification) [26, 38, 56, 80, 81, 97]. However, real-world applications often require multimodal learning that integrates complex visual and textual reasoning. Visual Question Answering (VQA) stands as a representative multimodal task, requiring models to jointly interpret visual content and natural language. For instance, answering questions like “What color is the car on the left?” or “How many trucks are visible?” requires robust object recognition and a nuanced grasp of the linguistic query structure [10, 70, 83, 88, 95].

The emerging field of Visual Question Answering Continual Learning (VQACL) [95] focuses on enabling models to improve VQA performance iteratively by learning from a sequence of tasks without catastrophic forgetting. Un-

<footnote n="1">¹Work done during an internship at Computer Vision Center (CVC), Universitat Autònoma de Barcelona.</footnote>
like static pretrained multimodal models [1, 44, 50] that rely on massive datasets and heavy computation, VQACL offers a more efficient and scalable alternative by allowing incremental adaptation without costly re-training [61, 95]. Research in VQACL is further driven by the broader scientific goal of developing multimodal agents that continuously improve their reasoning throughout deployment in a multimodal environments. A key challenge in VQACL is balancing *stability*—the preservation of past knowledge—and *plasticity*—the ability to learn new information—across both visual and linguistic modalities [5, 95]. This dual-modality requirement, combined with the need for generalisation ability, introduces additional complexity in learning. Models must retain visual and linguistic knowledge across tasks, while also generalizing to novel objects and unseen question types. For example, a model that masters counting vehicles should be able to transfer this counting capability to novel object categories, such as bicycles [27].

To achieve this balance, continual VQA methods rely on memory-based replay, where previously seen examples are stored and revisited to reinforce prior knowledge and mitigate forgetting [8, 9, 77, 95]. In VQACL, this necessitates storing full image-question pairs, which demands substantial memory resources, often amounting to thousands of samples per task (e.g. 5000 samples in VQACL [95]). Storing visual data poses two core challenges. First, it incurs high computational and storage overhead, which can be infeasible in resource-constrained real-world applications. Second, more importantly, it also raises serious privacy concerns. Visual data often contains sensitive and personally identifiable information, especially in domains like healthcare, finance, and surveillance [49, 74], where stringent regulations such as GDPR [22, 91] govern data usage and storage. In contrast, textual data—such as questions—is typically generic and non-identifiable, thereby posing minimal privacy risks. Memory-free methods [2, 38, 41, 93], though eliminating storage and privacy concerns by avoiding data retention entirely, often deliver suboptimal performance in multimodal settings like VQACL. This tension between memory-rehearsal and memory-free approaches prompts a key question: *Is it truly necessary to store visual data, or could retaining only past questions suffice to mitigate forgetting?* To explore these questions, we propose a novel intermediate setting: VQACL with Question-only Rehearsal (VQACL-QR) (Fig. 1). In this framework, only questions from past tasks are stored—eliminating the need for visual data—and offering a practical, privacy-preserving solution for continual VQA.

To address VQACL-QR, we propose **QUESTION-ONLY REPLAY WITH ATTENTION DISTILLATION (QUAD)**, a novel replay framework that balances stability and plasticity using only past questions—no visual data required. Our approach introduces two key contributions, each tar-

<figure><img src="image_3.png" /><figcaption>Figure 2. Out-of-Answer-Set Problem in Sequential Finetuning. Confusion matrices compare Sequential Finetuning (left) and QUAD (right) across three sequentially trained tasks: *Counting, Action, and Color* (y-axis: answers set vocabulary, x-axis: predicted answers). Diagonal values indicate model predictions on the current task, while off-diagonal shifts reveal predictions on previous tasks. Sequential Finetuning exhibits, misclassifying past-task questions with responses from the latest task (e.g., counting questions answered with ‘Yes/No’). QUAD mitigates forgetting, preserving prior knowledge while still adapting to new tasks.</figcaption></figure>

geting specific challenges in continual VQA. First, we introduce a **Question-only Replay** mechanism that leverages stored questions from previous tasks to regularize the current model through a strategic selection process that mitigates the *out-of-answer-set problem*, where models overfit to the current task’s answer space and consequently misanswer previous queries (see Fig. 2). Second, we propose **Attention Consistency Distillation**, a novel strategy that preserves attention patterns across tasks. It preserves intra-modal (text–text, image–image) and inter-modal (text–image) attention consistency, ensuring the model maintains focus on relevant regions even as it adapts to new information. Extensive experiments on standard VQACL benchmarks (VQAv2 and NExT-QA) demonstrate that QUAD achieves state-of-the-art performance, surpassing both memory-free approaches and rehearsal methods. Notably, QUAD surpasses prior methods that rely on image storage, demonstrating that *storing only questions is sufficient to mitigate forgetting*, validating the practicality of the Question-only Rehearsal (VQACL-QR) setting.

## 2. Related work

**Visual Question Answering (VQA)** is the task of responding to natural language queries by interpreting visual content [5, 45, 60]. Recent approaches leverage vision-language models (VLMs) built on transformer architectures [10, 19, 43, 71, 86] alongside pre-trained language models [88, 90]. For example, Cho et al. [11] introduced a generative transformer model that integrates visual and textual modalities for VQA. Many approaches enhance generalization by leveraging compositionality—a cornerstone of cognitive reasoning [35, 42]. For instance, Johnson et al. [33] explored the composition of visual attributes by creating a dataset designed for compositional reasoning, while
Whitehead et al. [84] used contrastive learning to enhance compositionality, disentangling reasoning skills from visual concepts. Despite these advances, implicit decomposition strategies can hinder generalization, and crafting effective contrastive samples remains challenging.

**Continual Learning (CL)** seeks to develop frameworks that incrementally assimilate new data while mitigating catastrophic forgetting. This is a fundamental challenge for many deep learning methods due to catastrophic forgetting (CF) [57]. CL methods are broadly categorized into knowledge distillation-based approaches, which constrain mappings between successive models to prevent forgetting [16, 31, 34, 46, 54, 66]; optimisation-based, which adjust gradient updates to minimize interference with prior tasks [7, 52, 69, 89] and representation-based strategies that learn robust, adaptable features [17, 18, 20, 21]. Recently, prompt-based methods have emerged [72, 78, 81, 82], employing visual prompts with pre-trained transformers in CL scenarios [51]. However, since most techniques are designed for Class-Incremental Learning (CIL) in unimodal settings, they fall short when applied to multimodal data and overlook the critical issue of compositional generalization in VQA [61, 95].

**Continual VQA.** Recent studies have explored multimodal continual learning in VQA [13, 28, 61, 73]. However, prior works like Greco et al. [28] primarily analyze forgetting dynamics without proposing dedicated solutions, often overlooking the role of pre-trained models [59]. The VQACL benchmark [95] evaluated conventional continual learning approaches such as [8, 9, 77, 95]. Other studies have investigated specific facets of continual VQA, including question diversity [28], compositionality [95] and domain adaptation [96]. In contrast, our work introduces a novel question-only replay mechanism with attention distillation for continual VQA that is both memory-efficient, and privacy-conscious, eliminating the need for stored prototypes as employed in the VQACL method [95]$^2$.

# 3. Question-only replay with Attention Distillation (QUAD)

## 3.1. Setting Overview

The VQACL setting [95] is designed to assess the capability of a model to adapt to a sequence of tasks, each involving both visual and linguistic inputs, in a continual learning environment. It approaches VQA as a generative task, where the objective is to generate textual answers given an image and a corresponding question [23, 95]. The model encounters a non-stationary stream of data, requiring it to learn and adapt incrementally over time without revisiting prior data. We consider a sequence of T tasks, denoted as

$T^1, T^2, \dots, T^T$. Each task $T^t$ is characterized by a set of image-question-answer triplets $(x^t, q^t, y^t)$, where $x^t \in \mathcal{X}^t$ denotes the image, $q^t \in \mathcal{Q}^t$ represents the question, and $y^t \in \mathcal{Y}^t$ corresponds to the answer.$^3$ The challenge is to train a model $\phi$ that can effectively learn the current task $T^t$ while retaining the knowledge from all previous tasks $\{T^1, T^2, \dots, T^{t-1}\}$.

In VQACL, the sequence of tasks is organized as a series of L macro-tasks, each comprising K sub-tasks, resulting in a total of $T = L \times K$ tasks. Each macro-task is designed to develop specific reasoning skills such as counting, color identification, or object recognition (i.e. linguistic task). For example, in a counting task, the model primarily engages with questions like “How many objects are there?” or “What number is shown?”.

Each linguistic macro-task is further divided into visually-driven sub-tasks. Formally, each macro-task $T^t$ is split into K visually-driven sub-tasks, $\{S_1^t, S_2^t, \dots, S_K^t\}$, which are learned sequentially. These sub-tasks $S_k^t$ are constructed by grouping the C distinct visual object categories, $\{c_i\}_{i=1}^C$, into K sets. This hierarchical structure mirrors the continuous nature of the visual and linguistic data streams that the model processes. The VQACL setting introduces two unique challenges for continual learning models. 1) Knowledge Retention: as the model progresses through the sequence of tasks, it must retain knowledge from earlier tasks to perform well on future tasks, where both visual and linguistic modalities must be preserved. 2) Generalization to Novel Compositions: this setting also evaluates the model's ability to generalize to novel combinations of visual concepts and reasoning skills that it has not encountered during training. This aspect is crucial for real-world applications where new object-skill combinations are frequently encountered. Details of task sequences, object groupings, and novel composition testing is in the Appendix.

## 3.2. Overview

We propose QUAD, a novel approach for VQACL-QR framework that avoids image storage by relying solely on previously encountered questions (see Fig. 3). Inspired by prior work in continual learning [14, 47], we adopt a regularisation framework in which the overall learning objective $\mathcal{L}_{VQACL}$ is composed of two main components:

$$ \mathcal{L}_{VQACL} = (1 - \lambda)\mathcal{L}_{\text{Plasticity}} + \lambda\mathcal{L}_{\text{Stability}}. \quad (1) $$

The plasticity term $\mathcal{L}_{\text{Plasticity}}$, drives the model to adapt to the current task $T^t$, while the weighting factor $\lambda > 0$ balances the trade-off between the two loss terms. Following common practice in VQA [4, 65, 95], we implement the plasticity loss using cross-entropy to compare the network's prediction for an input image-question pair $(x^t, q^t)$, against

<footnote>$^2$VQACL represents both the setting and the approach for continual learning in Visual Question Answering, as described in [95].</footnote>

<footnote>$^3$The sample index is omitted for clarity.</footnote>
<figure><img src="image_4.png" /><figcaption>Figure 3. Overview of QUAD. QUAD is composed of three components that jointly promote stability and plasticity in VQACL setting. (1) Question-Only Memory ($\mathcal{M}$) stores questions from past tasks, without visual data. (2) Question-only replay ($\mathcal{L}_{QR}$) leverages answers generated by the previous model $\theta^{t-1}$ for new image-question pairs, encouraging the current model $\theta^t$ to retain past knowledge. (3) Attention Consistency Distillation ($\mathcal{L}_{ACD}$) aligns the self-attention maps between $\theta^{t-1}$ and $\theta^t$ to maintain focus on relevant visual-linguistic relationships. The task-specific loss ($\mathcal{L}_{CE}$) is applied solely to current task samples, promoting adaptation to new data.</figcaption></figure>

the ground-truth answer $y^t$:

$$ \mathcal{L}_{\text{Plasticity}} = \mathbb{E}_{(x^t, q^t, y^t) \sim \mathcal{T}^t} \mathcal{L}_{\text{CE}}[\phi(x^t, q^t), y^t]. \quad (2) $$

The second loss component, the stability term $\mathcal{L}_{\text{Stability}}$, mitigates catastrophic forgetting. In the standard VQACL, where images from previous tasks are stored in memory $\mathcal{M}$, the stability term is computed analogously using cross-entropy, averaging the loss over triplets $(x^m, q^m, y^m) \in \mathcal{M}$. However, under the VQACL-QR setting, storage of images $x^m$ from past tasks in the memory $\mathcal{M}$ is disallowed. To overcome this limitation, we design a novel stability loss $\mathcal{L}_{\text{Stability}}$ tailored for VQACL-QR. Our approach integrates two complementary losses—question-only replay $\mathcal{L}_{QR}$ and attention distillation $\mathcal{L}_{ACD}$—to effectively compensate for the absence of past task images, ensuring robust knowledge retention within the constraints of the VQACL-QR framework. Thus, the stability term is expressed as:

$$ \mathcal{L}_{\text{Stability}} = \mathcal{L}_{\text{QR}} + \mathcal{L}_{\text{ACD}}. \quad (3) $$

Next, we describe the implementation of these loss components without storing past task images.

### 3.3. Question-only Replay

To enhance knowledge retention in continual VQA, we propose a questions-only replay strategy that replays stored questions by pairing them with current task images to simulate past tasks. For each image from the current task $x^t$, we pair it with a question $q^m$ sampled from the memory. By combining stored questions with new images, our approach encourages the model to recall and reinforce previously

acquired knowledge. Inspired by prior distillation-based works [14, 47], we use the model from the previous task, $\phi^{t-1}$ to generate answers for each new image-question pair $(x^t, q^m)$. The generated answers act as soft pseudo-labels for the current model $\phi^t$, enforcing consistency with prior knowledge. The loss is defined as:

$$ \mathcal{L}_{\text{QR}} = \mathbb{E}_{x^t \sim \mathcal{T}^t} \mathbb{E}_{q^m \sim \mathcal{M}} \mathcal{L}_{\text{CE}}[\phi^t(x^t, q^m), \phi^{t-1}(x^t, q^m)]. \quad (4) $$

Notably, we employ the network's output as soft pseudo-labels without applying the argmax operator. Using argmax would force the network to align exclusively with the most probable class; in contrast, soft pseudo-labeling allows for a more nuanced alignment with the full output distribution of $\phi^{t-1}$ [30, 76, 94].

These image-question pairs $(x^t, q^m)$ expose the model to a wide range of visual-question combinations, thereby improving the model's retention of prior knowledge. Importantly, this strategy enables the model to maintain versatility in answering diverse question types, even when predictions exhibit some uncertainty. Without this regularization, the model becomes vulnerable to the *out-of-answer-set problem*, wherein overfitting to the current task's answer space results in erroneous responses for questions from prior tasks (see Fig. 2). For example, a model trained on color recognition might erroneously return a color name when confronted with a counting question from a previous task. Notably, this phenomenon closely resembles class recency bias in class-incremental learning (CIL), where models disproportionately favor newly learned classes over previously encountered ones [55, 68].
**Question Selection for QUAD.** Randomly pairing current task images $x^t$ with past questions $q^m$ from memory can partially mitigate forgetting but can produce semantically incoherent image-question pairs. For instance, if the current macro-task $\mathcal{T}^t$ focuses on counting, naive pairing might associate an image of cars with the question “How many cows are in this image?”, leading to incoherent supervision and diminished performance. To address this, we introduce a targeted question selection strategy that prioritizes questions related to the object categories $c_i$ in the current visually-driven subtask $S^l$. This ensures that replayed examples remain contextually meaningful, reinforcing relevant knowledge while preventing misleading associations. By aligning stored questions with the current task’s visual concepts, QUAD facilitates effective adaptation across linguistic and visual subtasks, improving continual learning performance.

*Example:* Suppose the current visually-driven subtask $S^l$ involves learning to count cars. The question selection strategy prioritizes memory questions relevant to cars, such as “What’s the color of the car?”, ensuring that replay maintains associations between visual and linguistic queries.

## 3.4. Attention Consistency Distillation

While question-only replay using pseudo-labeling ensures output consistency, it fails to constrain internal representations, causing self-attention drift and disrupting alignment with previously learned tasks [24, 63, 79]. In contrast, feature distillation across multiple layers [14, 34, 63, 87] offers stronger regularization by aligning internal representations; however, it often restricts plasticity due to its rigid constraints. These methods enforce layer-wise alignment, which becomes problematic when encountering new image-question pairs ($x^t, q^m$) that were not present in previous training. Despite their semantic relevance, the model lacks prior exposure to these pairs, making strict layer-wise regularization overly restrictive. Our VQA model [61, 95] encodes image and text tokens within a unified transformer sequence, allowing self-attention to dynamically capture intra-modal (text-to-text, image-to-image) and inter-modal (text-to-image) dependencies. However, sequential fine-tuning gradually shifts self-attention [24, 79], causing the model to focus on different visual regions than those relevant to past tasks, ultimately degrading performance on prior knowledge [24, 75]. This drift arises because pseudo-labeling alone fails to stabilize internal representations, necessitating explicit regularization.

To mitigate this, we introduce attention consistency distillation (ACD), which aligns the self-attention maps of the current model $\phi^t$ and its predecessor $\phi^{t-1}$. By preserving intra-modal and inter-modal relationships, this regularization stabilizes attention patterns while maintaining flexibility. Distilling only attention maps guides the model to at-

tend to task-relevant visual regions for question answering while preserving adaptability to novel input patterns.

For an image-question pair ($x^t, q^m$) with $x^t \sim \mathcal{T}^t, q^m \sim \mathcal{M}$: we denote $A_k^t$ and $A_k^{t-1}$ as the corresponding self-attention maps in an attention head $k$ for the current and previous model. Thus, we define the self-attention consistency loss $\mathcal{L}_{\text{ACD}}$ via cross-entropy, ensuring that attention distributions remain aligned across tasks:

$$ \mathcal{L}_{\text{ACD}} = \mathbb{E}_{x^t \sim \mathcal{T}^t} \mathbb{E}_{q^m \sim \mathcal{M}} \mathbb{E}_{k \sim \mathcal{K}_{\phi}} \mathcal{L}_{\text{CE}} [A_k^t(x^t, q^m), A_k^{t-1}(x^t, q^m)], \quad (5) $$

where $\mathcal{K}_{\phi}$ represents all attention heads across layers of $\phi$.

Unlike prior L1-based approaches [14, 63], which impose uniform penalties and operate directly on the raw query-key products, ACD leverages cross-entropy on normalized attention maps (after softmax operation) to preserve the probabilistic structure of attention distributions. This gives greater emphasis to highly attended regions while low-attended areas remain flexible, preventing over-penalization of less critical regions. Asymmetric regularization (e.g., ReLU+L1) [63] partially addresses this issue by prioritizing reductions in attended regions but lacks a probabilistic alignment mechanism, making it less effective in maintaining structured dependencies in multimodal learning. In contrast, ACD offers an importance-weighted alignment that preserves essential cross-modal associations without hindering adaptation. The effectiveness of ACD is empirically demonstrated in Sec. 4, and further discussed in the Appendix.

# 4. Experiments

## 4.1. Experimental setup

**Implementation Details.** To ensure a fair comparison, we adopt the protocol of [95] for both feature extraction and training across datasets. For the visual embeddings, we use a Faster R-CNN [67] trained on the Visual Genome dataset [40] extracting 36 region-based object features per image in the VQAv2 dataset. For videos in the NEXT-QA dataset, we extract clip-level motion features using an inflated 3D ResNeXt-101 [29], setting $n = 16$ regions per clip. A two-layer MLP with GELU activation adapts these features for input into the transformer backbone. Our transformer backbone, based on T5 [64], consists of 12 blocks for both encoder and decoder modules, each containing 12 attention heads. The embedding dimension $d$ is tailored to the task-specific requirements. Training is conducted for 3 epochs per task, with a batch size of 80. We utilize the Adam optimizer [36] with an initial learning rate of $10^{-4}$. $\lambda$ is set to 0.5 in all experiments. All implementations are based on PyTorch [62]. Importantly, unlike [95], QUAD does not store any visual or question prototypes.
<table><caption>Table 1. Model performance on VQAv2 and NExT-QA. #Mem: memory size; Type #Mem: type of memory used, where ? and 📷 denote storing both questions and images in the memory buffer; Standard Test: standard testing; Novel Comp. Test: novel composition testing; AP: Final Average Performance (%); Forget: Average Forgetting (%). Best results are in **bold**, second-best are **underlined**.</caption><thead><tr><th rowspan="3">Methods</th><th colspan="6">VQAv2</th><th colspan="6">NExT-QA</th></tr><tr><th rowspan="2">#Mem</th><th rowspan="2">Type #Mem</th><th colspan="2">Standard Test</th><th colspan="2">Novel Comp. Test</th><th rowspan="2">#Mem</th><th rowspan="2">Type #Mem</th><th colspan="2">Standard Test</th><th colspan="2">Novel Comp. Test</th></tr><tr><th>AP (↑)</th><th>Forget (↓)</th><th>AP (↑)</th><th>Forget (↓)</th><th>AP (↑)</th><th>Forget (↓)</th></tr></thead><tbody><tr><td>Joint</td><td>-</td><td>None</td><td>51.64</td><td>-</td><td>51.10</td><td>-</td><td>-</td><td>None</td><td>35.92</td><td>-</td><td>36.24</td><td>-</td></tr><tr><td>Vanilla</td><td>None</td><td>None</td><td>14.92</td><td>30.80</td><td>11.79</td><td>27.16</td><td>None</td><td>None</td><td>12.68</td><td>25.94</td><td>12.59</td><td>28.04</td></tr><tr><td>EWC [37]</td><td>None</td><td>None</td><td>15.77</td><td>30.62</td><td>12.83</td><td>28.16</td><td>None</td><td>None</td><td>13.01</td><td>24.06</td><td>11.91</td><td>27.44</td></tr><tr><td>MAS [2]</td><td>None</td><td>None</td><td>20.56</td><td>11.16</td><td>23.90</td><td>6.24</td><td>None</td><td>None</td><td>18.04</td><td>10.07</td><td>21.12</td><td>10.09</td></tr><tr><td>ER [9]</td><td>5000</td><td>?</td><td>36.99</td><td>5.99</td><td>33.78</td><td>5.76</td><td>500</td><td>?</td><td>30.55</td><td>4.91</td><td>32.20</td><td>5.57</td></tr><tr><td>DER [8]</td><td>5000</td><td>?</td><td>35.35</td><td>8.62</td><td>31.52</td><td>8.59</td><td>500</td><td>?</td><td>26.17</td><td>5.12</td><td>21.56</td><td>12.68</td></tr><tr><td>VS [77]</td><td>5000</td><td>?</td><td>34.03</td><td>8.79</td><td>32.96</td><td>5.78</td><td>500</td><td>?</td><td>28.13</td><td>4.45</td><td>29.47</td><td>6.14</td></tr><tr><td>VQACL [95]</td><td>5000</td><td>?</td><td>37.46</td><td>6.96</td><td>35.40</td><td>4.90</td><td>500</td><td>?</td><td>30.86</td><td>4.12</td><td><b>33.85</b></td><td><b>3.80</b></td></tr><tr><td><b>QUAD (Ours)</b></td><td>5000</td><td>?</td><td><b>39.25</b></td><td><b>4.91</b></td><td><b>40.00</b></td><td><b>3.81</b></td><td>500</td><td>?</td><td><b>31.70</b></td><td><b>2.91</b></td><td><b>33.21</b></td><td><b>4.16</b></td></tr></tbody></table>

**Evaluation Metrics.** We utilise two established metrics for continual learning [6, 53, 95]: final average performance (AP), and average forgetting (Forget). The AP metric reflects the model's overall performance across all learnt tasks, highlighting its ability to consistently acquire new tasks. Let $a_{i,j}$ represent the performance of the model on task $T^i$ after it has completed learning this task $T^j$. Then, AP is calculated as: $AP = \frac{1}{T} \sum_{t=1}^{T} a_{t,T}$. Additionally, the Forget metric serves as a proxy for knowledge retention, quantifying performance degradation on prior tasks as new tasks are learned. It is computed as: $Forget = \frac{1}{T-1} \sum_{t=1}^{T-1} \max_{z \in \{t,...,T-1\}} (a_{t,z} - a_{t,T})$. To ensure a fair comparison, For NExT-QA, following [85, 95], we compute $a_{i,j}$ using Wu-Palmer Similarity (WUPS) to evaluate answer quality. For the VQAv2 dataset, as described in [95], we use the percentage of correctly answered questions as the value for $a_{i,j}$.

**Baselines.** We benchmark QUAD against five established continual learning methods spanning regularization and rehearsal-based strategies. This includes two regularization-based methods: Elastic Weight Consolidation (EWC) [38] and Memory Aware Synapses (MAS) [2], as well as three rehearsal-based approaches: Experience Replay (ER) [9], Dark Experience Replay (DER) [8], Virtual Sample (VS) [77], and VQACL [95] (see details in Appendix). We additionally report lower and upper performance bounds to frame the results: a lower bound (Vanilla) using naive finetuning without forgetting mitigation, and an upper bound (Joint) that trains on all tasks simultaneously. Following the evaluation protocol in VQACL [95], we employ two key evaluation strategies: standard testing, which evaluates the models' performance on previously encountered task types, assessing their ability to retain learnt knowledge, and novel composition testing, which challenges the models with previously unseen combinations of visual and linguistic elements, probing their capacity for compositional generalization. This twofold evaluation reveals

how well each method balances retention and adaptation (details in Appendix). Additional experiments using pretrained models are included in the Supp Mat.

## 4.2. Main results

**Performance Analysis On Standard Setting.** Tab. 1 shows a detailed comparison of continual learning approaches in the VQACL setting, with our proposed QUAD achieving superior performance across both standard and novel composition tests. QUAD consistently outperforms other methods in AP and forgetting demonstrating robust knowledge retention. On VQAv2, QUAD attains an AP of 39.25% in standard testing, outperforming the best rehearsal-based approach (VQACL) by 1.79%. Similarly, in NExT-QA, QUAD achieves an AP of 31.70%, exceeding competing methods by 0.84% to 4.73%. The forgetting rates for QUAD are also the lowest, with 4.91% and 2.91% for VQAv2 and NExT-QA, respectively, underscoring its capacity for stable knowledge retention.

In novel composition testing, QUAD demonstrates strong generalization, achieving top AP scores of 40.00% on VQAv2 and 33.85% on NExT-QA. Notably, despite not storing images, our approach maintains high performance in novel settings, with a minimal performance gap between standard and novel composition tests (0.75% for VQAv2 and 0.64% for NExT-QA). This narrow gap indicates that distilling knowledge using only seen questions effectively reinforces visual-linguistic associations, enabling the model to generalise well even in unfamiliar contexts. This performance underscores QUAD's capacity to construct robust, adaptable representations, showing that question-only distillation can successfully support compositional reasoning without the need for visual memory.

**Performance Analysis of Novel Composition Testing.** Tab. 2 provides a detailed comparison of model performance on novel and seen skill-concept compositions across VQAv2 and NExT-QA datasets. QUAD consistently sur-
<table><caption>Table 2. Fine-grained VQA performance AP (%) on the Novel and Seen skill-concept compositions of VQAv2 and NExT-QA. +Δ denotes the improvement of our method over the state of the art. Best results are in **bold**, second-best are **underlined**.</caption><thead><tr><th rowspan="2">Dataset</th><th rowspan="2">Method</th><th rowspan="2">Memory</th><th colspan="2">Group-1</th><th colspan="2">Group-2</th><th colspan="2">Group-3</th><th colspan="2">Group-4</th><th colspan="2">Group-5</th><th colspan="2">Avg</th></tr><tr><th>Novel</th><th>Seen</th><th>Novel</th><th>Seen</th><th>Novel</th><th>Seen</th><th>Novel</th><th>Seen</th><th>Novel</th><th>Seen</th><th>Novel</th><th>Seen</th></tr></thead><tbody><tr><td rowspan="5">NExT-QA</td><td>DER [8]</td><td>?</td><td>27.56</td><td>26.09</td><td>26.14</td><td>24.54</td><td>23.53</td><td>26.43</td><td>9.30</td><td>9.79</td><td>21.26</td><td>23.74</td><td>21.56</td><td>21.38</td></tr><tr><td>VS [77]</td><td>?</td><td>31.42</td><td>30.88</td><td>29.17</td><td>31.26</td><td>25.23</td><td>26.10</td><td>30.01</td><td>29.10</td><td>31.54</td><td>31.79</td><td>29.47</td><td>29.83</td></tr><tr><td>ER [9]</td><td>?</td><td>31.86</td><td>34.51</td><td>32.36</td><td>35.08</td><td>29.50</td><td>34.30</td><td>33.57</td><td>33.30</td><td>33.71</td><td>32.91</td><td>32.20</td><td>34.02</td></tr><tr><td>VQACL [95]</td><td>?</td><td>32.86</td><td>31.47</td><td>31.98</td><td>35.58</td><td>31.79</td><td>35.70</td><td>35.04</td><td>34.12</td><td>37.62</td><td>34.92</td><td><strong>33.85</strong></td><td><strong>34.35</strong></td></tr><tr><td><strong>QUAD (Ours)</strong></td><td>?</td><td>33.42</td><td>33.92</td><td>32.02</td><td>35.42</td><td>31.78</td><td>36.36</td><td>32.98</td><td>33.34</td><td>37.84</td><td>34.06</td><td><u>33.21</u></td><td><strong>34.62</strong></td></tr><tr><td rowspan="5">VQAv2</td><td>DER [8]</td><td>?</td><td>30.80</td><td>29.89</td><td>32.19</td><td>33.24</td><td>34.88</td><td>34.08</td><td>29.60</td><td>30.90</td><td>30.14</td><td>32.56</td><td>31.52</td><td>32.13</td></tr><tr><td>VS [77]</td><td>?</td><td>33.35</td><td>33.87</td><td>33.18</td><td>32.21</td><td>34.50</td><td>33.84</td><td>31.29</td><td>33.98</td><td>32.46</td><td>33.87</td><td>32.96</td><td>33.55</td></tr><tr><td>ER [9]</td><td>?</td><td>34.52</td><td>37.03</td><td>33.40</td><td>35.55</td><td>34.79</td><td>34.20</td><td>33.86</td><td>35.02</td><td>32.34</td><td>35.91</td><td>33.78</td><td>35.54</td></tr><tr><td>VQACL [95]</td><td>?</td><td>36.12</td><td>37.99</td><td>35.39</td><td>36.92</td><td>36.26</td><td>35.16</td><td>34.85</td><td>35.64</td><td>34.36</td><td>36.28</td><td>35.40</td><td>36.40</td></tr><tr><td><strong>QUAD (Ours)</strong></td><td>?</td><td>39.19</td><td>41.06</td><td>38.40</td><td>39.50</td><td>43.15</td><td>39.19</td><td>40.01</td><td>40.72</td><td>39.20</td><td>40.62</td><td><strong>40.00</strong></td><td><strong>40.21</strong></td></tr></tbody></table>

passes previous approaches. On VQAv2, QUAD shows significant gains over VQACL, with an average improvement of 4.60% on novel compositions and 3.81% on seen compositions. This result indicates enhanced compositional generalisation, as evidenced by the smaller gap between novel and seen performance compared to other methods. For NExT-QA, QUAD maintains competitive performance, with slight gains over VQACL on seen groups (average +0.27%) in a dataset that presents unique challenges, such as temporal and causal reasoning. For such tasks, storing images may be necessary to maintain a comprehensive task understanding. These consistent gains highlight QUAD's robustness in compositional reasoning, validating its effectiveness for continual VQA.

## 5. Ablation Study and Analysis

① **QUAD Components.** Tab. 3 shows the impact of questions-only replay and attention distillation in QUAD. We evaluate: (1) *Question-only replay alone*, (2) *attention distillation alone*, and (3) *the combination of both*.

Using question-only replay mechanism achieves moderate performance, with AP of 30.72% on VQAv2 and 29.04% on NExT-QA, indicating effective knowledge preservation across tasks. In contrast, attention distillation alone yields lower AP scores (13.34% on VQAv2 and 13.24% on NExT-QA) with high forgetting scores (32.08% on VQAv2 and 24.56% on NExT-QA), suggesting limited task adaptation. Combining $L_{QR}$ and $L_{ACD}$ achieves the best results, with AP scores of 39.25% on VQAv2 and 31.70% on NExT-QA, and the lowest forgetting rates.

② **Attention Distillation.** Tab. 4 demonstrates the effectiveness of our proposed $L_{ACD}$ within QUAD. Unlike prior methods such as Attn-dist (L1) and Asym-Attn [63], which impose L1 or ReLU+L1 losses on raw attention scores, our approach applies cross-entropy over normalized attention maps. QUAD outperforms previous methods, achieving an AP of 39.25% and Forgetting of 4.91% on VQAv2, and an AP of 31.70% with Forgetting of 2.91% on NExT-QA.

<table><caption>Table 3. Ablation study of QUAD components.</caption><thead><tr><th rowspan="2">L<sub>QR</sub></th><th rowspan="2">L<sub>ACD</sub></th><th rowspan="2">Memory Type</th><th colspan="2">VQAv2</th><th colspan="2">NExT-QA</th></tr><tr><th>AP (↑)</th><th>Forget (↓)</th><th>AP (↑)</th><th>Forget (↓)</th></tr></thead><tbody><tr><td>✓</td><td></td><td>?</td><td>30.72</td><td>13.74</td><td>29.04</td><td>4.58</td></tr><tr><td></td><td>✓</td><td>?</td><td>13.34</td><td>32.08</td><td>13.24</td><td>24.56</td></tr><tr><td>✓</td><td>✓</td><td>?</td><td><strong>39.25</strong></td><td><strong>4.91</strong></td><td><strong>31.70</strong></td><td><strong>2.91</strong></td></tr></tbody></table>

<table><caption>Table 4. Comparison of attention distillation methods.</caption><thead><tr><th rowspan="2">Method</th><th colspan="2">VQAv2</th><th colspan="2">NExT-QA</th></tr><tr><th>AP (↑)</th><th>Forget (↓)</th><th>AP (↑)</th><th>Forget (↓)</th></tr></thead><tbody><tr><td>L<sub>QR</sub> + Attn-dist (L1)</td><td>34.56</td><td>7.91</td><td>30.14</td><td>5.78</td></tr><tr><td>L<sub>QR</sub> + Asym-Attn [63]</td><td>38.15</td><td>5.57</td><td>31.18</td><td>4.13</td></tr><tr><td><strong>QUAD (Ours)</strong></td><td><strong>39.25</strong></td><td><strong>4.91</strong></td><td><strong>31.70</strong></td><td><strong>2.91</strong></td></tr></tbody></table>

These results highlight the benefit of distributional consistency in attention maps over unstructured alignment.

③ **Analysing Plasticity/Stability.** Fig. 4 compares how different methods manage forgetting and adaptation on the VQAv2 dataset. The sequential fine-tuning baseline (left) exhibits uniformly low off-diagonal scores, signaling a complete failure to retain knowledge from earlier tasks. This severe forgetting stems from overfitting to the current task's answer space, a phenomenon we call the *out-of-answer-set problem*, it occurs when the model overfits to the current task's answer space, preventing it from correctly responding to questions from earlier tasks. By contrast, question-only replay $L_{QR}$ (center) noticeably improves retention through our question-only replay mechanism, especially in tasks like “commonsense” and “count”. However, it remains less effective in complex reasoning tasks like “causal” and “subcategory”.

Our full method, QUAD (right), achieves the best balance: it sustains high diagonal accuracy while boosting off-diagonal retention. reflecting both adaptability and long-term memory. For example, QUAD preserves 62.6% accuracy on the 'judge' task, compared to only 35.0% with replay alone—highlighting the value of attention consistency distillation. Notably, tasks like 'type' that rely heavily on visual semantics remain challenging under the question-
<figure><img src="image_5.png" /><figcaption>Figure 4. **Plasticity/Stability analysis on VQAv2.** Each matrix shows the performance of a model trained on tasks (rows) and evaluated on tasks **sequentially** (columns). The diagonal (highlighted in **orange**) represents in-domain performance, while off-diagonal elements indicate cross-domain generalization. Higher values (darker colors) suggest better retention and plasticity. The progression from 'Sequential finetuning', to $L_{QR}$ and then to our full method, QUAD, highlights the improvement in retaining knowledge across sequential tasks.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Figure 5. **Sensitivity analysis to memory size.** Our method, QUAD, consistently achieves higher AP than baselines, demonstrating strong scalability on VQAv2 and stable performance on NExT-QA, especially as memory size increases.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Figure 6. **Question selection strategy in QUAD.** Figure shows AP for random (blue) and object-matched (purple) question selection across memory sizes on VQAv2, and NExT-QA. Object-matched selection consistently outperforms random selection.</figcaption></figure>

only setting, reaffirming the need for richer visual grounding in some categories. Nevertheless, QUAD excels in conceptually driven tasks such as “commonsense”, showing its effectiveness even with reduced supervision.

④ **Sensitivity to Memory Size.** Fig. 5 shows how different continual learning methods respond to varying memory budgets. Across all memory sizes, QUAD consistently outperforms baselines (ER, DER, VS, VQACL), demonstrating the effectiveness of our distillation strategy. On VQAv2, QUAD exhibits robust scalability, with AP increasing steadily from 1K to 5K samples. In constrained storage scenarios, QUAD maintains competitive performance by leveraging question-only distillation. On the more visually complex NExT-QA benchmark, QUAD still leads, though with narrower margins. This reflects the greater challenge of retaining visual-semantic alignment using question-only signals.

⑤ **Effectiveness of Object-Matched Question Selection.** Fig. 6 highlights the advantage of object-matched question selection compared to random pairing. We analyze

its impact across memory sizes on VQAv2 and NExT-QA, where it consistently improves AP as memory grows. This targeted selection ensures semantic relevance between the question and visual context, which helps reinforce meaningful cross-task associations. As a result, the model adapts effectively to new tasks while retaining prior knowledge.

## 6. Conclusion

In this work, we introduced **QUESTION-ONLY REPLAY WITH ATTENTION DISTILLATION (QUAD)**, a novel questions-only replay framework for continual VQA. Unlike conventional methods that store both images and questions, QUAD addresses storage and privacy concerns by retaining only past task questions. This design enables effective regularization without storing sensitive visual data, making it highly practical for privacy-conscious applications. Comprehensive evaluations on VQAv2 and NExT-QA show that QUAD consistently outperforms both memory-free and memory-rehearsal methods, achieving state-of-the-art performance. Surprisingly, our method, without any image exemplars, outperforms previous methods, which do require image storage.
**Acknowledgements.** This paper is supported by the French National Research Agency (ANR) in the framework of the JCJC project "BANERA under Grant ANR-24-CE23-4369, and was funded by the European Union's Horizon Europe research and innovation program under grant agreement No. 101120237 (ELIAS). It was also partially funded by Hi!PARIS Center on Data Analytics and Artificial Intelligence and was granted access to the HPC resources of IDRIS under the allocation AD011013860 made by GENCI.

## References

[1] Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, Albert Q. Jiang, Kartik Khandelwal, Timothée Lacroix, Guillaume Lample, Diego Las Casas, Thibaut Lavril, Teven Le Scao, Andy Lo, William Marshall, Louis Martin, Arthur Mensch, Pavankumar Muddireddy, Valera Nemychnikova, Marie Pellat, Patrick Von Platen, Nikhil Raghuraman, Baptiste Rozière, Alexandre Sablayrolles, Lucile Saulnier, Romain Sauvestre, Wendy Shang, Roman Soletskyi, Lawrence Stewart, Pierre Stock, Joachim Studnia, Sandeep Subramanian, Sagar Vaze, Thomas Wang, and Sophia Yang. Pixtral 12b, 2024. 2

[2] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In *ECCV*, 2018. 2, 6, 7

[3] Andrea Amaduzzi, Giuseppe Lisanti, Samuele Salti, and Luigi Di Stefano. Looking at words and points with attention: a benchmark for text-to-shape coherence. *arXiv preprint arXiv:2309.07917*, 2023. 3

[4] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In *ICCV*, 2015. 3

[5] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In *ICCV*, 2015. 2

[6] Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In *ECCV*, 2018. 6

[7] Arslan Chaudhry, Marc'Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. *arXiv preprint arXiv:1812.00420*, 2018. 3

[8] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet Kumar Dokania, Philip H. S. Torr, and Marc'Aurelio Ranzato. Continual learning with tiny episodic memories. *CoRR*, abs/1902.10486, 2019. 2, 3, 6, 7

[9] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K. Dokania, Philip H. S. Torr, and Marc'Aurelio Ranzato. On tiny episodic memories in continual learning. 2019. 2, 3, 6, 7

[10] Feng Cheng, Xizi Wang, Jie Lei, David Crandall, Mohit Bansal, and Gedas Bertasius. Vindlu: A recipe for effective video-and-language pretraining. In *CVPR*, 2023. 1, 2

[11] Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and-language tasks via text generation. 2021. 2

[12] Deepayan Das, Davide Talon, Massimiliano Mancini, Yiming Wang, and Elisa Ricci. One vlm to keep it learning: Generation and balancing for data-free continual visual question answering. *IEEE*, 2025. 4

[13] Riccardo Del Chiaro, Bartłomiej Twardowski, Andrew Bagdanov, and Joost van de Weijer. Ratt: Recurrent attention to transient tasks for continual image captioning. In *NeurIPS*, 2020. 3

[14] Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa. Learning without memorizing. In *CVPR*, 2019. 3, 4, 5, 1, 2

[15] Sayanton V Dibbo. Sok: Model inversion attack landscape: Taxonomy, challenges, and future roadmap. In *2023 IEEE 36th Computer Security Foundations Symposium (CSF)*. IEEE, 2023. 1

[16] Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In *ECCV*, 2020. 3

[17] Arthur Douillard, Alexandre Ramé, Guillaume Couairon, and Matthieu Cord. Dytox: Transformers for continual learning with dynamic token expansion. In *CVPR*, 2022. 3

[18] Beyza Ermis, Giovanni Zappella, Martin Wistuba, Aditya Rawal, and Cedric Archambeau. Memory efficient continual learning with transformers. *NeurIPS*, 2022. 3

[19] Clayton Fields and Casey Kennington. Vision language transformers: A survey. *arXiv preprint arXiv:2307.03254*, 2023. 2

[20] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. *arXiv preprint arXiv:2010.01412*, 2020. 3

[21] Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, and Jian Zhang. A unified continual learning framework with general parameter-efficient tuning. *arXiv preprint arXiv:2303.10070*, 2023. 3

[22] General Data Protection Regulation GDPR. General data protection regulation. *Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC*, 2016. 2

[23] Akash Ghosh, Arkadeep Acharya, Sriparna Saha, Vinija Jain, and Aman Chadha. Exploring the frontier of vision-language models: A survey of current methodologies and future directions. 2024. 3

[24] Nathan Godey, Éric de la Clergerie, and Benoît Sagot. Anisotropy is inherent to self-attention in transformers. *arXiv preprint arXiv:2401.12143*, 2024. 5
[25] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org. 2

[26] Dipam Goswami, Yuyang Liu, Bartłomiej Twardowski, and Joost van de Weijer. Fecam: Exploiting the heterogeneity of class distributions in exemplar-free continual learning. In NeurIPS, 2023. 1

[27] Claudio Greco, Barbara Plank, Raquel Fernández, and Rafaella Bernardi. Psycholinguistics meets continual learning: Measuring catastrophic forgetting in visual question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, 2019. Association for Computational Linguistics. 2

[28] Claudio Greco, Barbara Plank, Raquel Fernández, and Rafaella Bernardi. Psycholinguistics meets continual learning: Measuring catastrophic forgetting in visual question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. 3

[29] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet? In CVPR, 2018. 5

[30] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. 4

[31] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In CVPR, 2019. 3

[32] Tianyu Huai, Jie Zhou, Xingjiao Wu, Qin Chen, Qingchun Bai, Ze Zhou, and Liang He. Cl-moe: Enhancing multi-modal large language model with dual momentum mixture-of-experts for continual visual question answering. In CVPR, 2025. 4

[33] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017. 2

[34] Minsoo Kang, Jaeyoo Park, and Bohyung Han. Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation. In CVPR, 2022. 3, 5

[35] Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, et al. Measuring compositional generalization: A comprehensive method on realistic data. arXiv preprint arXiv:1912.09713, 2019. 2

[36] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 5

[37] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017. 6

[38] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017. 7

[39] Chris Kolb, Tobias Weber, Bernd Bischl, and David Rügamer. Deep weight factorization: Sparse learning through the lens of artificial symmetries. ICLR, 2025. 2

[40] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 2017. 5

[41] Canasai Kruengkrai and Junichi Yamagishi. Mitigating the diminishing effect of elastic weight consolidation. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4568–4574, Gyeongju, Republic of Korea, 2022. International Committee on Computational Linguistics. 2

[42] Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017. 2

[43] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and-language learning via sparse sampling. In CVPR, 2021. 2

[44] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023. 2

[45] Li Li, Jiawei Peng, Huiyi Chen, Chongyang Gao, and Xu Yang. How to configure good in-context sequence for visual question answering. In CVPR, 2024. 2

[46] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947, 2017. 3

[47] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935–2947, 2018. 3, 4

[48] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR, abs/1405.0312, 2014. 5

[49] Chi Liu, Tianqing Zhu, Jun Zhang, and Wanlei Zhou. Privacy intelligence: A survey on image privacy in online social networks. ACM Computing Surveys, 2022. 2

[50] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. 2

[51] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9): 1–35, 2023. 3

[52] David Lopez-Paz and Marc'Aurelio Ranzato. Gradient episodic memory for continual learning. NeurIPS, 2017. 3

[53] David Lopez-Paz and Marc'Aurelio Ranzato. Gradient episodic memory for continual learning, 2022. 6
[54] Yichen Lu, Mei Wang, and Weihong Deng. Augmented geometric distillation for data-free incremental person reid. In CVPR, 2022. 3

[55] Zheda Mai, Ruiwen Li, Hyunwoo Kim, and Scott Sanner. Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning. In CVPR, 2021. 4, 5

[56] Imad Eddine Marouf, Subhankar Roy, Enzo Tartaglione, and Stéphane Lathuilière. Weighted ensemble models are strong continual learners. In ECCV, 2024. 1

[57] James L McClelland, Bruce L McNaughton, and Randall C O'Reilly. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102(3):419, 1995. 1, 3

[58] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation. 1989. 1

[59] Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. An empirical investigation of the role of pre-training in lifelong learning. arXiv preprint arXiv:2112.09153, 2021. 3

[60] Nandita Naik, Christopher Potts, and Elisa Kreiss. Context-vqa: Towards context-aware and purposeful visual question answering. In ICCV, 2023. 2

[61] Mavina Nikandrou, Lu Yu, Alessandro Suglia, Ioannis Konstas, and Verena Rieser. Task formulation matters when learning continually: A case study in visual question answering. arXiv preprint arXiv:2210.00044, 2022. 2, 3, 5

[62] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raimon, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS. 2019. 5

[63] Francesco Pelosi, Saurav Jha, Andrea Torsello, Bogdan Raducanu, and Joost van de Weijer. Towards exemplar-free continual learning in vision transformers: an account of attention, functional and weight regularization. In CVPR, 2022. 5, 7, 1, 2

[64] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. 2020. 5

[65] Sahithya Ravi, Aditya Chinchure, Leonid Sigal, Renjie Liao, and Vered Shwartz. Vlc-bert: Visual question answering with contextualized commonsense knowledge. 2023. 3

[66] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl: Incremental classifier and representation learning. In CVPR, 2017. 3

[67] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. NeurIPS, 2015. 5

[68] Grzegorz Rypeść, Sebastian Cygert, Tomasz Trzcinski, and Bartłomiej Twardowski. Task-recency bias strikes back: Adapting covariances in exemplar-free class incremental learning. NeurIPS, 2025. 4, 5

[69] Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In ICLR, 2021. 3

[70] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In ECCV, 2022. 1

[71] Kunyu Shi, Qi Dong, Luis Goncalves, Zhuowen Tu, and Stefano Soatto. Non-autoregressive sequence-to-sequence vision-language models. In CVPR, 2024. 2

[72] James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In CVPR, 2023. 3

[73] Tejas Srinivasan, Ting-Yun Chang, Leticia Pinto Alva, Georgios Chochlakis, Mohammad Rostami, and Jesse Thomason. Climb: A continual learning benchmark for vision-and-language tasks. NeurIPS, 2022. 3

[74] Miao Tian, Yushu Zhang, Yongming Zhang, Xiangli Xiao, and Wenying Wen. A privacy-preserving image retrieval scheme with access control based on searchable encryption in media cloud. Cybersecurity, 2024. 2

[75] Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv:1905.09418, 2019. 5

[76] Nidhi Vyas, Shreyas Saxena, and Thomas Voice. Learning soft labels via meta learning. 2021. 4

[77] Timmy S. T. Wan, Jun-Cheng Chen, Tzer-Yi Wu, and Chu-Song Chen. Continual learning for visual search with backward consistent feature embedding. In CVPR, 2022. 2, 3, 6, 7, 8

[78] Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompt learning with pre-trained transformers: An occam's razor for domain incremental learning. NeurIPS, 2022. 3

[79] Zihan Wang, Jiuxiang Gu, Jason Kuen, Handong Zhao, Vlad Morariu, Ruiyi Zhang, Ani Nenkova, Tong Sun, and Jingbo Shang. Learning adaptive axis attentions in fine-tuning: Beyond fixed sparse attention patterns. In Findings of the Association for Computational Linguistics: ACL 2022, pages 916-925, Dublin, Ireland, 2022. Association for Computational Linguistics. 5

[80] Zhen Wang, Liu Liu, Yiqun Duan, Yajing Kong, and Dacheng Tao. Continual learning with lifelong vision transformer. In CVPR, 2022. 1

[81] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In ECCV, 2022. 1, 3

[82] Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In CVPR, 2022. 3
[83] Ziyang Wang, Yi-Lin Sung, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Unified coarse-to-fine alignment for video-text retrieval. In *ICCV*, 2023. 1

[84] Spencer Whitehead, Hui Wu, Heng Ji, Rogerio Feris, and Kate Saenko. Separating skills and concepts for novel visual question answering. In *CVPR*, 2021. 3

[85] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In *CVPR*, pages 9777-9786, 2021. 6

[86] Junbin Xiao, Pan Zhou, Tat-Seng Chua, and Shuicheng Yan. Video graph transformer for video question answering. In *ECCV*, 2022. 2

[87] Yangyang Xu, Yibo Yang, and Lefei Zhang. Multi-task learning with knowledge distillation for dense prediction. In *ICCV*, 2023. 5

[88] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via frozen bidirectional language models. *NeurIPS*, 2022. 1, 2

[89] Enneng Yang, Li Shen, Zhenyi Wang, Shiwei Liu, Guibing Guo, and Xingwei Wang. Data augmented flatness-aware gradient projection for continual learning. In *ICCV*, 2023. 3

[90] Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model for video localization and question answering. *arXiv preprint arXiv:2305.06988*, 2023. 2

[91] Razieh Nokhbeh Zaeem and K. Suzanne Barber. The effect of the gdpr on privacy policies: Recent progress and future promise. *ACM Trans. Manage. Inf. Syst.*, 2020. 2

[92] Jiliang Zhang, Chen Li, Jing Ye, and Gang Qu. Privacy threats and protection in machine learning. In *Proceedings of the 2020 on Great Lakes Symposium on VLSI*, 2020. 1

[93] Wenxuan Zhang, Paul Janson, Rahaf Aljundi, and Mohamed Elhoseiny. Overcoming generic knowledge loss with selective parameter update. In *CVPR*, 2024. 2

[94] Xiao Zhang, Yixiao Ge, Yu Qiao, and Hongsheng Li. Refining pseudo labels with clustering consensus over generations for unsupervised object re-identification. In *CVPR*, 2021. 4

[95] Xi Zhang, Feifei Zhang, and Changsheng Xu. Vqacl: A novel visual question answering continual learning setting. In *CVPR*, 2023. 1, 2, 3, 5, 6, 7, 8

[96] Yanzhe Zhang, Xuezhi Wang, and Diyi Yang. Continual sequence generation with adaptive compositional modules. In *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 3653-3667, Dublin, Ireland, 2022. Association for Computational Linguistics. 3

[97] Da-Wei Zhou, Hai-Long Sun, Han-Jia Ye, and De-Chuan Zhan. Expandable subspace ensemble for pre-trained model-based class-incremental learning. In *CVPR*, 2024. 1
# Ask and Remember: A Questions-Only Replay Strategy for Continual Visual Question Answering

## Supplementary Material

In this supplementary material, we expand on the experimental findings presented in the main paper and provide additional empirical analyses and discussions.

Section 7 outlines the ethical considerations of our work, while Section 8 discusses the limitations of our proposed method, QUAD. Sections 9 and 10 delve into the benefits of our attention consistency distillation (ACD) approach compared to conventional L1-based attention regularization. In Section 11, we present a detailed analysis of the computational and memory footprint of QUAD. Section 12 examines the impact of the balancing hyperparameter $\lambda$. Additional results using pretrained vision-language models (BLIP-2 and LLaVA-7B) are reported in Section 13.

Section 14 provides further insights into the out-of-answer-set problem encountered in continual VQA. Sections 15 and 16 describe the datasets used, task orderings, and evaluation protocol. Section 17 offers an extended analysis of the plasticity-stability trade-off on the NExT-QA dataset. Finally, Section 18 details the baseline methods used for comparison throughout our study.

## 7. Ethics Statement

Our method, QUAD, is designed to improve continual learning in Visual Question Answering (VQACL) while maintaining generalization and privacy through distillation using questions-only. We do not foresee any negative societal impact from this work, as it does not involve the generation of harmful or biased data. However, like any machine learning system, there remains a potential risk if it is applied unethically or without proper oversight. QUAD's design includes mechanisms to enhance privacy, reducing the storage of sensitive visual data. Despite this, its applicability beyond the specific datasets and tasks used in our experiments remains to be thoroughly tested, and we caution against the unconsidered deployment of the method in sensitive applications without further validation.

## 8. Limitations of QUAD

While QUAD effectively reduces storage requirements and enhances privacy by eliminating the need to store images, it may be suboptimal for tasks that heavily rely on detailed visual or spatial reasoning. Certain VQA tasks, such as object classification, fine-grained attribute recognition, or spatial relationships, inherently require access to visual information to retain critical knowledge from previous tasks. For instance, as shown in Fig. 4, QUAD struggles to maintain

performance on the 'type' task in VQAv2, which depends on visual cues, whereas it performs well on conceptually driven tasks like 'commonsense' reasoning.

Our findings suggest that question-only replay is particularly well-suited for constrained scenarios where privacy and storage efficiency are primary concerns. However, in settings where high fidelity in visual reasoning is essential, storing a subset of representative images may be necessary to preserve task-specific knowledge and improve overall performance. Future work could explore hybrid approaches that selectively retain visual information while leveraging question-based replay, striking a balance between efficiency and task-specific retention.

Furthermore, QUAD prevents storing original sensitive visual information, aligning with GDPR constraints, which permit data storage only when strictly necessary for the task. However, our approach specifically addresses storage-related privacy concerns and does not guarantee protection against attacks such as inversion attacks [15, 92].

## 9. Discussion about Attention Consistency Distillation

**Problem setup.** Consider a self-attention mechanism where the attention matrix at layer $l$, head $k$, for an input sequence $x$ at task $t$ is given by:

$$A_{l,k}^{t}(x) = \frac{Q_l K_l^T}{\sqrt{d}}, \qquad (6)$$

where $Q_l, K_l \in \mathbb{R}^{N \times d}$ are the query and key matrices at layer $l$, $d$ is the dimensionality of the attention keys, and $A_{l,k}^{t}(x) \in \mathbb{R}^{N \times N}$ represents the attention map at layer $l$ and head $k$.

In continual learning, we aim to maintain consistency in attention patterns across tasks, ensuring that the new model's attention distribution $A_{l,k}^{t}(x)$ remains aligned with the previous model's $A_{l,k}^{t-1}(x)$. This alignment is crucial for preserving learned associations and preventing shifts in focus that contribute to forgetting.

**L1 Regularization for Attention Alignment.** One widely adopted approach to constraining attention shift is L1 regularization [14, 63], which penalizes the absolute differences between attention maps:

$$\mathcal{L}_{\text{L1}} = \sum_{l \in \mathcal{S}} \sum_{k \in \mathcal{K}} \sum_{i,j} \left| A_{l,k}^{t}(x) - A_{l,k}^{t-1}(x) \right|. \qquad (7)$$

where $\mathcal{S}$ denotes the set of layers, and $\mathcal{K}$ represents the set of attention heads across layers.
<figure><img src="image_8.png" /><figcaption>Figure 7. Entropy Difference. Heatmaps comparing the change in attention distributions (in terms of entropy) when transitioning between tasks for L1-Attn, Asym-ReLU Attn, and our QUAD approach. Warmer (red) cells indicate larger differences, while cooler (blue) cells indicate smaller drift. Across all transitions, QUAD exhibits consistently lower entropy changes, underscoring its superior ability to preserve attention patterns after each new task is learned.</figcaption></figure>

The gradient of the L1 loss with respect to $A_{l,k}^t$ is given by:

$$ \frac{\partial \mathcal{L}_{\text{L1}}}{\partial A_{l,k}^t} = \text{sign}(A_{l,k}^t - A_{l,k}^{t-1}). \qquad (8) $$

However, a key limitation of prior L1-based approaches is that they operate directly on the raw query-key products, rather than on the normalized attention distributions obtained after applying softmax. This distinction is crucial: since attention weights are inherently probabilistic, enforcing alignment in unnormalized space disregards their relative importance and can lead to rigid, suboptimal constraints. Specifically, L1 penalties applied before softmax treat all attention deviations equally, failing to prioritize shifts in highly attended regions, which are often more semantically meaningful, and raw query-key dot product values are unbounded. Moreover, such methods impose sparse, discontinuous gradients, potentially hindering the model's ability to dynamically adapt to new knowledge [25, 39].

**Attention Consistency Distillation (ACD).** Instead of treating attention maps as raw numerical matrices, our ACD method interprets them as probability distributions and enforces alignment across tasks via cross-entropy as follows:

$$ A_k^t(x) = \text{Softmax} \left( \frac{QK^T}{\sqrt{d}} \right), \qquad (9) $$

To maintain attention consistency across tasks, we minimize the cross-entropy loss between the previous task's attention distribution $A_{l,k}^{t-1}(x)$ and the current one $A_{l,k}^t(x)$:

$$ \mathcal{L}_{\text{ACD}} = \sum_{l \in S} \sum_{k \in K} \sum_{i,j} -A_{l,k}^{t-1}(x) \log A_{l,k}^t(x), \qquad (10) $$

**Gradient of ACD Loss.** The gradient of the cross-entropy loss with respect to $A_{l,k}^t$ is:

$$ \frac{\partial \mathcal{L}_{\text{ACD}}}{\partial A_{l,k}^t} = -\frac{A_{l,k}^{t-1}}{A_{l,k}^t} + 1. \qquad (11) $$

Unlike L1 loss, which applies a uniform penalty to all deviations, cross-entropy scales the correction based on the importance of attended regions. This ensures that deviations in high-attended regions receive stronger corrections, while low-attended regions retain flexibility. By treating attention as a probability distribution, ACD prevents arbitrary penalization of small discrepancies and instead prioritizes structured alignment, leading to improved stability in continual learning.

# 10. Analysis of Attention Drift

To assess the effectiveness of QUAD in mitigating attention drift in continual VQA, we compare it to L1-Attention Regularization (L1-Attn) [14] and Asymmetric ReLU-Attention Regularization (Asym-ReLU Attn) [63] using two metrics: *Cross-Attention Coherence Drift*, and *Entropy Difference Drift*. These metrics quantify attention drift as the model learns new tasks, providing a comprehensive evaluation of each method's ability to maintain structured attention distributions across tasks:

* **Entropy Difference:** Measures the absolute difference between the entropy of two attention distributions (lower is better). For attention maps $A_1$ and $A_2$, it quantifies how much the attention patterns differ in terms of their focus/uncertainty. A value of 0.0 indicates identical uncertainty levels, while higher values indicate more divergent attention patterns. Formally defined as:

$$ \text{EntropyDiff}(A_1, A_2) = |\mathcal{H}(A_1) - \mathcal{H}(A_2)| $$

where $\mathcal{H}(A)$ is the entropy of attention distribution $A$:

$$ \mathcal{H}(A) = -\sum_{i,j} A \log_2(A) $$

Here, $A$ represents the attention map in the attention matrix. This metric is particularly useful for detecting changes in attention focus: low entropy indicates focused
<figure><img src="image_9.png" /><figcaption>Figure 8. Cross-Attention Coherence. Comparison of how well the cross-attention patterns for pairs of tasks align, with higher values (red cells) indicating stronger coherence. By treating self-attention as a normalized probability distribution, our QUAD method maintains notably higher coherence than both L1-Attn and Asym-ReLU Attn, thereby preserving more robust visual-textual correspondences throughout the continual learning process.</figcaption></figure>

attention on specific tokens, while high entropy indicates more distributed attention.

*   **Cross-Attention Coherence [3]:** Measures the similarity between two attention distributions by computing their normalized dot product (higher is better). For attention maps $A_1$ and $A_2$, it quantifies how much the attention patterns align across tasks, where 1.0 indicates perfect alignment and 0.0 indicates completely different attention patterns. Formally defined as:

$$ \text{Cross-AttnCoh}(A_1, A_2) = \frac{\sum A_1 \cdot A_2}{\sqrt{\sum A_1^2} \cdot \sqrt{\sum A_2^2}} $$

where $A_1$ and $A_2$ are the attention maps. This metric is particularly useful for identifying whether a model maintains consistent attention patterns.

Fig. 7 reveal QUAD's substantial advantage over L1-Attn and Asym-ReLU Attn in preserving attention distributions during task transitions. Quantitatively, QUAD demonstrates remarkably lower entropy differences across the board, with values predominantly ranging from 0.000 to 0.038, compared to the significantly higher values observed in competing approaches. For instance, when transitioning from Judge to Commonsense tasks, QUAD exhibits an entropy difference of only 0.015, while L1-Attn and Asym-ReLU Attn show values of 0.091 and 0.040 respectively—a reduction of up to 83.5%. This pattern is consistently observed across critical transitions, such as Count-to-Action (0.013 for QUAD vs. 0.039 for L1-Attn) and Subcategory-to-Causal (0.057 for QUAD vs. 0.113 for L1-Attn). The average entropy difference across all transitions for QUAD (0.038) is substantially lower than both L1-Attn (0.044) and Asym-ReLU Attn (0.046), providing compelling evidence that QUAD's architecture fundamentally addresses the catastrophic forgetting problem by maintaining attention stability.

Fig. 8 demonstrate QUAD's superior ability to maintain consistent attention patterns across different tasks compared

to baseline approaches. Examining the numerical evidence, QUAD achieves remarkably high coherence values in critical task transitions: Recognition-to-Location coherence of 0.642 versus 0.646 for L1-Attn and 0.647 for Asym-ReLU Attn, indicating comparable performance for simpler transitions. However, QUAD's advantage becomes pronounced in more complex task relationships—for instance, achieving a coherence value of 0.333 for Type-to-Subcategory transitions compared to 0.318 for L1-Attn and 0.306 for Asym-ReLU Attn, representing a substantial 4.7-8.8% improvement. Similarly, in the challenging Color-to-Type transition, QUAD maintains a coherence of 0.255 versus 0.248 for L1-Attn and 0.239 for Asym-ReLU Attn. Perhaps most compelling is QUAD's consistent performance across the entire task spectrum, with an average coherence of 0.323, marginally outperforming both L1-Attn (0.320) and Asym-ReLU Attn (0.328). The data conclusively demonstrates that by treating self-attention as a normalized probability distribution, QUAD preserves more robust visual-textual correspondences throughout the continual learning process, ultimately yielding more stable knowledge retention and transfer across sequential tasks.

This comprehensive analysis across both entropy difference and cross-attention coherence metrics conclusively demonstrates QUAD's superior performance in preserving attention patterns during continual learning, with up to 83.5% reduction in entropy shifts and 8.8% improvement in coherence for complex transitions.

## 11. Computational Analysis

Efficient memory and storage management is crucial for continual VQA, where scalability is a key challenge. This section analyzes storage requirements, computational complexity, and GPU memory usage of our text-only replay approach compared to image-based methods. By storing only past task questions, we significantly reduce storage complexity from $O(N \cdot (I + L_q + L_a))$ to $O(N \cdot L_q)$, where N
is the number of stored samples, *I* is the image size, and *L*<sub>q</sub> and *L*<sub>a</sub> represent the question and answer lengths in bits.

In terms of *GPU memory usage*, question-only replay has a minimal impact since the number of processed input pairs remains the same. The primary reduction stems from loading fewer images, but this accounts for less than 5% of the total memory footprint, which is dominated by gradients, weights, and activations. This makes our approach particularly appealing in scenarios where storage is constrained but GPU memory availability remains a concern.

From a *computational complexity* perspective, our method does not introduce any additional overhead. The computational cost remains unchanged when processing images from past or current tasks. The forward and backward passes are identical, ensuring that our approach maintains the same efficiency while significantly improving storage scalability.

This analysis validates our design choices, demonstrating that question-only replay can achieve competitive performance while substantially reducing storage requirements. This efficiency makes it highly scalable and practical for real-world deployment.

## 12. Effect of $\lambda$

We investigate the sensitivity of our model to the balancing coefficient $\lambda$ in Fig. 9, which governs the trade-off between adaptation to new tasks (plasticity) and retention of prior knowledge (stability) in our QUAD framework. The results demonstrate that performance peaks at $\lambda$ = 0.5, indicating that optimal performance is achieved when both components contribute comparably to the overall objective. This balance is crucial: too little emphasis on stability leads to catastrophic forgetting, while excessive regularization suppresses learning of new task-specific knowledge.

<figure><img src="image_10.png" /><figcaption>Figure 9. Sensitivity to $\lambda$. The plot demonstrates the relationship between $\lambda$ and average precision (AP) on VQAv2.</figcaption></figure>

Notably, QUAD consistently outperforms the standard VQACL baseline for $\lambda$ ≥ 0.4, underscoring the effectiveness of our tailored stability components—question-only replay ($\mathcal{L}_{QR}$) and attention consistency distillation ($\mathcal{L}_{ACD}$). The synergy between these modules allows QUAD to mitigate forgetting despite the absence of past task images, a key constraint in our continual learning setting. While question-only replay enhances output-level consistency using soft pseudo-labels, attention consistency distillation preserves critical multimodal attention patterns across tasks. Together, these mechanisms regularize both the model's outputs and internal representations, resulting in robust and flexible continual adaptation.

## 13. Pre-trained models/VQA architectures

We extend our evaluation to recent continual learning approaches—CL-MoE [32] and GaB [12]—using pretrained vision-language models BLIP-2 and LLaVA (Tabs. 5, 6). On BLIP-2, QUAD achieves the highest average precision (AP = 50.27) and lowest forgetting (1.04), outperforming both VQACL and GaB variants. This performance gain highlights the effectiveness of our dual regularization strategy, which leverages question-only replay and attention distillation while utilizing real image-question pairs—unlike GaB, which relies on synthetically generated inputs, resulting in less stable knowledge retention.

On LLaVA, QUAD demonstrates consistent improvements over both sequential fine-tuning (Vanilla) and VQACL across all subtypes of questions, notably in compositional (62.15). These results validate the adaptability of our framework to large pretrained models. While CL-MoE surpasses all methods in AP (52.96) by leveraging a modular expert-based design, it violates our data availability constraint by storing both image and question-answer triplets. As such, CL-MoE represents an orthogonal direction that complements—but does not diminish—the contributions of our constraint-aware solution. Our results collectively confirm the robustness and generalizability of QUAD across architectures while strictly adhering to realistic memory constraints.

<table><caption>Table 5. BLIP-2 performance. Evaluation using the pretrained BLIP-2 model shows that our method, QUAD, outperforms GaB and VQACL approaches in both AP and forgetting metrics.</caption><thead><tr><th>Method</th><th>Memory</th><th>Memory size</th><th>AP (↑)</th><th>Forget (↓)</th></tr></thead><tbody><tr><td>Vanilla</td><td>-</td><td>-</td><td>41.29</td><td>15.98</td></tr><tr><td>VQACL</td><td>?</td><td>5000</td><td>49.80</td><td>1.18</td></tr><tr><td>GaB-classifier</td><td>?</td><td>5000</td><td>47.65</td><td>3.61</td></tr><tr><td>GaB-clustering</td><td>?</td><td>5000</td><td>48.40</td><td>1.40</td></tr><tr><td>QUAD</td><td>?</td><td>5000</td><td>50.27</td><td>1.04</td></tr></tbody></table>
<table><caption>Table 6. **LLaVA-7B Performance.** Evaluation using the pre-trained LLaVA-7B model.</caption><thead><tr><th>Method</th><th>Memory</th><th>Mem. Size</th><th>Rec.</th><th>Loc.</th><th>Jud.</th><th>Com.</th><th>Cou.</th><th>Act.</th><th>Col.</th><th>Typ.</th><th>Sub.</th><th>Cau.</th><th>AP (†)</th></tr></thead><tbody><tr><td>Vanilla</td><td>-</td><td>-</td><td>19.25</td><td>14.81</td><td>54.59</td><td>56.97</td><td>24.23</td><td>46.20</td><td>27.58</td><td>26.09</td><td>36.47</td><td>18.89</td><td>32.51</td></tr><tr><td>VQACL</td><td>●</td><td>5000</td><td>34.14</td><td>32.19</td><td>66.15</td><td>63.00</td><td>33.01</td><td>60.91</td><td>34.64</td><td>38.48</td><td>47.94</td><td>24.42</td><td>43.49</td></tr><tr><td>CL-MoE</td><td>●</td><td>5000</td><td>46.50</td><td>37.18</td><td>75.22</td><td>71.39</td><td>40.90</td><td>69.54</td><td>43.66</td><td>52.68</td><td>55.55</td><td>20.74</td><td>52.96</td></tr><tr><td>QUAD</td><td>●</td><td>5000</td><td>35.87</td><td>33.17</td><td>66.93</td><td>62.15</td><td>34.09</td><td>61.28</td><td>35.03</td><td>38.87</td><td>48.66</td><td>25.53</td><td>44.16</td></tr></tbody></table>

## 14. Out-of-Answer-Set Problem Evaluation

To empirically analyze the *out-of-answer-set problem*, we designed a controlled continual learning experiment within the VQACL setting. Our objective was to demonstrate how sequential fine-tuning without appropriate regularization leads to catastrophic forgetting, causing the model to misclassify previous-task questions by selecting answers from the current task's answer space. This phenomenon, which we note is related to *class recency bias* in Class-Incremental Learning (CIL) [55, 68], arises when the model disproportionately favors responses from the most recently learned task, even when answering questions about past tasks.

To assess this, we structured the training process into three sequential tasks: counting, action recognition, and color identification from VQAv2 dataset. Each task contained a fixed set of possible answers:

* **Counting Task:** The model learned to predict numerical answers from the set {One, Two, Three}.

* **Action Recognition Task:** The model answered binary yes/no questions from the set {Yes, No}.

* **Color Identification Task:** The model identified object colors from the set {Red, Blue, Green}.

At each stage, the model was trained on the current task while being evaluated on all previous tasks to measure forgetting-induced answer space drift. For evaluation, we tested the model on 10 questions per task and verified whether its predicted answers belonged to the corresponding *expected answer set* of the task. A misclassification was recorded if a model produced an answer outside the defined set, indicating that it had lost the ability to correctly respond using prior knowledge.

We compared two settings: (1) Sequential Fine-tuning (No Replay), where the model was updated on each new task without access to previous data, and (2) QUAD (Ours), which incorporated question-only replay to retain past knowledge with attention distillation.

To quantify the severity of the *out-of-answer-set problem*, we analyzed the *prediction distribution shift* across tasks using confusion matrices. Specifically, we examined whether the model, when tested on past-task questions, incorrectly answered using responses restricted to the most recent task. For instance, a model fine-tuned on the *color task* but evaluated on *counting questions* was expected to misclassify numerical questions as colors (e.g., responding

“Red” instead of “Two”). Similarly, after training on *action* task, past counting questions were likely to be misclassified as “Yes” or “No”.

The results, visualized in Fig. 2, revealed that sequential fine-tuning caused a stark shift in the prediction distribution, with nearly all responses aligning with the most recent task's answer set. In contrast, QUAD mitigated this effect by preserving prior-task responses, demonstrating the effectiveness of question-only replay in preventing catastrophic forgetting without requiring image exemplars.

## 15. Detailed Description of the VQACL Setting

This section provides a detailed overview of the Visual Question Answering Continual Learning (VQACL) setting, as introduced by [95]. The VQACL setting is designed to test a model's ability to generalise and retain knowledge across a sequence of tasks involving both visual and linguistic modalities, with a particular focus on compositional generalisation and knowledge retention.

The VQACL setting is organised into a two-level hierarchy of tasks that challenge both the visual and linguistic capabilities of the model.

* **Linguistically-Driven Tasks.** At the higher level, the VQACL setting comprises a series of linguistically-driven tasks, denoted as $T^1, \dots, T^T$, where $T$ represents the total number of tasks. Each task focuses on a specific reasoning skill, such as counting or color identification, and is characterized by a particular type of question. For example, a task focused on counting might involve questions beginning with "How many" or "What number". In our experiments, the VQAv2 dataset consists of $T = 10$ such tasks, while the NExT-QA dataset includes $T = 8$ tasks.

* **Visually-Driven Subtasks.** Nested within each linguistically-driven task are a series of visually-driven subtasks $S_1^t, \dots, S_K^t$. Each visually-driven subtask is associated with a specific object group $G_k$, formed by partitioning the total set of object classes $\{c_i\}_{i=1}^C$ into $K$ groups. These groups are then randomly assigned to different subtasks within each linguistic-driven task. In our implementation, both the VQAv2 and NExT-QA datasets are divided into $K=5$ visual subtasks, covering a total of $C=80$ object classes, following the categorization used in the COCO dataset [48].

* **Novel Composition Testing.** The VQACL setting also includes a novel composition testing process, designed to evaluate the model's compositional generalization abilities—its capacity to apply learned concepts to new combinations of objects and questions.

**Training and Testing Procedure.** During training, the model is exposed to a subset of the visual-driven subtasks within each linguistically-driven task. Specifically, one
<table><caption>Table 7. Linguistic-driven task statistics of VQA v2 in the VQACL setting. Stan. Test denotes the standard test set.</caption><thead><tr><th>Task</th><th>Train</th><th>Val</th><th>Stan. Test</th><th>Examples</th></tr></thead><tbody><tr><td>Recognition</td><td>131,478</td><td>5,579</td><td>5,628</td><td>What is on the floor? What does the sign say?</td></tr><tr><td>Location</td><td>12,580</td><td>611</td><td>611</td><td>Where is the giraffe? Where are the people standing?</td></tr><tr><td>Judge</td><td>160,179</td><td>7,126</td><td>7,194</td><td>Is the baby playing ball? Are the windows big?</td></tr><tr><td>Commonsense</td><td>25,211</td><td>1,114</td><td>1,100</td><td>Do the elephants have tusks? Do the dogs know how to swim?</td></tr><tr><td>Count</td><td>62,156</td><td>2,651</td><td>2,658</td><td>How many beds? How many seats are there?</td></tr><tr><td>Action</td><td>33,633</td><td>1,498</td><td>1,373</td><td>Are they drinking wine? Is the person flying?</td></tr><tr><td>Color</td><td>50,872</td><td>2,322</td><td>2,192</td><td>What color is the bedspread? What color are the gym shoes?</td></tr><tr><td>Type</td><td>23,932</td><td>1,119</td><td>1,089</td><td>What type of building is this? What type of animal is shown?</td></tr><tr><td>Subcategory</td><td>31,594</td><td>1,477</td><td>1,416</td><td>What brand is the umbrella? What brand are his shoes?</td></tr><tr><td>Causal</td><td>5,868</td><td>231</td><td>200</td><td>Why does he have glasses on? Why is the dog jumping?</td></tr></tbody></table>

<table><caption>Table 8. Linguistic-driven task statistics of NExT-QA in the VQACL setting. Stan. Test denotes the standard test set. CW: CausalWhy; TN: TemporalNext; TC: TemporalCurrent; DL: DescriptiveLocation; DB: DescriptiveBinary; DC: DescriptiveCount; DO: DescriptiveOther; CH: CausalHow.</caption><thead><tr><th>Task</th><th>Train</th><th>Val</th><th>Stan. Test</th><th>Examples</th></tr></thead><tbody><tr><td>CW</td><td>13,552</td><td>1,928</td><td>3,333</td><td>Why is the lady sitting down? Why is the baby's hair wet?</td></tr><tr><td>TN</td><td>5,685</td><td>895</td><td>1,399</td><td>What does the baby do after picking up the toy? What did the lady do after adjusting the shirt?</td></tr><tr><td>TC</td><td>4,797</td><td>663</td><td>1,165</td><td>What event is happening? What sport is the man doing?</td></tr><tr><td>DL</td><td>1,942</td><td>295</td><td>482</td><td>Where are the two people dancing? Where is this video taken?</td></tr><tr><td>DB</td><td>2,928</td><td>277</td><td>495</td><td>Is the baby able to walk? Does the girl cry?</td></tr><tr><td>DC</td><td>1,378</td><td>192</td><td>365</td><td>How many babies are there? How many dogs are there?</td></tr><tr><td>DO</td><td>2,549</td><td>356</td><td>672</td><td>What season is this? What does the man use to stir the food in the pan?</td></tr><tr><td>CH</td><td>4,400</td><td>683</td><td>1,174</td><td>How did the singer project her voice? How did the boy in the box move forward?</td></tr></tbody></table>

visual-driven subtask $S_k^v$ is randomly excluded from the training phase for each linguistic-driven task. This excluded subtask is reserved for testing and serves as a novel composition, where the model must answer questions about unseen combinations of objects and reasoning skills.

**Cross-Validation and Fair Testing.** To ensure a fair evaluation of the model's generalization capabilities, the VQACL setting employs a K-fold object-independent cross-validation process. This involves repeating the training and testing procedure K times, each time excluding a different visual-driven subtask. This ensures that the model encounters all object classes across different folds, thereby providing a comprehensive assessment of its ability to generalize to new combinations of objects and tasks.

**Continual Learning Challenges.** The VQACL setting presents a significant challenge for continual learning models, requiring them to balance the retention of knowledge from previously learnt tasks (stability) with the ability to adapt to new, continually arriving tasks (plasticity). By structuring tasks to involve both new and previously en-

countered concepts, the VQACL setting effectively tests the model's ability to minimize catastrophic forgetting while enabling knowledge transfer across tasks.

## 16. Details of Evaluation Datasets

In this section, we provide a detailed overview of the two datasets used in our evaluation: VQA v2 and NExT-QA. Each dataset has been carefully structured into different tasks, which are used to evaluate the performance of our continual learning models.

We summarize the statistics of each dataset, focusing on both linguistic and object-related tasks. Tables 7 and 8 (previously described) present the linguistic-driven task breakdown, including categories such as *Recognition*, *Commonsense*, *Count*, and others.

Additionally, we grouped the objects in each dataset into five distinct object groups to facilitate better understanding and comparison of the models' object recognition capabilities. Tables 9 and 10 offer a detailed breakdown of the
<table><caption>Table 9. Detailed information about the five object groups in VQA v2.</caption><thead><tr><th>Task</th><th>Objects</th></tr></thead><tbody><tr><td>Group 1</td><td>hot dog, fork, orange, snowboard, potted plant, person, toilet, laptop, surfboard, bench, bus, dog, knife, pizza, handbag, bicycle</td></tr><tr><td>Group 2</td><td>horse, cell phone, elephant, boat, zebra, apple, stop sign, microwave, spoon, cup, skateboard, tie, umbrella, sandwich, bear</td></tr><tr><td>Group 3</td><td>donut, truck, frisbee, giraffe, dining table, motorcycle, parking meter, car, oven, airplane, bed, sheep, baseball bat</td></tr><tr><td>Group 4</td><td>skis, baseball glove, tennis racket, tv, traffic light, kite, cake, keyboard, bottle, remote, bird, carrot</td></tr><tr><td>Group 5</td><td>suitcase, couch, broccoli, cow, fire hydrant, chair, mouse, cat, banana, wine glass, backpack, bowl, sports ball, train</td></tr></tbody></table>

<table><caption>Table 10. Detailed information about the five object groups in NExT-QA.</caption><thead><tr><th>Task</th><th>Objects</th></tr></thead><tbody><tr><td>Group 1</td><td>bicycle, camel, bat, microwave, snake, sofa, traffic light, hamster/rat, chicken, oven, stop sign, vegetables, skateboard, bird, toilet, racket</td></tr><tr><td>Group 2</td><td>crab, camera, lion, ball/sports ball, crocodile, screen/monitor, baby walker, cat, squirrel, frisbee, cattle/cow, sheep/goat, adult, scooter, electric fan, stool</td></tr><tr><td>Group 3</td><td>piano, watercraft, kangaroo, train, fruits, pig, suitcase, bear, tiger, bench, elephant, motorcycle, horse, snowboard, surfboard, handbag</td></tr><tr><td>Group 4</td><td>ski, stingray, antelope, toy, child, duck, guitar, dish, fish, cake, turtle, leopard, laptop, panda, table, cup</td></tr><tr><td>Group 5</td><td>penguin, faucet, car, bottle, bus/truck, aircraft, baby, bread, baby seat, cellphone, sink, rabbit, backpack, chair, dog, refrigerator</td></tr></tbody></table>

objects associated with each group in VQA v2 and NExT-QA, respectively. This categorization will aid in analyzing how the models perform across different object categories.

These two datasets, each structured uniquely in terms of linguistic tasks and object types, allow us to rigorously assess the models in varied real-world scenarios. Together, these benchmarks enable a comprehensive evaluation of the continual learning approaches proposed in this work.

## 17. Extended Analysis of Plasticity/Stability Trade-Off

Fig.10 compares the impact of three continual learning strategies on performance across tasks in the NExT-QA dataset. The sequential finetuning baseline (left) demonstrates severe forgetting, with consistently low off-diagonal values. Specifically, tasks like temporal reasoning (TN and TC) exhibit the worst performance, as these tasks require advanced reasoning over time sequences, which is inherently challenging for the model.

Introducing pseudo-label distillation through $\mathcal{L}_{PL}$ (center) mitigates the issue of forgetting by enforcing output consistency with the previous model. This results in improved cross-domain retention, particularly in easier tasks like 'DB' and 'DL'. However, its performance on complex tasks such as "DO" (Descriptive Others) and 'CH' (Causal How) remains suboptimal, as these tasks require the model to maintain intricate visual-linguistic relationships, which $\mathcal{L}_{PL}$ alone struggles to address.

Our method, QUAD (right), achieves the highest overall performance by combining pseudo-labeling with attention consistency distillation. This dual mechanism effectively balances stability and plasticity, as evidenced by the consistently high diagonal values and substantial improvements in

off-diagonal cross-domain generalization. Notably, QUAD performs significantly better on retraining prior knowledge (row 6, 7). The results underscore the strength of QUAD in preserving visual-linguistic associations and mitigating the *out-of-answer-set problem* across tasks in NExT-QA.

## 18. Continual Learning Methods

We assess and benchmark five prominent continual learning methods, encompassing two regularization techniques (EWC [38], MAS [2]) and three rehearsal-based methods (ER [9], DER [8], VS [77], and VQACL[95]). To ensure a consistent evaluation, all methods are implemented using their official codebases and integrated into the same transformer backbone as described in Section 5.1.

EWC [38] is a regularization method designed to preserve knowledge of prior tasks by selectively reducing updates on critical parameters. This is achieved by leveraging the Fisher Information Matrix, which quantifies the importance of parameters and incorporates an auxiliary L2 loss between significant parameters from old and new tasks.

MAS [2] similarly applies regularization, aiming to prevent significant changes to parameters vital for previous tasks by introducing an L2 loss. In contrast to EWC, MAS measures the sensitivity of the output with respect to parameter perturbations to estimate parameter importance.

ER [9] is a rehearsal method that utilizes a fixed-size memory buffer, where visited examples are stored and randomly sampled for retraining. In line with our approach, the memory size for ER is fixed at 5,000 for VQA v2 and 500 for NExT-QA. Given its simplicity and effectiveness, ER serves as the baseline for our proposed method.

DER [8] is another rehearsal technique that employs reservoir sampling to manage memory, ensuring every vis-
<figure><img src="image_11.png" /><figcaption>Figure 10. Comparison of feature distillation methods on NExT-QA. Each matrix shows the performance of a model trained on tasks (rows) and evaluated on tasks (columns). The diagonal (highlighted in orange) represents in-domain performance, while off-diagonal elements show cross-domain generalization. Higher values (darker colors) indicate better performance.</figcaption></figure>

ited sample has an equal chance of being stored. DER also incorporates a dark knowledge distillation strategy, which aims to align the network's outputs with logits recorded during training, thus encouraging consistency in responses to prior examples. In our experiments, DER also utilizes memory sizes of 5,000 for VQA v2 and 500 for NExT-QA.

VS [77] is a rehearsal-based method that emphasizes feature consistency between current and past data. To address forgetting, VS introduces two losses: a neighbor-session model coherence loss and an inter-session data coherence loss. For more details, we refer readers to Wan et al. [77]. The memory size for VS is similarly set to 5,000 for VQA v2 and 500 for NExT-QA.

VQACL [95] represents a rehearsal-based approach, incorporating a prototype module to learn both task-specific and invariant features, facilitating robust and generalizable representations for VQA tasks.