CluMo: Cluster-based Modality Fusion Prompt for Continual Learning in Visual Question Answering
Authors: Yuliang Cai, Mohammad Rostami


Reviewer paper summaries: The paper proposes CluMo, a multimodal prompt learning approach for continual learning in VQA.
The core idea is to get separate image and text prompt keys by clustering image and text representations, which are then combined to index a multimodal learnable prompt. 
Prompt learning is combined with knowledge distillation using ALBEF as the frozen pretrained backbone.
The method is evaluated on the CLOVE benchmark showing improved performance against baselines.
Summary of strengths: 1. The method shows consistent performance improvement over previous prompt-learning baselines.
2. The idea of learning multimodal prompts corresponding to combinations of image and text prompt keys has potential for multimodal continual learning.
Summary of weaknesses: 1. The method is evaluated only with the ALBEF backbone, which consists of an image encoder, a text encoder and a multimodal encoder. It is unclear how generalizable this method is to other architectures.
2. The authors should compare, at least in the Related Work section, with the previous multimodal prompt learning work: Decouple Before Interact: Multi-Modal Prompt Learning for Continual Visual Question Answering, (Qian, Zi, et al., ICCV 2023)
3. Although the authors provide several ablations, some hyperparameter choices remain underexplored: 

a) How does the prompt length $L_p$ affect performance, and is there potentially a trade-off between the prompt length and the number of prompts?

b) According to Algorithm 1, the prompt keys, which are the cluster means, are recomputed using the random samples from each batch. How sensitive is the performance to batch size? 

4. The claims about CulMo showing improved forward transfer are not adequately supported. The authors suggest that the results in Figure 4 indicate forward transfer based on the performance of the first learned task after training for each task. However, forward transfer refers to a model's ability to learn a new task using its past knowledge.
Comments, suggestions and typos: Questions / Suggestions
1. Since the encoder parameters remain frozen, would performance change if you perform K-means clustering using all task data at the beginning of each task? Similarly, instead of overwriting the clusters with each new batch, would it be reasonable to use a moving average of cluster means instead?
2. In Algorithm 1, I would suggest using the notation $K_v$ instead of $P_i$ and $K_t$ instead of $P_t$ to refer to the Image and Text Prompt Keys for consistency and to avoid confusion with the prompts.

Typos
- Figure 4: "DualCluster" instead of "CluMo"
- Line 549: "1and 00 prompt pool sizes" instead of "and 100 prompt pool sizes" 
- Line 715: "DuamPrompt" instead of "DualPrompt"
- Captions for Tables 2, 3, 4 should be placed below the table (https://acl-org.github.io/ACLPUB/formatting.html)
Overall assessment scores: 2.5