Leave the Bias in Bias: Mitigating the Label Noise Effects in Continual Visual Instruction Fine-Tuning
Authors: Xiaoyu Tan, Teqi Hao, Xihe Qiu, Shaojie Shi, Yuan Cheng, Wei Chu, Xu Yinghui, Yuan Qi


Reviewer paper summaries: * This paper proposes a framework LeaBnB ("Leave the Bias in Bias") for continuous visual instruction fine-tuning (CVIF) to mitigate the impact of label noise introduced during the collection of domain-specific data.
* Specifically, the Polyak averaging is applied in projection layers to reduce overfitting. In addition, LLMs' bias understanding ability is utilized to recognize the bias instruction data.
* The experimental results in general VQA tasks and medical VQA tasks demonstrate that the proposed method is robust to different levels of label noise.
Summary of strengths: * The experiments show the robustness of the proposed method against different levels of label noise.
Summary of weaknesses: 1. Further experiments on the benefits of the proposed method in bias learning are needed. In my opinion, higher accuracy in VQA tasks does not reflect such benefits much.
2. More case studies are supposed to be presented to demonstrate the effectiveness of the proposed method.
3. The implementation details are not elaborated. Besides, the code, prompts, and fine-tuned models are not promised to be released.
Comments, suggestions and typos: **Comments**:

1. See "Summary Of Weaknesses."
2. Line 257: "we introduced different levels of label noise into these datasets." How exactly are different levels of label noise introduced?
3. The intuition for adopting the Polyak Averaging is somewhat vague.

**Suggestions**:

1. Format: The captions should be below the table.
2. Presentation: This work can be extended to an 8-page long paper. Please consider including the content in the Appendix. Streamline the Related Work section and add more analytical experiments.
Overall assessment scores: 3