ELDER: Enhancing Lifelong Model Editing with Mixture-of-LoRA
Authors: Jiaang Li, Quan Wang, Zhongnan Wang, Yongdong Zhang, Zhendong Mao


Reviewer paper summaries: This paper presents ELDER (Enhancing Lifelong moDel Editing with mixtuRe of Low-Rank Adapter), a novel approach for lifelong model editing of large language models (LLMs). The key contributions are:
- A mixture-of-LoRA (Low-Rank Adapter) structure that allows for continuous and smooth association between edit data and adapters, improving robustness and generalization.
- A guided loss function to help align adapter allocations with edit semantics.
- A deferral mechanism to retain original LLM capabilities on general tasks after editing.
The authors demonstrate ELDER's effectiveness through experiments ontwo datasets (ZsRE and COUNTERFACT). They show improved performance in reliability, generalization, and retention of general abilities compared to existing methods.
Summary of strengths: - The mixture-of-LoRA approach is an innovative solution to the challenges of lifelong model editing, addressing issues of robustness and generalization that plague existing discrete mapping methods.
- The paper provides extensive experiments comparing ELDER to multiple baselines across different models and datasets, demonstrating consistent improvements.
- ELDER shows better scalability to longer edit sequences and improved editing efficiency compared to existing methods. Their deferral mechanism also maintains the LLM's performance on general tasks after editing, which is crucial for practical applications.
Summary of weaknesses: - While the empirical results are strong, the paper could benefit from a deeper theoretical analysis of why the mixture-of-LoRA approach works so well for this task.
- The paper primarily focuses on evaluating generalization through rephrased edits. It would be valuable to see how ELDER performs on more complex data containing edited knowledge.
- The paper doesn't thoroughly discuss the potential increased computational costs of the mixture-of-LoRA approach compared to simpler editing methods, especially during inference.
Comments, suggestions and typos: - Consider adding a brief case study showcasing ELDER's performance on a real-world scenario of evolving knowledge.
- It would be interesting to see an analysis of how the number of LoRAs (N) and the top-k selection affect performance and computational requirements.
- The paper could benefit from a more detailed ablation study exploring the individual contributions of the mixture-of-LoRA structure, guided loss, and deferral mechanism.
- It would be interesting to see future work on potential applications of ELDER beyond factual knowledge editing, such as in personalization or domain adaptation scenarios.
Overall assessment scores: 4