SiRA: Sparse Mixture of Low Rank Adaptation
Authors: Yun Zhu, Nevan Wichers, Chu-Cheng Lin, Xinyi Wang, Tianlong Chen, Lei Shu, Han Lu, Yinxiao Liu, Liangchen Luo, Jindong Chen, Lei Meng


Reviewer paper summaries: This paper proposes a sparse mixture of expert variant of LoRA. The proposed model can achieve better performance than LoRA and other baselines by using sparse and dynamic computation. Further analysis suggests that the proposed model can provide more orthogonal low rank sub-spaces than others.
Summary of strengths: - This paper proposes a straightforward method to conduct the parameter efficient tuning.
- This paper is clear and easy to follow.
Summary of weaknesses: - The proposed model is only tested on one base model.
- The proposed components look more like engineering practice than technical contributions.
- Typos and grammar issues.
Comments, suggestions and typos: Comments:

- One major concern about this paper is that the proposed components are only tested in one base model. It would be better if more large language models can be incorporated.

- The proposed components are more like engineering practice rather than technical merits. More illustrations about the motivation for these components should be included.

- More contents should be added to illustrate how to balance the auxiliary loss and the training loss.

Typos and grammar issues

1. On Line 070, “non trivial” should be “non-trivial”.
2. On Line 180, “wasn’t” should be “was not”.
3. The captions of Tables 1, 5, and 6 do not end with a period.
Overall assessment scores: 2.5