SiRA: Sparse Mixture of Low Rank Adaptation
Authors: name: Yun Zhu, name: Nevan Wichers, name: Chu-Cheng Lin, name: Xinyi Wang, name: Tianlong Chen, name: Lei Shu, name: Han Lu, name: Canoee Liu, name: Liangchen Luo, name: Jindong Chen, name: Lei Meng

Parameter Efficient Tuning has been an prominent approach to adapt the Large
Language Model to downstream tasks. Most previous works considers adding the
dense trainable parameters, where all parameters are used to adapt certain
task. We found this less effective empirically using the example of LoRA that
introducing more trainable parameters does not help. Motivated by this we
investigate the importance of leveraging "sparse" computation and propose SiRA:
sparse mixture of low rank adaption. SiRA leverages the Sparse Mixture of
Expert(SMoE) to boost the performance of LoRA. Specifically it enforces the top
k experts routing with a capacity limit restricting the maximum number of
tokens each expert can process. We propose a novel and simple expert dropout on
top of gating network to reduce the over-fitting issue. Through extensive
experiments, we verify SiRA performs better than LoRA and other mixture of
expert approaches across different single tasks and multitask settings.