ST-LoRA: Space- and Time-based extension of LoRA
Authors: Evgeny Izutov, Bevzyuk Semyon


Reviewer paper summaries: This paper proposes two ways to improve LoRA: (1) Periodically merge LoRA back to the pretrained weight during finetuning; (2) Use Mixture-of-Expert to train multiple LoRA modules at the same time. Through experiments with T5-small and BART-Largs on XSUM, the proposed method, ST-LoRA, is slightly better than the strong baselines, like ReLoRA, DeltaLoRA and AdaMix.
Summary of strengths: 1. The paper is easy to follow and the motivation is clear.
Summary of weaknesses: 1. ***Lack of thorough experiments***. Only two out-dated LLMs, T5-small and BART-large, and one benchmark, XSUM, are used. Although a larger LLM on instruction tuning dataset is shown in Table 5, I doubt the results. Because a perplexity of >200 means the experiments are no usable.

2. ***Lack of Novelty***. ST-LoRA is simply a combination of ReLoRA/DeltaLoRA and AdaMix.

3. ***Better representation is needed***. Table 2 and 3 need some highlights for better comparison. In addition, the paper might be more suitable for short paper instead of a long paper, since one page is not used and 4 pages are used for related works and background.
Comments, suggestions and typos: NA
Overall assessment scores: 2