LoPT: Low-Rank Prompt Tuning for Parameter Efficient Language Models
Authors: Shouchang Guo, Sonam Damani, Keng-hao Chang


Reviewer paper summaries: This paper presents a novel approach for low-rank learning in prompt tuning, introducing two low-rank approximations, LoPT-1 and LoPT-2. These methods aim to improve parameter efficiency by significantly reducing the number of trainable parameters while maintaining performance levels comparable to full parameter tuning.
Summary of strengths: The introduction of two low-rank approximations LoPT-1 and LoPT-2 for prompt tuning.  Both LoPT methods can achieve substantial reductions in the number of trainable parameters, which is critical for deploying large models in resource-constrained environments.
Summary of weaknesses: 1. The proposed methods, LoPT-1 and LoPT-2, are straightforward and lack significant innovation compared to existing approaches in parameter-efficient learning.
2. The comparisons in "3.2 Comparisons and Results" rely on results cited from other papers rather than being conducted under the same experimental setup. This raises concerns about the fairness and validity of the comparisons.
3. The experimental evaluation is limited to five simple classification datasets, which may not be sufficient to demonstrate the generalizability and robustness of the proposed methods.
4. The claim that LoPT-2 offers no significant advantage over LoPT-1 is not well-supported by the experimental results in Tables 1 and 2.
Comments, suggestions and typos: 1. It is recommended to include comparisons with baseline methods using the same parameter settings in Table 2, providing average values for each method. This will offer a fairer assessment of the proposed methods.
2. Expanding the comparison to include more datasets from the SuperGLUE/CLUE benchmarks would provide a more comprehensive evaluation.
3. The conclusion that the proposed method reduces trainable parameters by a factor of 5 while maintaining performance should be supported with more extensive experimental evidence beyond the two datasets (SST-2 and AGNews) used in Table 1.
Overall assessment scores: 2