Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes a new method to improve the inference efficiency of a family of parameter-efficient fine-tuning methods. The authors propose to add input-dependent biases to the transformer's weight matrices (Q, K, V) instead of concatenating prefixes which results in decreasing the dimensions of matrices needed to evaluate the transformer attention. Further, the authors propose two reparameterization tricks to make the calculation of these matrices tractable. Evaluation on GLUE and SuperGLUE benchmark datasets shows that the proposed method performs on par (sometimes better) with previously proposed parameter-efficient fine-tuning methods while being faster at inference. 
strength_and_weaknesses: Strengths:

+ The motivation for this work is well-founded.

Weaknesses / Questions

- Authors decide to exclude recent parameter efficient tuning methods which seem unfair. Their rationale for skipping those methods is that those methods do not work with multi-task inference. Even if that is the case, authors should still compare their method to the recent baselines.
- Without the above evaluation, it is hard to comment on the efficacy of the proposed method.
- According to table 2, the std for AoT P-Tuning is generally considerably higher than baseline models. This suggests that the model might not be stable.
clarity,_quality,_novelty_and_reproducibility: The proposed method is novel but the work ignores some of the recent developments in parameter-efficient fine-tuning methods. The writing of the paper is clear and the ideas are presented well.
summary_of_the_review: In this work, the authors propose a new method for improving the inference time of parameter-efficient methods. The writing of this paper is clear and the method is well presented. Although the idea is novel and works well when compared to P-Tuning, authors miss out on some of the most important baselines. Rationale provided by the authors to skip those baselines is unjust and thus I cannot recommend accepting this paper and thus I vote to reject this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed ahead-of-time (AoT) P-Tuning, a new prompt-tuning method that adds an input-dependent offset to each layer's activation (based on the vocabulary index). The proposed method does not increase the sequence length, leading to now significant computation increase. It achieves competitive transfer learning accuracy vs. cost trade-off. 
strength_and_weaknesses: Strength:
1. Firstly, the paper is generally well-written and easy to follow.
2. The proposed method does not increase the sequence length, which does not lead to significant computation overhead for the tuned model.
3. The proposed method achieves competitive tuning accuracy compared to existing P-tuning work. 

Weakness:
1. Compared to P-tuning, the work is more similar to adapter-based tuning (since the offset is input-dependent; packing multiple tasks in a batch does not sound like a more important factor). However, the authors did not compare the proposed method with existing adapter-based tuning methods, which also does not significantly increase computation.
2. The latency measurement setting (batch size 256, sequence length 128) seems unfair. The sequence length is smaller than usual, and the batch size is larger than usual, which all favors the proposed method, leading to a smaller average overhead. Can the authors also compare the latency with P-Tuning using a batch size 1, sequence length 1024, and discuss if the overhead is still smaller than P-tuning?
3. The comparison also left out some recent state-of-the-art lite tuning methods that also do not increase sequence length like LoRA. I think it is not fair to exclude LoRA and adapter tuning in experiments, since in many cases, people would perform inference with single batch size.  
clarity,_quality,_novelty_and_reproducibility: The clarity, quality, and reproducibility are good. 
The proposed method introduces a new P-tuning alike design that does not increase the sequence length, which has a certain technical novelty. 
summary_of_the_review: Please refer to the strength and weaknesses section for details. I would like to hear from authors' feedback for the final decision.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper studies the problem of parameter-efficient fine-tuning for large transformer-based models. More specifically, authors consider a subset of parameter-efficient fine-tuning methods that support multi-task inference.
Authors extend the popular soft prompt-tuning (or p-tuning) technique, resulting in a technique called "Ahead-of-Time P-tuning". Authors demonstrate that their method can match or outperform prefix tuning (p-tuning v2) on standard GLUE / SuperGLUE benchmarks on popular MLM transformers. The paper also emphasizes that Ahead-of-time p-tuning allows more efficient inference than standard prefix tuning.
strength_and_weaknesses: Preface: I strongly believe that __the current manuscript can mislead the reader__ by unjustly omitting SoTA parameter-efficient methods. I make my case below and suggest several options to correct that. I am also open to discussion if authors disagree with my assessment.

__Strengths:__ 
- the paper is generally well-written and easy to follow, apart from minor typos
- authors work on a real-world problem, optimizing for the types of models that are often used in NLP applications
- the proposed method is evaluated in a fairly standard setup, making it easier to understand the reported statistics
- authors make considerable effort to make their work reproducible.

Regardless of my case against the paper, I deeply respect authors' intent to specify the exact versions and even providing a dockerfile that refers to pytorch/cuda in use. It would also be nice - but not critically required - to include some human-readable instructions on running the code, e.g. readme.

__Weakness:__

As authors note in the paper, P-tuning v1 and v2 have been largely surpassed by methods such as LoRA. However, authors decide to exclude these methods from experiments because "Adapters do not imply performing multi-task inference" (see full quote below‡).

To the best of my understanding, the only way this statement _could_ be true is when simultaneously inferencing multiple tasks in one batch - since non-simultaneous multui-task inference with adapters can be done by simply swapping adapters between batches. Disregarding the fact that this is a fairly niche scenario, it is simply wrong: __both LoRA and simpler adapters (Houlsby et al. 2019) support simultaneous multi-task inference,__ and actively run it in many - though not most - applications.

For Houlsby et al (2019) adapters, if every sequence in a batch uses its own adapter weights, you can run multi-task inference using batched GEMM, (e.g. bmm in PyTorch). This way, each sequence can be processed with its own adapter. For simplicity, I provide a simple code snippet to illustrate this principle later at the end of this section†.
LoRA adapters are parallelized the same way, except that there is no nonlinearlity, and the adapter outputs are applied to Q/K/V matrices separately.

On top of that, industrial frameworks for inferencing with adapters typically optimize this process by (1) grouping together samples for the same task and (2) writing custom kernel that runs BMM with _few_ adapters per batch, instead of one adapters per sample. However, even when this technique is not used, it is trivial to run the common parts of the transformer model in parallel for many tasks, then use a for loop to apply task-specific bits.

Finally, there are other methods for parameter-efficient fine-tuning that belong to neither adapter nor p-tuning category. Notably,

1. IA3 (Liu et al, 2022; [1]) can run multi-task inference out-of-the-box and claims to outperform p-tuning in low-resource settings at no extra computation cost
2. BitFit (Zaken et al, 2021 [2]) can run multi-task inference out-of-the-box and claims to be competitive with p-tuning at no extra inference cost
3. FISH (Sung et al, 2021 [3]) can run multi-task inference with sparse batch GEMM, similar to adapters

In its current state, the paper may mislead a non-experienced reader into thinking that if they want to run multi-task inference, they would be restricted to P-tuning variants and not allowed to use other parameter-efficient fine-tuning methods - and thus may end up with an inferior solution. I believe that this is a major concern that outweighs several positive aspects listed in the "Strengths" section.

I suggest two ways this could be potentially alleviated:

1. comparing against all the relevant baselines for parameter-efficient fine-tuning (adapters, BitFit / IA3, FISH, etc), which may yet reveal that AOT p-tuning outperforms them in some applications;
2. stating, _throughout the paper,_ that this is a specific optimization for prompt tuning - and still showing how it compares to other parameter-efficient prompt-tuning methods, even if unfavorably;

While I give authors the benefit of the doubt, it may be difficult to address this concern in the time-frame for author response. For performance comparisons, if authors are not willing to implement efficient multi-task inference for each method, their performance can be upper-bounded by single-task code with popular inference frameworks (e.g. by extending FasterTransformer[4]).


‡ - full quotes, for the reference

> One may note that the proposed method is more similar to Adapters Tuning (Houlsby et al., 2019) than P-Tuning. Although, Adapters do not imply performing multi-task inference, thus we refer to the proposed method as a variant of P-Tuning, rather than a special case of Adapters.

> Based on this experimental design choice, we exclude experiments with Adapters (Houlsby et al., 2019; He et al., 2022), as well as with LoRA (Hu et al., 2022). While a wide range of efficient fine-tuning methods could be similar to the proposed method (Ding et al., 2022; He et al., 2022), they do not allow to perform multi-task inference, which is the motivation to use AoT P-Tuning.


† - sample code for inferencing adapters for multiple tasks in parallel

```python
import torch   # tested with torch 1.10.1, no cuda required to run this test
batch, seq_length, d_model, d_adapter = 3, 5, 1024, 8

# pre-adapter activations, e.g. outputs of previous layer
inputs = torch.randn(batch, seq_length, d_model)

# adapter weights, one for each sequence in a batch
adapters_weight1 = torch.randn(batch, d_model, d_adapter)
adapters_weight2 = torch.randn(batch, d_adapter, d_model)

# apply adapters in parallel via batched GEMM
adapters_hidden = torch.bmm(inputs, adapters_weight1)
adapters_nonlinearity = torch.relu(adapters_hidden)
adapters_out = torch.bmm(adapters_nonlinearity, adapters_weight2)

# verify that this is equivalent to non-simultaneous computation
for i in range(batch):
    weight1, weight2 = adapters_weight1[i], adapters_weight2[i]
    reference = torch.matmul(inputs[i], weight1).relu().matmul(weight2)
    assert torch.allclose(adapters_out[i], reference)
```


[1] Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning, Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, Colin Raffel , https://arxiv.org/abs/2205.05638

[2] BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models, Elad Ben Zaken, Shauli Ravfogel, Yoav Goldberg

[3] Training Neural Networks with Fixed Sparse Masks, Yi-Lin Sung, Varun Nair, Colin Raffel , https://arxiv.org/abs/2111.09839

[4] https://github.com/NVIDIA/FasterTransformer
clarity,_quality,_novelty_and_reproducibility: To reiterate, the paper is generally well-written and easy to follow, apart from minor typos.
Authors make considerable effort to make their work reproducible, including the supplementary code with pinned dependency versions and a Dockerfile that can be used to release a public docker image. The only thing I found lacking is the precise scripts for reproducing the experiments - or a README explaining how to run them - which can be easily fixed in a revised version of the paper.


Typos:

- End of page 3: "transforms into O((n + p)2)" - extra closing ")"
- Page 4: "lockup of xj-th prompt embedding" -  this may be a typo of "lookup" -- or a deep metaphor that i failed to understand.



summary_of_the_review: This is a well-written, easily reproducible paper that has a fundamental flaw: it unfairly ignores many advanced baseline methods based on - to the best of my knowledge - an incorrect assumption about their multi-task capabilities. Above, I made an argument (and a code snippet) showing that the more advanced baselines can in fact support multi-task inference in several ways. If my argument is correct, the paper is fundamentally flawed and will mislead readers into disregarding important results in parameter-efficient fine-tuning. As such, I will only recommend accepting the paper if either (1) my argument is formally proven incorrect, with minor revisions as requested or (2) my argument is accepted, in which case the paper will need a major revision / repositioning that may be difficult to review in one round.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new parameter-efficient method for fine-tuning a transformer model, called AoT P-tuning. Experiments show that it performs at least as well as P-tuning v2 while being 1.3x faster. As P-tuning, AoT P-tuning enables efficient multi-task inference.
strength_and_weaknesses: This paper introduces a new variant of P-tuning and shows its effectiveness (especially in terms of inference speed). The paper is well written and I don't see any particular weaknesses beside the set of experiments being somewhat limited (as pointed out by the other reviewers).
clarity,_quality,_novelty_and_reproducibility: The paper is overall very clear. The method introduced is relatively simple but novel. A reproducibility section is provided to make it possible to reproduce the results (although I haven't tried to do so).

One suggestion: please make Figures 3 and 4 larger.
summary_of_the_review: This paper is overall quite solid: it proposes a new method, it provides empirical validation, and it is well written.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work introduces a method for parameter-efficient fine-tuning of pre-trained language models. The authors consider two reparametrizations of learnable weights for this method. The work builds on soft prompt-tuning (or p-tuning). Reviewers found the manuscript clear, valued the methodological novelty and the effort to make results reproducible.

However, several reviewers expressed concerns that stronger baselines had not been considered in the experimental validation to the point that two reviewers found the paper misleading. The authors revised the paper accordingly and compared their method to missing baselines. The results showed that the proposed method is close to  state of the art methods. Besides the fact that a small number of concerns remained after the discussion, the large number of changes (new experiments, revised conclusion, etc.) made to the manuscript during the rebuttal/discussion suggest this work is not ready for publication and should be resubmitted with a better positioning of the current work and discussion of the results.
justification_for_why_not_higher_score: Only one reviewer voted for acceptance. He initially gave a score of 8 without motivating his rating. After asking to justify the score, he lowered to 6, but again without providing a good justification. The reviewer did not participate to the discussions.
Decision: Reject