Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper explores constructing PET modules in an automatic manner, and S 3PET with NAS is proposed, which conducts the differentiable PET structure search through bi-level optimization and proposes shifted global sigmoid method to explicitly control the number of trainable parameters. Extensive experiments show superior performance with low trainable parameters budgets. 
strengths_and_weaknesses: Pros:

(1)	Automated search for parameter-efficient tuning.

(2)	State-of-the-art performance with low trainable parameters. 

Cons:

Although the authors employ NAS for parameter-efficient tuning, some descriptions are not clear. 

(1)	The authors state that they choose the set of positions where the sum of p_i. However, how to determine the number of positions? If all positions are used, does the performance improve? It shows the oracle.

(2)	In Table1, the authors show the results on GLUE and SuperGLUE. However, which dataset is employed for NAS? Is it searched on GLUE and SuperGLUE separately? If so, I still think it lacks generalization ability. Adding NAS step is also time-consuming, the authors should also report the time cost of NAS compared to the training step. In Table2, it seems that the performance improvement is not significant. It is obvious that NAS can improve performance, so it lacks insight. In Table 3, the authors conduct the transferability experiments, but it is not discussed why such a method can guarantee transferability. Maybe it is because the dataset for searching must be very large. 

questions: As listed in Weaknesses, the following questions need to be solved. 

(1)	Evaluation details and oracle.

(2)	Experiments details and more explanations. 

limitations: There is not serious negative societal impact of this work currently.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper develops a neural architecture search method for the adapter module (or called PET modules) in parameter-efficient tuning schemes.  The authors construct a unified search space for parameter-efficient tuning, and adopted binary differentiable NAS method for architecture search. A shifted global sigmoid is developed to control the sparsity of the PET structure. The searched PET structure achieves competitive results with about 1/5 trainable parameters compared to previous manually designed methods.
strengths_and_weaknesses: Strengths:
1. This paper is generally in good form and easy to follow. Figures and tables are professional.

2. The studied problem, automatic design of PET structure, is novel and interesting.

3. The unified search space may be useful for future research; the proposed Shifted Global Sigmoid for sparsity control is sound and effective.

4. The experimental evaluation is extensive, covering transferability of the searched structure, compare of search space, compare of sparsity method, etc. 

Weakness:
1.  My main concern is the contradiction between the search cost and the application scenario of S3PET. The motivation of PET method is to reduce the model adaptation cost. However, performing the proposed PET search would cost 5 to 8 times training time (as in line 263), which may contradict the motivation of PET methods. I understand the authors claim of the search cost is comparable to (or cheaper than) manual design. However, similar effort is needed for the proposed method, e.g. on search space design or hyper-parameter tuning. Overall, this issue of search cost could severely limit the application scenarios and the significance of S3PET. 

2. Some important literatures are not introduced. (1) The proposed method is highly relevant to differentiable NAS methods with discrete binary gate, e.g. ProxylessNAS [*1], FBNet [*2], etc. In DARTS, the options of gate are activated together in each step, while in these methods, only one option is activated and the structure parameters are optimized by replacing the hard option with soft approximation in backward calculation, same to this paper. (2) Some automatic model adaptation method should be introduced: [*3] [*4].


[*1] Han Cai, et al. ProxylessNAS: Direct Neural Architecture Search On Target Task And Hardware. ICLR, 2019.

[*2] Bichen Wu, et al. FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search. CVPR, 2019.

[*3] Guo, Y., et al. Spottune: transfer learning through adaptive fine-tuning. CVPR, 2019.

[*4] Guo, Y., et al. Adafilter: Adaptive filter fine-tuning for deep transfer learning. AAAI, 2020.

questions: 1. How does PET methods save model adaptation cost? Optimizing an PET module with gradient decent requires all the subsequent network layers to calculate their gradient in the backward pass. All the parameters (after the first PET module) still need to compute and save gradient. Thus, the computational cost and memory cost is actually not reduced.


2. Minor suggestions:

   (1) It would be better to use \eqref in latex for reference of equations (add parentheses for equation numbers).

   (2) Avoid using "Moreover" constantly in consecutive sentences (in Line 66-68). Some words like "Firstly... Secondly..." might be better.
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to design an automatic scheme to search the PET models for parameter-efficient finetuning.  Extensive experiments are given, which demonstrates that the auto-searched method surpasses the manual and random ones with less trainable parameters.

strengths_and_weaknesses: Strength
---The ideal of this work is interesting, while combines the architecture search and PET models tuning.
The given experiments compared with seveal pior arts, which show the superior of the this proposes method.

Weakness
-- Although this work claims that the trainable parameters are reduced obviously, the runtime of the finetuning should be reported to further verify the effectness of this work. 
questions: See weakness
limitations: yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents S3PET an approach to automatically search optimal trainable parameter efficient structure within language models. The proposed approach consists of three main components: A unified search space with probabilistic gating, a differentiable parameter efficient training search algorithm and an explicit sparsity control algorithm. Results on GLUE and SuperGLUE benchmarks show competitive performance with SOTA while using less number of trainable parameters. 
strengths_and_weaknesses: Strengths:
- Unlike predefined trainable substructures in previous works, the proposed approach searches the optimal trainable structure and is guided by downstream performance.
- One important advantage of the approach seems to be the strong results even when the number of training parameters is small (~0.01%).
This is significantly less than prior methods. 

Weaknesses:
- The experiments only focus on GLUE/SuperGLUE classification tasks. There are no results on generation tasks like translation or summarization. 
- It would have been interesting to see some analysis on training data requirements for the proposed approach. E.g., would it still work if there are limited data. 
- The only pretrained model experimented with is T5-large. Performance under larger models or decoder-only models is unexplored.
- It would have been interesting to see some discussion on how expensive the training of the proposed approach is. There is some discussion in the appendix but I would encourage authors to move those to main text. 
questions: The authors do not compare with Prompt Tuning baselines and they mention that it takes much longer to converge [L208].
If there are some initial experiments about this, it would be good to report numbers on how long it would actually take to converge. 
limitations: The authors did not provide any explicit discussions on limitations. For example in terms of efficiency of training, the approach seems to be much slower than pre-defined sparse structures.
review_title: null

Meta-review: metareview: This paper presents work on parameter-efficient tuning of large pre-trained model.  The main contribution is an automated search for the parameter efficient tuning modules, in a neural architecture search style.

The reviewers raised questions regarding the overall efficiency of the training scheme given the cost to the neural architecture search.  However, they believed that the empirical results and overall novelty of the approach together were a solid contribution to research in this area.  The additional clarifications brought into the main text help to make these efficiency concerns clearer and better position the work.  Based the overall novelty of approach and results, this paper is ready for publication in NeurIPS.
confidence: Certain
Decision: Accept