Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a selective token generation method for additive learning under pre-trained language models. Specifically, the authors introduce a selective module to decide the generation policy of tokens, either from the LM generation policy or the RL generation policy. The experiments are conducted on data-to-text task and text summarization task. Results show the improvements over the traditional baselines. 
main_review: Overall speaking, this paper is somehow not qualified enough. Below are the main points:
1. The algorithm is the RL-based method, the difference is that the author introduces a policy selection policy to decide the generation policy. This is straightforward and not new to the NLP generation community. The novelty and technical contribution are truly limited. If this is not the case, the results are also not surprising to see the improvements. Therefore, it is hard to see the shining points.
2. Even the method is not interesting, the experimental comparisons are also not enough. The authors do not give a comprehensive study of the results and the comparison with other papers or baselines. This is again, not qualified. 
3. The selective policy is introduced in a self-learning way, which is automatically learned by the same hidden states of generation tokens. Therefore, a study of the policy selection policy is required. However, it is missed, also for other parts in the model. 
The authors are highly encouraged to make a major modification of the paper. 
summary_of_the_review: The paper does not give enough information to claim the importance of this paper, neither the method nor the results.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes STG, a method of adding a task-specific head to a pretrained language model, and finetuning it with RL along with a policy selector that chooses when to add this policy-specific logits to the pretrained logits for a given downstream task, targetting the low data regime.
main_review: Overall the paper is well written, barring some unnecessary complexity in the description of the technique (details below), and the paper does a good job iterating through details of the method and considering some alternatives. However I find the results unconvincing, primarily in their improvement over the base method. There are also some questions I have with the proposed method and how it is supposed to solve the overfitting issue like the paper claims. As summary, I would say the paper is clear and the proposal is not something I am familiar with in the literature, but the improvement seems marginal.

**Detailed Comments:**

* **<Introduction>** You describe large-scale training data as needing high quality annotations, but much of the larger models are trained on unsupervised text. I think this should be clarified. 

* **<Section 2.2>** Can you explain the footnote?

* **<Section 2.2>** How exactly is the approach motivated by auxiliary training?

* **<Sections 2/3>** Equations 6 and 7 are clear and convey the key aspect of the proposed method well (absent the RL loss), however the majority of the other questions seem more dense and wordy than necessary, and are hard to follow at times (what is G in equation 1?). I think the sections introducing the method could benefit from more clear and concise mathematical formation (i.e. equation 9 can be re-written with less repetition).

* **<Section 3>** The motivation for STG is to prevent overfitting., I understand why wanting to rely on the pretrained logits mostly and only sometimes use the finetuned ones would help ameliorate overfitting, what I don't understand is exactly how this instantiation helps. Why does the RL not learn to always use the finetuned logits downstream, since surely that would help on the finetuning loss?

* **<Section 4>** Prompt engineering is brought up, how well does that compare to the other methods discussed in terms of finetune performance?

* **<Section 4>** Why not try NE(sum)?

* **<Section 5>** Why does the hierarchical policy reduce the action space?

* **<Table 5>** STI should be replaced with STG.
summary_of_the_review: Overall the paper is well written and I am not aware of this exact method being used in the literature, but the improvement strikes me as marginal in results with unclear motivation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper is interested in the problem of exposure bias of sequential text generation: tokens are drawn from the data during training, while during inference, the tokens are sampled from the model’s distribution.
Specifically, the authors consider this problem in the context of few-shot learning, where only a few labeled examples are available.
The authors introduce a method that combines task-specific adapters and reinforcement learning-based training: at each token prediction, the model chooses between the distribution of the original pre-trained LM and the distribution induced by the task-specific adapters on top of the pre-trained LM. The next token is sampled according to the selected distribution.
The authors observed gains in two tasks (data-to-text generation and summarization).

main_review: Strengths:
- The method is novel and brings knowledge of reinforcement learning to the challenges of NLP.
- Paper is overall well written and easy to follow

Weaknesses:
- Gains with the method are limited, especially in summarization. Breath of results is also limited as the authors consider only two tasks
- I would have appreciated seeing human evaluations. It is not obvious from the few samples in the appendix that the method leads to qualitatively superior generations.
- Beyond the numerical gains, it is not clear why the method leads to better generations. For instance, it is not clear how the token predictions at each step differ between the baseline and the introduced method?

Questions:
- The paper seems to be built on the hypothesis that fine-tuning PLMs in low data regimes leads to overfitting. Yet, it is not obvious from the numbers for PLM in line Table 1 and 2. Thoughts?
- Another core hypothesis seems to be the intuition that there are “task-relevant parts in sequence generation”. I couldn’t find any definition of what “task relevant parts” are. How does it differ from learning the format of the task?
- Why use the fine tuned model as an initialization for the model with adapters (non-stg). It seems related but I don’t understand this sentence: “this fine-tuning phase can be skipped when the advanced networks were used in the PLM such as External Encoders or Adapters which can cover the large domain shift”.
- Table 2: are the numbers for non-stg, ne and stg rouge points or % difference with the PLM baseline?
- Could you elaborate on the robustness of the method? For instance, what is the standard deviation between multiple runs?

Additional Feedback:
- Typo - Equation (1) - What is \theta_g? I think you mean \theta_a.

summary_of_the_review: The introduced method is somehow novel, but the experiments have very limited breadth and analysis don't go beyond discussion.
Gains are also limited, and it's not clear looking at the current results that the solutions proposed really answer the problems exposed in the introduction (exposure bias, task-relevant parts, overfitting in low-data regime).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors proposes a method to improve large pre-trained language model's ability in few-shot language generation. The main idea is to freeze the large pre-trained language model (PLM), fine-tune a copy of this PLM on the task at hand, and use a selector to switch between generating the next token from the freezes PLM or the fine-tuned PLM. Instead of the commonly used maximum likelihood estimation, the authors cast the optimization problem as a reinforcement learning (RL) problem using the task evaluation metric such as BLEU or ROUGE as the reward. The authors perform experiments on few-shot data-to-text and text summarization settings, comparing to a few baselines (fine-tuned PLM and a few variants of the proposed method).
main_review: The reformulation of the proposed idea (selector to switch generation between a base PLM and a task-specific PLM) as a reinforcement learning problem is interesting. However, I have some major concerns on the technical clarify/rigor and experiment presentations that hopefully authors could clarify and help me better understand your contributions. 

---

On the technical side:
- during optimization, the authors seem to be computing the reward at each time step (equation (9)) of the generation instead of on the whole-sequence level. However, as far as I know (I may be wrong), when RL methods are applied to NLG, the reward is typically computed on the whole-sequence instead of on the sub-sequence at each time of the generation. If the reward is computed on the partial generated sequence, how is it actually implemented? Why not compute the loss on the entire sequence which I think makes more sense for the BLEU and ROUGE metrics because both are operated on the whole-sequence level?
- What are the instantiations of $Q$, $A$, and $V$? From my understanding, $Q$ and $V$ are actually not computed and $A$ equals the metric score such as BLEU score.
- The selector module performs hard sampling, i.e., outputting either 1 or 0, in equations (5), (6), and (7). However, during _actual_ optimization, the selector performs soft sampling, i.e., outputting a probability between [0,1]. Why is there a mismatch? If the softmax is required as an approximation to the hard sampling, why do the authors need it? From my understanding, if the authors choose to use RL, then the loss function does not necessarily need to be differentiable, so approximating $\pi_s$ as softmax may not be necessary? Which implementation (hard or soft sampling) did the authors use in the experiments? 
- What is the "penultimate representations" in the paper? Is it the last layer hidden states of all previously generated tokens, the last layer hidden state of the last generated token, or all layer hidden states of the last generated token?
---
On the experiment side:
- Overall I think more discussion and analyses are needed. Without these additional results, it is difficult to get insights into why and how the method works. The authors provide some discussions but I think it is insufficient; these claims should ideally be supported by empirical results. More below.
- My main concern is Table 2, which does not seem to make sense to me. If I understand correctly, for the ROUGE metric, the higher number the better, and it cannot be a negative number. In Table 2, PLM, which is a baseline, achieves far better results than any other methods under comparison by a huge margin, including the authors own proposed method. Moreover, one ROUGE result is negative. What happened?
- There is another baseline that I think the authors should compare but did not. How about setting $A$ in equation (9) to 1.0 and just run MLE using equation (9) as the loss? To me, $A$, which results from the authors' RL formulation of the optimization problem, is in practice simply a scaling factor to the usual MLE loss if they did not involve RL. This comparison would suggest how useful is RL training compared to MLE.
- There are different BLEU metrics relying on unigram, bi-gram, etc, and similarly for ROUGE metric. How does the different choices of these variants of the metrics as the reward function impact results? 
- What does the selector do and when does it switch? Some more insights would be really useful. For example, the authors can provide some results on which word in the generation is generated by the frozen PLM $\pi_{\rm LM}$ and which word is generated by the fine-tuned policy $\pi_a$.
summary_of_the_review: Because of my concerns in the rigor and clarify in both the technical approach and the experiments, the paper needs further development and is not yet ready for publication at ICLR. Therefore, I unfortunately cannot champion acceptance for this paper at this time.
review_title: null

Meta-review: This paper presents a reinforcement learning inspired algorithm to train task-specific adapters to adapt pretrained language models for downstream tasks. The paper attempts to tackle an important problem. All reviewers have concerns about whether the results are strong enough to justify claims made in the paper. I appreciate revisions that have been done by the authors during the rebuttal period. However, I believe that the paper is still below the bar for ICLR. I recommend rejecting this paper.
Decision: Reject