Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper presents a HyperGrid Transformer approach to fine-tuning, where one takes a pre-trained transformer model and then modifies it by introducing hypernetworks that modify the 2nd FFN in each transformer block by generating additional weights conditioned on input. These hyper-networks are trained on all tasks in GLUE/SuperGLUE datasets simultaneously and are task aware through prefixing of a task specific token to input. This allows one to fine-tune only a small number of parameters and end up with a model that performs quite well on all tasks at the same time, not much worse than fine-tuning the entire transformer model on all of these tasks.

This is an interesting paper in the area of hypernetworks with results suggesting potential for impact where one can achieve good accuracies on tasks without having to fully fine-tune gigantic pre-trained models.

I do have some questions for the authors though:

-  local vs global - global seems to be just like learning another weight matrix not conditioned on anything? from the name one would expect local to be conditioned on some specific parts of the input while global is conditioned on entire input. what is the intuition on why they help?

-  what's the intuition between the differences of performance of the various setups (LG, GL, L^2, etc)

-  figure 9 doesn't exist. if coarser variants are better, what happens when the entire weight matrix is treated as 1 block (so you just learn a scalar weight)? what about learning a single scalar for each weight in the FFN (i.e. block size is 1)?

-  have you tried adding dynamic weights to projections in the multi head self attention modules (e.g. to projections for q,k,v)

-  table 1 - why are QNLI results so much worse for HGT(LG) than all other results, but seems to be doing better on most other tasks; how stable are all of your results (i.e. what is the variance across seeds?) 

- parameter counts are confusing - how does one compute that a "multiple networks" needs 3.2b params, but a "single network" needs 0.2b params? are these the trainable weights for each setup, so you count 16 tasks x 0.2b weights? maybe it is better to report total num of params + trained params or else somehow make it more clear what the number of parameters means

- have you tried finetuning the hypernetworks on individual tasks?

---- update:
Thanks for the update. I guess the "intuition" is driven mostly by empirical results, which I suppose is ok but may be worth digging into a bit more. I have updated my rating.

review_title: Interesting paper about applying hypernetworks to Transformers

final_score: rating: 6: Marginally above acceptance threshold
review_content: This manuscript presents a HyperGrid Transformer, which is engaged in learning a single model to account for multi-tasks in NLP. The core idea of HyperGrid Transformer is to learn task-conditional dynamic weights in a grid-wise manner in the feed-forward layers, where the weights are factorized in local and global components. This idea is simple, materializing the goal of reducing the parameter cost for the used multi-task network. However, the conducted experiments look nice, showing promising performance on GLUE/SuperGLUE. Therefore, from my point of view, this work is worthy of a publication at ICLR. 
review_title: An empirically good method for a single multi-task Transformer 

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose HyperGrid Transformers with a decomposable hypernet-work that learns grid-wise projections to specialize regions in weight matrices for different tasks. Usually, people would use different models to solve different tasks respectively. In this paper, the authors focus on using a single model to solve all tasks and it will save a lot of model parameters for natural language understanding. And the authors have done comprehensive experiments on GLUE and SuperGLUE, and prove that the proposed single model can achieve much better performance than baseline and competitive performance with multiple task-specific models.

Pros:
1. The idea to make use of decomposable grid wise projection is interesting. This is to somehow add regularization to the weights.
2. The proposed method has been widely evaluated on GLUE/SuperGLUE tasks, and achieve good performance.

Cons:
1. The baseline details are not clear. When using a single model as baseline, how many layers are shared across tasks? What's the sample strategy for different tasks? Is it possible to train a single model on multi-tasks for some steps, then fix most layers and only finetune some task specific layers? I feel the baseline is a bit weak, although I cannot come up with a stronger one that can be easily adapted to the pertained model.
2. It seems "Task Conditioning" is a very important trick. The authors should have some ablation study on it. Or maybe add it to the baseline.

Overall, it's great to see people working a single model to solve all tasks. And I would be happy to increase my score if the authors could convince me regarding the baseline which is quite tricky.

####update####
The experiment results are not surprised, but strong enough. Still no very strong baseline provided in this submission, but it might be good to set up a benchmark in this direction. However, T5 model needs more computational resource and the experiment results are hard to replicate. Overall, I would like to keep rating.

review_title: Review #4

Meta-review: The paper proposes "HyperGrid Transformers" a modified transformer architecture for learning a single model for multiple tasks in NLP.  The proposed method was evaluated on popular GLUE/SuperGLUE tasks and reported competitive results with the baselines (the improvements are somewhat marginal). The paper contains some interesting idea of using a decomposable hypernetwork to learn grid-wise projections for different tasks, which may not be particularly novel in machine learning context but seems new for multitask NLP. Reviewers generally agree the paper is above acceptance bar, however some concerns were raised about clarity of baselines and fairness of experimental comparison as well as stronger baselines. Authors improved some of them in the rebuttal, but there is still some room to further improve the quality of presentation and writing in the final version. 
Decision: Accept (Poster)