HyperGrid Transformers: Towards A Single Model for Multiple Tasks
Authors: Yi Tay
Zhe Zhao
Dara Bahri
Donald Metzler
Da-Cheng Juan
Venue: ICLR 2021 Poster (2021)
Keywords: Transformers
Multi-Task Learning

Abstract: Achieving state-of-the-art performance on natural language understanding tasks typically relies on fine-tuning a fresh model for every task. Consequently, this approach leads to a higher overall parameter cost, along with higher technical maintenance for serving multiple models. Learning a single multi-task model that is able to do well for all the tasks has been a challenging and yet attractive proposition. In this paper, we propose HyperGrid Transformers, a new Transformer architecture that leverages task-conditioned hyper networks for controlling its feed-forward layers. Specifically, we propose a decomposable hypernetwork that learns grid-wise projections that help to specialize regions in weight matrices for different tasks. In order to construct the proposed hypernetwork, our method learns the interactions and composition between a global (task-agnostic) state and a local task-specific state. We conduct an extensive set of experiments on GLUE/SuperGLUE. On the SuperGLUE test set, we match the performance of the state-of-the-art while being $16$ times more parameter efficient. Our method helps bridge the gap between fine-tuning and multi-task learning approaches.

Published as a conference paper at ICLR 2021
HYPERGRID TRANSFORMERS: TOWARDS A SINGLE MODEL FOR MULTIPLE TASKS
Yi Tay, Zhe Zhao, Dara Bahri, Donald Metzler, Da-Cheng Juan Google Research Mountain View, California {yitay, zhezhao, dbahri, met z. zer,, dacchee (goge
ABSTRACT
Achieving state-of-the-art performance on natural language understanding tasks typically relies on fine-tuning a fresh model for every task. Consequently, this approach leads to higher overall parameter cost, along with higher technical maintenance for serving multiple models. Learning single multi-task model that is able to well for all the tasks has been challenging and yet attractive proposition. In this paper, we propose HyperGriid Transformers, a new Trans- former architecture that leverages task-conditioned hyper networks for control- ling its feed-forward layers. Specifically, work that learns grid-wise projections that help to specialize regions weight matrices for different tasks. In order to construct the proposed hypemnttwrk, method learns the interactions and composition between a global (task-agnostic) state and local task-specific state. We conduct extensive set of experiments on GLUUE//uperGL..E On the SuperGLUUF test set, we match the performance of the tatt--of-the-ar while being 16 times more parameter efficient. Our method helps bridge the gap between fine-tuning and multi-task learning approaches.
propose decomposable hypernet-
INTRODUCTION
Learning single multi-task model that performs well across multiple targeted tasks is an attractive proposition for many reasons (Kaiser et al. 2017 Ruder 2017, Clark et al.) 2019b). Although ex- tremely challenging. this paradigm enables substantial savings in overall parameter costs, along with eliminating the need for maintaining multiple models in production (Stickland and Murray) 2019). However, achieving state-of ttate-of-the-art performance on natural language understanding bench- marks today (Wang et al. 2018 2019) still relies fine-tuning new model for every single task. This methodology is infeasible in many situations. Moreover, certain tasks rely on an extensive en-
semble of models and/or task-specific fine-tuning tricks (Liu al. 2019b Devlin al. 2018 Clark
et al. 2020).
The single-task fine-tuning paradigm is well-established to be the dominant approach (Raffel al. 2019), as training multiple tasks using single set of parameters can be problematic in many ways, such as catastrophic forgetting (French and Chater 2002 McCloskey and Cohen 1989 McClelland ct al. 1995; Kirkpatrick et al., (017)) or the inherent difficulty of finding a consistently good model for tasks (Clark et al. 2019b; Wu al.) 20200. Inevitable task conflicts and difficulty fitting all models within set of hard parameters is also a challenging problem for multi-task co-training.
In this paper, we propose a new Transformer architecture, the HyperGrid Transformer for efficient modeling of multiple tasks within single set of model parameters. HyperGrid Transformers rely on a hypernetwork- based (Ha al., (2016) module that performs gridwise decomposable hyper pro- jections. This module is task conditioned and dynamically learns to generate weights of the feed- forward layers of the Transformer model.
Overall, eventual goal is to dispense with task specific fine-tuning tricks altogether. While neural networks typically maintain the same consistent set of parameters for all input instances, the proposed HyperGrid Transformers introduces instance-specifie parameters by conditioning the current input. This setup enables our model to learn task-specific reparameterizztion for each input instance, which mitigates several challenges of multi-task co-training.

---

Published as a conference paper at ICLR 2021
Our proposed HyperGrid module belongs to family of hypernetworks (Ha al. 2016). in which side network is responsible for weight generation for the main network. In our case, cank-conditioned hypernetworks provide greater flexibility and expressiveness for capturing the dynamics of multiple tasks within a single set of parameters. Specifically, we introduce two novel algorithmic improve- ments over the existing methods.
First, we introduce the notion of grid-wise projections in which we assume a structural layout in vanilla projection layers. For each input sample, our projections dynamically control the parameters in grid-wise, region-specific manner. The structural segmentation of feed-forward layers is similar in spirit to mixture-of-ex expextssts gating (Shazeer et al.) (017), albeit at a lower-level. Conversely, standard hypernetwrrss only consider row-wise weiggtit of weight matrices.
Second, we introduce decomposable hyper-projeccions. The key idea to learn rich compositional and pairwise interactions between dual hypernetworks. A dual setup is adopted, where we explore different hypernetwork composition variants. We introduce a novel local-global setup, which com- poses local instance-specific and task-specific hyper-projection with task agnostic global state embedding. This is intuitive since this setup is not only highly expressive and flexible but also serves as factorization of local and global components. To the best of our knowledge, our work is the first to explore this setup with respect to learning conditional parameters.
Finally, we conduct extensive experiments GLUE/SuperGLUE, Our proposed model is able to match the performance individually fine-tuned state-of-the-art Text-to-Tex Transformers (TS) (Raffel al. 2019) models with single model that is learned fit all GLUE and SuperGLUE tasks at once. Moreover, our single model also outperforms strong baselines that employ ensembling and other task-specific tricks (Liu al. 2019b Clark al. 2020
Our Contributions
The contributions of this paper can be summarized as follows:
We propose HyperGrid Transformers, a form of hypernetwork- based Transformer that learns task-conditioned dynamic weights for its feed-forward layers.
The key novelty behind HyperGrid Transformers is the factorization of local and global components for weight generation. Our weight generation grid-wise and imbues the model with a structural layout.
We conduct extensive experiments on natural language understanding benchmarks (GLUEE/uperGLLUE) With single model, match the sate-of-thheart T5 model that is finetuned in a per-task fashion (multiple models), resulting in 16x parameter savings.
2 HYPERGRID TRANSFORMERS
This section outlines the key idea the proposed algorithm.
2.1
HYPERGRID MODULE
HyperGrid operates on weight matrices (linear transformations), i.e., Y WX 6. In hypernet- work formulation, instead of letting W be free weights, we generate W using parameterized side network (.).
Y = Wa + b where =
= H(X)
(1)
where W Rdmxdd In the where X is single vector E Rdm,, may parameterize H()) with a simple feed-forward layer.
=(X)=
O W
(2)
where 1 is column vector of ones, o is the sigmoid activation function and U E Bdd The key idea that the hypemetwork generates vector, i.c., UX RR that is broadcast (multiplied by 1) and multiplied by W, acting as a row-wise scaling of W. We are also able to reduce U E Rd where dy mod 0 and repeat the vector di times form the original dimension of dy. These methods only consider scaling one dimension of W (e.g.. row-wise). We now consider methods beyond simple row-wise weight scaling.
2

---

Published as a conference paper at ICLR 2021
2.1.1 DECOMPOSABLE GRIDWISE PROJECTIONS
In our method, we propose grid-wise projections that segments W into a grid, i... blocks of dm x d,
Transformer Weights
d. We generate blocks by the outer product of Ly E Rd and Le Rdc. Note that d, and de are user-specified hyperparameters that control the grid-size for the fan-in and fan-out of the out- put matrix. For simplicity, we consider divisible blocks where d, < dm, dm mod d, 0, and de < dd,dj mod de O. In this case:
Repae
H(X)
(3)
Input
where (L) Rdd.xde vb.) is a re-
Local
peat vector function that repeats its input de times Figure I: Detailed Illustration of the proposed De- on row axis and de
composable Gridwise Projections. Two decom- d times on the column axis. posable vectors compose form gating ma- We name this approach the L variant, short for trix which is expanded to construct task-adaptive weight matrices.
Local-Local Gridwise Projection.
Composition between Local and Global Factors The decomposable grid-wise projections learn Lr and Le from X, which makes conditioned on local, instance-wise information. Here, we postulate that it may be beneficial for either I, or Le be global embedding. By keeping I, global, trainable embedding, this can be formulated as:
H(X) = ((o
(4)
where Ge E Rd4, In this case, Ly is conditioned from X, the specific input sample. On the other hand, G. remains consistent across all input samples. Hence, the outer product essentially rich dyadic composition between local and global factors.
Local-Global and Global-Local is easy to see that there are two ways of composing and G. The above method considers the Local-Global approach where the fan-in uses a local hypernetwork and the global part uses a trainable embedding. An alternative that flips this around to use a Global- Local composition is evaluated in our experiments. Namely, this can be expressed as:
H(X) ((a((C,((E))))) O W
(5)
2.2 DYNAMIC WEIGHT GENERATION WITH HYPERGRID
This section describes how we use task-conditioned hypernetworks to influence and generate the parameters for HyperGrid Transformers.
TransConner Block
Task Conditioning The local network part of Hyper- Grid T, learned via task embedding T Rdm.. which provides a task identifier and information to the hypernetwork. HyperGrid Transformers, we first ap- ply self-attention of the task embedding by concatenat- ing with the input sequence. This is described as:
Decomposible Hyper Projectr
Traafaza
T' MHSA((T; XJ)
(6)
Tait
where [:] is a concatenation the length dimension and MHSA(.) is the multi-head self-attention function. The input sequence X interacts with the task embed- ding to generate 7" which is used in our hypernetwork module.
HlyperGri
Trua tréeliie
Figure 2: Illustration of the proposed Hyper- Grid architecture.

---

Published as a conference paper at ICLR 2021
Weight Gating The HyperGrid module is added at the position-wise forw-forward layers of the Transformer models. More specifically, we equip the second positional FFN after the ReLU activa- tions with HyperGrid. There several reasons for doing the fan out of this layer is typically scaled up to very large values (Raffel et al. 2019). Hence, the influence this layer has the greatest potential benefit the Transformer model. Second, early periments on both of the positional feed-forward layers yielded no substantial improvements. Hence, we opt to only modify the second positional FFN of the Transformer model.
In most Transformer implementations,
Initialization In our experiments, we take advantage of existing pretrained model (Raffel al, 2019) and add additional HyperGrid parameters that are fine-tuned along with the rest of the net- work. The overall formulation of the HyperGrid-enhancee Transformer can be written as:
Y, = - Wi) Wx(X,-1)
where denotes the layer i. We construct a new HyperGrid (with non-shared parameters) for each layer. Since W has been pretrained, we also add residual connection of the original Wi(X:-i) computation to the mixture.
Parameter Costs We note that the parameter counts added by HyperGrid are relatively negligible since d and de are small. In the LG setting, the model adds d, de parameters at each layer. On the GL setting, the parameter cost added is d, + dyde- The most expensive option is L where the added cost is d, + dyde. These added parameter costs are often negligible for large Transformer models.
EXPERIMENTAL RESULTS
We conduct experiments GLUE (Wang et al., 2018) and SuperGLUF. (Wang et al. (019) which are consolidated benchmarks of multiple challenging NLP and NLU tasks. While most of the work in this area has been focused achieving good task-specific performance, trying to get good performance with a single model on all GLUE and SuperGLUE tasks. Therefore, most of our experiments conducted on mixture of all GLUE and SuperGLUE tasks.
work focuses
3.1 EXPERIMENTAL RESULTS
In this section, we discuss the empirical results of our experiments. Further details about the ex- perimental setup can found in the appendix. Pertaining to parameter counts (reported as # in our experiments), this is the total parameter cost for serving the entire suite of tasks. If a single model (finetuned on single task) is parameters, then the costs of serving N tasks a multi-model setup would be NX.
3.1.1 SINGLE MODEL VERSUS MULTIPLE MODELS
These experiments investigate different model settings, namely using (1) multiple models (MM) or single model (SM) for all tasks. The MM model trains a fresh model on every single task and retains the best checkpoint for all tasks. Meanwhile, the SM model is trained on all tasks at once. We compare them against HGT, our single model HyperGrid Transformer approach.
Details about Baselines For the Single Model Baseline, this is the identical model without our HyperGrid layers which serves fair comparison to our model. This enables to observe the effect HyperGrid directly. As for the multi-model method, this done by finetuning directly on each task and reporting results from the best checkpoint (of each task). This is a very strong baseline because the model is also allowed to choose specific best checkpoints for each task which the single model approach is prohibited against. As for the sampling strategy in the single- model approach (baseline and our approaches), we use a proportionate mix tasks according to the number of samples in the each task/dataset.
4

---

Published as a conference paper at ICLR 2021
Model
[0]
Avg
CoL.A
SST
MR
STS
QQP
MNLI
QNLT
RTE
T5 (SOTA)
3,2B
83.4
53.8
92.7
88.9
88.0
91.6
84.4
90.5
76.3
Multi Model
3.2B
85.7
59.6
94.2
90.1
89.1
90.6
86.5
93.7
82.0
Single Model
0.2B
85.0
57.3
89.5
90.2
86.2
93.1
80.9
Our Single-Model Approaches
HGT (L*)
0.2B
85.2
59.4
90.6
90.1
88.9
86.5
93.1
79.1
HGT (LG)
0.2B
57.9
94.6
89.2
90.1
90.3
86.7
81.2
84.2
HGT (L)
0.2B
85.6
59.9
94.0
89.1
89.9
90.2
86.5
93.1
81.1
Table Experimental results on GLUE dev set.
Model
[0]
Avg
BQ
CB
CP
MultiRC
Record
RTE
WiC
WSC
T5! (SOTA)
3.2B
71.4
76.6
91.2/92.0
66.2
66. 1/25.8
69. 1/68.2
75.3
68.0
78.6
Multi Model
3.2B
74,8
82.9
96.4/92.0
63.0
19.1/44.0
77.6/76.8
83.8
71.6
73.1
Single Model
0.2B
73.6
81.5
77.3/83.9
64.0
78.2/43.3
76.9/76.1
84.1
66.9
74.0
Our Single-Model
Approaches
HGT (L-))
0.2IB
75.3
82.4
85.3/91.1
64.0
77.8/42.7
76.8/75.9
83.4
67.1
80.8
HGT (LC)
0.2B
74.8
82.5
83. 1/89.3
64.0
77.9/42.8
77.1/76.3
84.1
65.5
78.8
HGT (L)
0.2B
74.5
82.5
81.5/89.3
78.8/41.0
76.8/76.0
85.9
66.5
78.8
Table 2: Experimental results on SuperGLUE dev set.
Results 2 reports results of our experiments on the GLUE and SuperGLUE benchmark. The first key observation is that the single model (SM) approach is outperformed by multi model (MM). This is a well known phenomena and therefore multi-model generally adopted when the absolute best score is desired every single task. The interesting result is that close to the performance of single model with our approach. As a result, the multi-model has 16x more parameters. To fit both GLUE and SuperGLUE, this would require 16x the parameters. Given that our goal is to bridge the performance of single model versus multiple models for multiple tasks, we find that this result considerably successful. Moreover, we observe that our single- model approach outperforms the mudti-model baseline by +0.6% on average across 8 tasks. We observe similar trends as on the GLUE benchmark. Naturally, the best model is the multi-model model which involves finetuning specialized model for each task. The gap between multi-model and single-model 74.8 versus 73.6. Our approach bridges this gap, improving the single-model score 74.5, competitive with the multi-mode approach.
able
rather
3.1.2 PERFORMANCE GAINS ACROSS MODEL SIZES
We investigate the gains of the proposed HyperGrid over the base model various sizes of the T5 model. For models larger than Base, we train with 64 TPU V3 chips for 200K steps and select best checkpoint for all tasks based on the benchmark score.
Findings Table 3 reports results of GLUE and SuperGLUE scores (and their macro-average). We find that performance gains on SuperGLUE averages reasonably good (+1.9% on Large). The model still outper- forms the vanilla model on GLUE with marginal performance gains. Overall, on a macro-average tasks, find an overall +1.0% im- provement across three sizes. These
Model
GLUE
SuperGLUE
Base
84.99
73.55
79.27
Ours
85.22 (+0.27%)
75.30 (+2.7%)
80.26 (+1.3%)
Large
88.22
80.04
84.13
Ours
88.31
(+0.1%)
81,56 (+1.9%)
84.94 (+1.0%)
3B
89.53
84.22
86.87
Ours
89.67 (+0.2%)
85.75 (+1.8%)
87.71 (+1.0%)
18
Table 3: Effect of HyperGrid Transformers across all model sizes. HyperGrid improves single-model co-training consis- tently overly different model sizes. Improvement over Su- results show that performance gains perGLUE is greater than GLUE.
scale with model size.
3.1.3 EFFECT OF MODELING CHOICES
To ascertain the effectiveness our approach, we test different architectural variants of HyperGrid Transformers, along with other architectural variants considered during model development.
5

---

Published as a conference paper at ICLR 2021
Setup We evaluate all four model variants of HyperGrid Transformers (L, LI, GL and LG). For the other architectural variants, we were mainly interested to know a hypernetwork setup (weight gating) is better than gating on the output representations (details can be found in the supplementary material). For the base setting, we ran the baseline T3 model (single-model) four times and reported the mean and standard deviation of the runs. When comparing the performance gain of our method, we compare against the max run of the baseline runs. We report relative performance gains/loss against this max baseline score. We conduct ablation studies on the four composition types on the large models
Variant
GLUE
SuperGLUE
AVG
Findings of HyperGrid Variants
Base Models
Table reports our key ablation results. Pertaining to results of the base models, overall find- ing is that HyperGrid generally im- proves performance over the max baseline. Gains are mainly on Su- perGLUE while maintaining good performance on GLUE. The over- all average gain about +1%. Amongst the different variants of HyperGrid, the best performing model on this setup is the L setup. On the large setting, we find that the LG model performs the best while the L and L variants per-
Baseline
85.03 (+ 0.087)
73.77 (+0.110)
79.40 (H-0.099))
Base (Max)
85.11
73.83
79.40
85.60 (+0.6%)
74.50 (+0.9%)
80.05 (+0.8%)
I
85.22 (+0. 1%)
75.30 (42.0%)
80.26 (+1.1%)
GL
85.12 (+0.0%)
75.00 (+1.6%)
80.05 (+0.8%)
LG
85.43 (+0.4%)
74.78 (+1.3%)
80.10
OG
85.13 (+0.0%)
73.31 (-0.7%)
79.22
OG (16)
84,94 (-0.2%)
73.10 (-1.0%)
79.01 (-0,5%)
OG (32)
84.84 (-0.3%)
72.93 (-1.2%)
78.89 (-0.6%)
OG (64)
85.07 (-0.0%)
74.11 (40.45)
79.59 (+0.2%)
Large Models
Baseline
88.22
80.04
84.13
88.07 (-0.2%)
80.51 (+0.6%)
84.29 (+0.2%)
LA
88.05 (-0.2%)
80.68 ++.88%
84,36 (+0.3%
GL
88.33 (+0.1 1%)
80.30 (+0.3%)
84.32 (+0.2%)
LG
88.31 (+0.1%)
81.56 (+1.9%)
84.94 (+1.0%)
form similar to the baseline.
Table 4: Ablation Study. OG stands for Output Gating.
Is Output Gating Better? The other architectural variants (OutGate) not perform well and generally perform with net loss in performance as compared to the baseline. As such, ascertain that gating on weights is more effective than gating on the output representations. This verifies that Sypermetwork-based approach is indeed effective opposed to simple task-conditioned output gating.
3.1.4 EFFECT OF GRID SIZE ON PERFORMANCE
We investigate the effect of Grid size (fan-in and fan-out) of our proposed HyperGrid method. The purpose of this experiment is to discover how fine-grained or coarse-grained the hypernetwork should be. Notably, smaller values of dr, de signify former weights.
coarse caaseegrained control of the Trans-
Setup We searched d, (fan-in) and de (fan-out) in the ranges of {4, 8, 16, 32, 128, 256} and (8, 16, 32, 128, 256) respectively and report the results on GLUE + SuperGI.UE (macro-average) by varying a single value. When varying d,, we took the average all d, runs and plot the max, mean and min. Likewise, when varying de, took the average of all d, runs and plot max, mean and average. We report scores across the L LG. and GL variants of HyperGrid.
Findings pertaining Grid Size Figures Figures illustrates performance across varied grid sizes. We observe that a clear trend exists. For most settings, a small fan-out (de) works well (e.g., 32) as noted by many spikes around this region. For fan-in (d,)) a smaller value also works well. However, performance gets better at higher fan-out d, values again (e.g., > 128). Trends are quite consistent across all three variations that we considered. These results suggest that a more coarse grid may be more effective, as the regions within the grid become larger.
Due to the relative increased cost of searching large models, we performed sparingly low number of ablations on large models.

---

Published as a conference paper at ICLR 2021
Figure 3: fan-in on L setting.
Figure 4: fan-in on LG setting. Figure 5: fan-in GL setting.
Figure 6: fan-out on 2 setting. Figure 7: fan-out on LG setting. Figure 8: fan-out on GL setting.
3.1.5 PERFORMANCE ON TEST SET
For our final runs, we submit our model predictions to the GLUE and SuperGLUE test servers.
Model
Avg
CoLA
SST
MR
STS
QQP
MNLI
QNLI
RTE
WNLI
Bert"
80.5
60.5
94.9
84.5
86.5
89.3
92.7
65.1
RoBERTa"
88.1
67.8
96.7
89.8
91.9
90.2
90.8
95.4
88.2
89.0
Albert"
69.1
97.1
91.2
92.0
90.5
91.3
89.2
89.0
XLNet"
70.2
97.1
90.5
92.6
90.4
90.9
88.5
89.1
Electra'
89.4
71.7
97.1
90.7
92.5
90.8
95.8
92.5
T5 (3B)
48B
88.5
67.1
97.4
90.0
89.8
82.1
91.3
96.3
91.1
89.7
T5 (1IB)
176B
89.7
70.8
97.1
90.0
92.1
82.5
90,9
96.7
92.5
93.2
HGT (3B)
3B
88.2
65.6
97.5
89.0
91.6
81.9
90.9
95.9
90.1
89.7
HGT (11B)
89.4
69.0
97.6
89.2
92.6
82.0
91.3
96.4
91.5
93.2
Models with large ensembles. All models are single-tasker taakel fine-tuned except ours. Parameter costs are reported considering ensembles and cost required fit all of GLUE and SuperGLUE.
Table 5: Test set performance GLUE (Wang al.
2018
Model
10]
Avg
BQ
CB
CP
MultiRC
Record
RTE
WiC
WSC
BERT+
2.7B
71.5
79.0
84,8/90.4
73.8
70.0/24.1
72.0/71.3
79.0
69.6
64.4
RoBERTa
56B
84.6
87.1
90.5/95.2
90.6
84,5/52.5
90.6/90.0
88.2
69.9
89.0
(3B)
48B
86.4
89.9
90.3/94.4
92.0
86.8/58.3
91.2/90.4
90.7
72.1
90.4
T5 (11B)
176B
88,9
91.0
93.0/96.4
94.8
88.2/62.3
93.3//2.5
92.5
76.1
93.8
HGT 3B
3B
84.7
89.2
81.7/90.4
89.6
86.6/58.7
91.1/99.3
90.8
70.6
87.7
HGT 11B
11B
88.9
91.1
93.9/96.8
94.6
88. 1/62.4
93.5/92.7
92.3
74.6
91.6
Table 6: Test set performance SuperGLUE WWng et al. 2019
Our HyperGrid Transformers
achieves competitive performance to the sate-oofthe-aae with a single model. Parameter costs refers to total number of parameters used to fit all GLUE and SuperGLUE. tasks

---

Published as a conference paper at ICLR 2021
Setup We run experiments with 3B and 1IB HyperGrid Transformer model in multi-task setup (GLUE + SuperGLUE), We initialize with the T5 pre-trained checkpoints. Since this is a relatively expensive run, we only train the single model HyperGrid once using 32 x 128 grid with the LG (local-global) setting. For GLUE, we compare against baselines reported in (Clark et al. 2020) which includes models such BERT (Devlin al. 2018), ALBERT Lan et al. (20T9). RoBERTa (Liu et al. 2019b), and XLNet (Yang et al. 2019). Note that all these models are ensembles and heavily rely on task-specific fine-tunining strategies. More details can be found in the supplementary material.
Results on Test Set We find that our single model approach can achieve highly competitive results on both GLUE and SuperGLUE. Our model achieves strong performance of $8.9 on SuperGLUE, matching the reported T5 results while having 16 times fewer total parameters. On GLUE, the performance gap also small, almost matching the T5 model at 89.4 versus 89.7. The gap on the base model remains similar at 88.2 versus 88.5. On SuperGLUE, our 3B model achieves 84.7, respectable score that matches the performance of RoBERTa ensembles fine-tuned individually with task specific tricks (Liu et al. 2019b).
RELATED WORK
Multi-task learning (MTL) (Caruana 1997 is long standing research problem. Learning single unified model that does well on multiple tasks is an uphill battle given well-known problems such as catastrophic forgetting (Kirkpatrick al. 2017). As such, learning large number of tasks with single set of model parameters is an extremely challenging endeavour. Moreover, the disproportion- ate amount of data per task is also potentially problematic (Lee et al. 2017 Pfeiffer et al. 2020), which results in models overfitting on high resource tasks but underfitting on low resource tasks.
Early work multi-task NLP typically considered hierarchical taxonomy of tasks (Hashimoto et al. (2016) where a clear hierarchy of tasks exist, such as POS - Chunking - entailment. The Joint Many-Tast (JMT) model explores an incremental and hierarchical paradigm for building multi- task NLP models. Similarly, (Sanh et al., 2019) proposed hierarchical multi-task model based on the intuition of low-level and high-level tasks. Another line of recent work explores casting all tasks into form of question answering problem (McCann et al. 2018) and using interpolated pointer-generator (See et al. 2017) mechanism for generating 'answers'.
means for improved model quality has been frequently explored. In relatively recent work, (Liu et al. 2019a) proposed MTDNN, a multi-task deep neural network that shares parameters between several NLP tasks. The model achieves strong performance on the GLUE benchmark. However, MTDNN simply leverages MTL as a form of pretraining and uses task-specific models for final evaluation. The recent T5 (Text-to-Text Transfer Transformers) model (Raffel et al., (211)) frames all NLP problems as a Seq2Seq (Sutskever et al. 2014) problem. However, the best results are again obtained by task-specific fine-tuning.
Exploiting task relatedness
Orthogonal to other research efforts, (Clark et al.) 2019b) proposed Born Again Neural Networks (BAM), a clever way to obtain a single multi-task network by knowledge distillation. (Stickland and Murray 2019) proposed Projected Attention Layers for task-specific fine-tuning of BERT (Devlin al.
2018). (Zaremoodi et al. 2018) proposed Adaptive Knowledge Sharing for low-resource neural machine translation. Our work is related to the literature surrounding hypernetworks (Ha ct al.
2016) which have been found to useful areas such as continual learning (von Oswald et al, 2019 Learning task-adaptive parameters avoid catastrophic forgetting has also been go-to strategy for continual learning (Yoon et al. 2019). Outside the NLP domain, flexible parameter sharing approaches are also dominant strategies for learning multi-task models (Ma et al. 2018 2019).
"Since we did not co-train with the WNLI dataset due to issues stated in (Raffel et al. 2019), we simply report T5 results on WNLL To be fair, we ignore WNLI parameter counts for all hascline models. 'The authors of (Raffel et al. 2019) explored
exploreed this approach hut did not find it to he satisfactory.
8

---

Published as a conference paper at ICLR 2021
5 CONCLUSION
We proposed Hypergrid Transformers, new Transformer architecture that leverages Grid-wise De- composable Hyper Projections (HyperGrid), hypernetwork-basee projection layer for task condi- tioned weight generation. We learn and fit all GLUE and SuperGLUE tasks within the same set of model parameters and achieve competitive results the same state-of-the-art model that is spe- cially and individually fine-tuned on each and every task. On GLUE//uperGLLU, this efficient single-model model method results in 16x fewer parameters.
REFERENCES
Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. In Proceedings of the second PASCAL challenges workshop on recognising textual entailment, volume 6, pages 6-4. Venice, 2006.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo, The fifth pascal recognizing textual entailment challenge. In TAC, 2009.
Rich Caruana. Multitask learning, Machine learning, 28(1):41-75 1997.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio and Lucia Specia. Semeval-2017 task
I: Semantic textual similartyymultilnngua and coos-llnguul focused evaluation. arXiv preprint arXiv: 1700.00005055 2017.
Christopher Clark, Kenton Lee, Ming-
Ming-Wei Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019a.
Kevin Clark, Minh-Thang Thang Luong, Urvashi Khandelwal, Christopher D Manning, and Quoc V Le. Bam! born-again multi-task networks for natural language understanding. arXiv preprint arXiv:/907.044 2019b.
Kevin Clark, Minh-Thang Luong, Quoe V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003. 10555, 2020.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pages 177-190. Springer, 2005.
Marie-Cathering De Marneff, Mandy Simons, and Judith Tonhauser. The commitmentbank: In- vestigating projection in naturally occurring discourse. proceedings of Sinn und Bedeutung 23, 2019.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: / /810.04805, 2018.
William B Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005.
Robert M French and Nick Chater. Using noise to compute error surfaces in connectionist networks: A novel means of reducing catastrophic forgetting. Neural computation, 1757)1775 2002.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third pascal recognizing textual entailment challenge. In Proceedings of the ACI-PASCAL workshop on textual entailment and paraphrasing, pages 1-9. Association for Computational Linguistics, 2007.
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. arXiv preprint
2016.
Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. A joint many-task model: Growing neural network for multiple nip tasks. arXiv preprint arXiv: /61 1.01587, 2016.
Shankar Iyer,
Nikhil Dandekar, and Kornel Csernai. First
quora
dataset
release: Question
pairs, 2017.
URL httpss:/data. www.ddaaaauuu...oooooooooooo com/
F rst Sett--uora-Dataset--tee
Quest on-Pairs

---

Published as a conference paper at ICLR 2021
Lukasz Kaiser, Aidan N Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, and Jakob Uszkoreit. One model to learn them all. arXiv preprint arXiv:1706.05137, 2017.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond the surface:a challenge set for reading comprehension over multiple sentences. In Pro- ceedings of North American Chapter of the Association for Computational Linguistics (NAACL), 2018.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinske Barwinska, et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):351-3352, 2017.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv: 1942, 2019.
Jason Lee, Kyunghyun Cho, and Thomas Hofmann, Fully character-level neural machine translation without explicit segmentation. Transactions of the Association for Computational Linguistics, 5: 365-378, 2017.
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thir- teenth International Conference on the Principles of Knowledge Representation and Reasoning, 2012.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:1901./1504 2019a.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint Xixx:199777 1692, 2019b.
Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. Modeling task relation- ships in multi-task learning with multi-gate mixture-of-ent -ffture-of-eeperts. In Proceedings of the 24th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 1930-1939, 2018.
Jiaqi Ma, Zhe Zhao, Jilin Chen, Ang Li, Lichan Hong, and Ed H Chi. Snr: Sub-network routing for flexible parameter sharing in multi-task learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 216-223, 2019.
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. arXiv preprint
2018.
James L McClelland, Bruce L McNaughton, and Randall C CO'Reilly. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102(3):419, 1995.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109- 165. Elsevier, 1989.
Jonas Pfeiffer, Ivan Vulié, Iryna Gurevych, and Sebastian Ruder. Mad-x: An adapter- adapter-based frame- work for multi-task cross-lingual transfer. arXiv preprint arXiv:2005.00055 2020.
Mohammad Taher Pilehvar and os's e Camacho-Colladee Wie: 10, 000 example pairs for evaluating context-sensitive representations. CoRR, abs/1808800 2018. URL http:://axxii org/ abs/ bas/1808..m 09121
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 1910. 10683, 2019.
10

---

Published as a conference paper at ICLR 2021
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.052500 2016.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series, 2011.
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint 17X.::7706.00098, 2017.
Victor Sanh, Thomas Wolf, and Sebastian Ruder. A hierarchical multi-task approach for learning embeddings from semantic tasks. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 33, pages 6949-6956, 2019.
Abigail See, Peter J Liu, and Christopher D Manning. Get to the point: Summarization with pointer- generator networks. arXiv preprini arXiv: 011.74444444444 2017.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. arXiv preprini arXiv: 1804.042335 2018.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv: 007000000388 2017.
Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorfloow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414-10423, 2018.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pages 1631-1642, 2013.
Asa Cooper Stickland and Iain Murray. Bert and pals: Projected attention layers for efficient adap- tation in multi-task learning. arXiv preprint 19Xiv:1902.02000 2019.
Ilya Sutskever, Oriol Vinyals, and Quoe V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112, 2014.
Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F Grewe. Continual learning with hypernetworks. arXiv preprint orXiv:/906.0000 2019.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07446 2018
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3261- 3275, 2019.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint arXiv: 12471, 2018
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long Papers), pages 1 1112-1 Association for Computational Linguistics, 2018. hRL.:tt:: web. wra.booog/aaneee www../..rg/aaattoooo///11111//1/
Sen Wu, Hongyang R Zhang, and Christopher Ré. Understanding and improving information trans- fer in multi-task learning. arXiv preprint 4rXiv:2005.00944, 2020.

---

Published as a conference paper at ICLR 2021
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, pages 5754-5764, 2019.
Jachong Yoon, Sachoon Kim, Eunho Yang, and Sung Ju Hwang. Oracle: Order robust adaptive continual learning. arXiv preprint arXiv: arXiv:/902.099 010.9999299222 2019.
Poorya Zaremoodi, Wray Buntine, and Gholamreza Haffari. Adaptive knowledge sharing in multi- task learning: Improving low-resoures neural machine translation. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), pages 636-661, 2018.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv: 1810. 12.80.128800 12885, 2018.
12

---

Published as a conference paper at ICLR 2021
6 SUPPLEMENTARY MATERIAL
6.1 DATASETS
6.1.1 GLUE
The datasets in GLUE are CoLA (Corpus of Linguistic Acceptability) WWrsstatt et al. 2018
Sentiment Treebank SST-2 Socher et al. (0033), Microsoft Research Paraphrase Corpus (MRPC)
(Dolan and Brockett 2005). QQP (Quora Question Pairs) (Iyer et al. 2017). Semantic Textual
Similarity Benchmark (STSB) (Cer et al. 2017), MNLI (Multi-Genre Natural Language Infer-
ence) Williams et al. (018). 8). QNLI (Rajpurkar et al. 2016), RTE (Dagan et al., 2005). Wino-
grad Schema Challenge WNLI (Levesque al. 2012). More details can be found https: glue md
mriithu..com/eensrfiiooo. http:///aaattttttaeeee blbb
6.1.2 SUPERGLUE
The datasets in SuperGI.UE. (Wang et al. 2019) are BoolQ (Boolean Questions) (Clark et. 201 9a CB (Commitment Bank) (De Marneff et al. 2019). CoPA (Roemmele et al.) 2011) (Choice Plau-
sible Alternatives), MultiRC (Multi-Sentence Reading Comprehension Dataset) Khashabi ct al. 2018). Record (Reading Comprehension with Commonsense Reasoning) (Zhang et al. 2018), RTE
(Recognizing Textual Entailment) (Dagan et al. 2005 BBa--Him et al. 20066; Giampiccolo et al. 2007 Bentivogli et al., 2009), Word-in-Conters (WiC) (Pilehvar and os Camacho- ColladooCoolaoos 2018).
and WSC (Winograd Schema Challenge) (Levesque al.] | 2012). We use Tensorflow datasets for loading and preprocessing these datasets. More details can be found at wttptttpss///oo //gt:::::/i/ gtt.bbb.gbbbttgggg// com/
fensorflcw/dataee http://ttttteeeoeboeaaeeeeeeeeee docs cat log/ sttp:/upeer__ee
6.2
EXPERIMENT SETTINGS
This section describes most of the hyperparameter settings for experiments.
6.3
DATASETS AND EXPERIMENTAL SETUP
Our experiments are built upon the existing nate-of-the-art model, T5. We run most of our ex- periments using the base T5 setting, which is comprised of 220M parameters. We fine-tune for maximum of 100K steps. We initialize our models with the released pretrained checkpoints] Our implementation in Mesh Tensorflow (Shazeer et al. 2018). We consider the following setups for the baseline T5 model. First, we compare with the T5 results reported in the original paper (Raffel et al. 2019). These results are denoted with T5t. Second, we compare with T5 (PTFT), which stands for innerain-finetune. In this setup, we fine-tune T5 model for each task individually follow- ing common practice. Finally, we compare with T5 (MTL) which is fair comparison of T5 without HyperGrid. In this setting, T5 is co-trained and results are reported from single model checkpoint selected from the best overall GLUE dev score. Note that in the MTL setting, we co-train GLUE and SuperGLUE within the same model.
Experiments for Base Models For all experiments with base models, train models for 100K steps with a batch size 128. We use the en en_mix mixture which samples each task proportionately to the number of examples in the dataset. Learning rate is constant 0.001 with Adafactor (Shazeer and Stern] 2018). All results for baselines are reported with scores at the last checkpoint. During fine-tuning, the embeddings
fine-tuned. Experiments are run with 16 TPU V3 chips and typically completed in about 8 to 10 hours.
Experiments with Large Models We increased the search for large models to 200K steps pick the best checkpoint for all models based on the best GLUE score. Experiment and hyperparameter settings remain identical although we use 64 TPU V3 chips for finetuning which typically take about 12 hours to complete.
https: / /i thub.
fer- tra former
This model is not directly comparahle as they used less pretraining steps. No dev score results comparable setup reported. We report this score for the sake of completeness.
13

---

Published as a conference paper at ICLR 2021
Experiments with 3B and 11B Models For the large models, only use 1 2 HyperGrid configurations 32x or 32x256 in LG mode for finetuning the model. We submit each model only once the leaderboard Finetuning hyperparameters remain identical. We pick single checkpoint based on the best GLUE score. Finetuning for the 3B model is using 64 TPU V3 chips and the B model is fine-tuned with 128 TPU V3 chips.
6.4 COMPARING WITH OUTPUT GATING
One of the model architecture variants we compared with is Output Gating. It can be formulated as:
Y max W2 + b, (a(UX)11)
Comparing to the HyperGrid, which gates the weights in the Relu layer, output gating directly gates the Relu layer outputs. We can apply either the basic projection method (Equation (2)), or the grid-wise projection method with block-wise projection on layer outputs.
There are two key differences: (1) Output Gating applies sigmoid gating on Relu layer outputs, while HyperGrid applies sigmoid gating on weights before the Relu function. Output gating is similar the Eexture-of-Exper architecture while concatenating the expert outputs. (2) Based on this formulation, the full grid-based projection cannot be applied to output gating.
6.5 FURTHER ARCHITECTURAL ABLATIONS
We include more architectural ablations to supplement the results and findings of the paper.
Ablation Architectures We run experiments for 6 different ablations. The first (1) (4) con- cerning with where to apply HyperGrid. (1) applies on the entire network (all QKV + both FFNs), (2) applies to both positional FFNs, (3) applies on the 1st FFN only and (4) applies on the 2nd FFN only. For (5) and (6), we evaluate on different weighting schemes for HyperGrid. (5) projects HyperGrid
scalar value by pooling (in essense, instead of upsampling the grid, we downsample the grid into a scalar value). As fo (6), this is the setting where the the and column size of the HyperGrid is essentially the same as the FFN.
Ablation
AVG
Baseline
79.40
(1) (QKV Transforms + FFN)
79.05
(2) FFN (both FFN)
80,20
(3) FFN (Ist FFN only)
80.23
(4) FFN (2nd FNN only)
80.26
(5) Scalar HyperGrid
79.20
(6) Max-Grid HyperGrid
80.10
Table 7: More results (SuperGLUE + GLUE) studies on applying HyperGrid to different parts of the Transformer.
Results Our ablation studies show that the best place place the FFN only the 2nd FFN. Using HyperGrid on all the layers, apart from slowing down the network, would also degrade per- formance. This can be observed because the difference between (1) and (2) is the addition of QKV HyperGrids to the model. In (5) and (6), we also note that Scalar HyperGrid performs poorly and worse than the Baseline model. Finally, the Max-Grid HyperGrid performs reasonably well. How- ever, this incurs a huge cost because the hyper network is now larger (as composed to learning small grids and upsampling).
Disccuunting submissions that he incomplete error submissions.
14


Reviews:
[{"review_id": "lMWlBYpFNLp", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose HyperGrid Transformers with a decomposable hypernet-work that learns grid-wise projections to specialize regions in weight matrices for different tasks. Usually, people would use different models to solve different tasks respectively. In this paper, the authors focus on using a single model to solve all tasks and it will save a lot of model parameters for natural language understanding. And the authors have done comprehensive experiments on GLUE and SuperGLUE, and prove that the proposed single model can achieve much better performance than baseline and competitive performance with multiple task-specific models.\n\nPros:\n1. The idea to make use of decomposable grid wise projection is interesting. This is to somehow add regularization to the weights.\n2. The proposed method has been widely evaluated on GLUE/SuperGLUE tasks, and achieve good performance.\n\nCons:\n1. The baseline details are not clear. When using a single model as baseline, how many layers are shared across tasks? What's the sample strategy for different tasks? Is it possible to train a single model on multi-tasks for some steps, then fix most layers and only finetune some task specific layers? I feel the baseline is a bit weak, although I cannot come up with a stronger one that can be easily adapted to the pertained model.\n2. It seems \"Task Conditioning\" is a very important trick. The authors should have some ablation study on it. Or maybe add it to the baseline.\n\nOverall, it's great to see people working a single model to solve all tasks. And I would be happy to increase my score if the authors could convince me regarding the baseline which is quite tricky.\n\n####update####\nThe experiment results are not surprised, but strong enough. Still no very strong baseline provided in this submission, but it might be good to set up a benchmark in this direction. However, T5 model needs more computational resource and the experiment results are hard to replicate. Overall, I would like to keep rating.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "VTlmNdHmg-k", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a HyperGrid Transformer approach to fine-tuning, where one takes a pre-trained transformer model and then modifies it by introducing hypernetworks that modify the 2nd FFN in each transformer block by generating additional weights conditioned on input. These hyper-networks are trained on all tasks in GLUE/SuperGLUE datasets simultaneously and are task aware through prefixing of a task specific token to input. This allows one to fine-tune only a small number of parameters and end up with a model that performs quite well on all tasks at the same time, not much worse than fine-tuning the entire transformer model on all of these tasks.\n\nThis is an interesting paper in the area of hypernetworks with results suggesting potential for impact where one can achieve good accuracies on tasks without having to fully fine-tune gigantic pre-trained models.\n\nI do have some questions for the authors though:\n\n-  local vs global - global seems to be just like learning another weight matrix not conditioned on anything? from the name one would expect local to be conditioned on some specific parts of the input while global is conditioned on entire input. what is the intuition on why they help?\n\n-  what's the intuition between the differences of performance of the various setups (LG, GL, L^2, etc)\n\n-  figure 9 doesn't exist. if coarser variants are better, what happens when the entire weight matrix is treated as 1 block (so you just learn a scalar weight)? what about learning a single scalar for each weight in the FFN (i.e. block size is 1)?\n\n-  have you tried adding dynamic weights to projections in the multi head self attention modules (e.g. to projections for q,k,v)\n\n-  table 1 - why are QNLI results so much worse for HGT(LG) than all other results, but seems to be doing better on most other tasks; how stable are all of your results (i.e. what is the variance across seeds?) \n\n- parameter counts are confusing - how does one compute that a \"multiple networks\" needs 3.2b params, but a \"single network\" needs 0.2b params? are these the trainable weights for each setup, so you count 16 tasks x 0.2b weights? maybe it is better to report total num of params + trained params or else somehow make it more clear what the number of parameters means\n\n- have you tried finetuning the hypernetworks on individual tasks?\n\n---- update:\nThanks for the update. I guess the \"intuition\" is driven mostly by empirical results, which I suppose is ok but may be worth digging into a bit more. I have updated my rating.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "89C2yFpIMRc", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This manuscript presents a HyperGrid Transformer, which is engaged in learning a single model to account for multi-tasks in NLP. The core idea of HyperGrid Transformer is to learn task-conditional dynamic weights in a grid-wise manner in the feed-forward layers, where the weights are factorized in local and global components. This idea is simple, materializing the goal of reducing the parameter cost for the used multi-task network. However, the conducted experiments look nice, showing promising performance on GLUE/SuperGLUE. Therefore, from my point of view, this work is worthy of a publication at ICLR. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes "HyperGrid Transformers" a modified transformer architecture for learning a single model for multiple tasks in NLP.  The proposed method was evaluated on popular GLUE/SuperGLUE tasks and reported competitive results with the baselines (the improvements are somewhat marginal). The paper contains some interesting idea of using a decomposable hypernetwork to learn grid-wise projections for different tasks, which may not be particularly novel in machine learning context but seems new for multitask NLP. Reviewers generally agree the paper is above acceptance bar, however some concerns were raised about clarity of baselines and fairness of experimental comparison as well as stronger baselines. Authors improved some of them in the rebuttal, but there is still some room to further improve the quality of presentation and writing in the final version. 
Decision: Accept (Poster)