Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference
Authors: Tao Lei
Junwen Bai
Siddhartha Brahma
Joshua Ainslie
Kenton Lee
Yanqi Zhou
Nan Du
Vincent Y Zhao
Yuexin Wu
Bo Li
Yu Zhang
Ming-Wei Chang
Venue: NeurIPS 2023 poster (2023)
Keywords: conditional computation
inference efficiency
parameter efficiency
large models
TL;DR: a parameter-efficient transfer learning method that also improves inference efficiency

Abstract: We propose Conditional Adapter (CoDA), a parameter-efficient transfer learning method that also improves inference efficiency. CoDA generalizes beyond standard adapter approaches to enable a new way of balancing speed and accuracy using conditional computation.
Starting with an existing dense pretrained model, CoDA adds sparse activation together with a small number of new parameters and a light-weight training phase.
Our experiments demonstrate that the CoDA approach provides an unexpectedly efficient way to transfer knowledge.
Across a variety of language, vision, and speech tasks, CoDA achieves a 2x to 8x inference speed-up compared to the state-of-the-art Adapter approaches with moderate to no accuracy loss and the same parameter efficiency.

Conditional Adapters: Parameter-efficiet Transfer Learning with Fast Inference
Tao Lei' Junwen Bai Siddhartha Brahma Joshua Ainslie Kenton Lee Yanqi Zhou
Nan Du Vincent Y. Zhao
Yuexin Wu
Bo Li Yu Zhang Ming-Wei Chang
Google
Abstract
We propose Conditional Adapter (CoDA), parameter-effiient transfer learning method that also improves inference efficiency. CoDA generalizes beyond standard adapter approaches enable new way of balancing speed and accuracy using conditional computation. Starting with an existing dense pretrained model, CoDA adds sparse actívation together with small number of new parameters and a light- training phase. Our experiments demonstrate that the CoDA approach provides unexpectedly efficient way to transfer knowledge. Across a variety of language, vision, and speech tasks, CoDA achieves a 2x to 8x inference speed-up compared to the oate-of-thheart Adapter approaches with moderate no accuracy loss and the same parameter efficiency.
Introduction
Large pretrained models have achieved groundbreaking results but the main impediment deploy them has been the cost of adaptation and inference. Due to the ever growing size of the pretrained models, for example, finetuning has become increasingly expensive it requires separate copy of the full model and updates to all parameters for every downstream task. Atrameter-efficiee transfer learning such as Adapter [Houlsby et al. 2019) and Prompt Tuning [Lester et al., 2021] have been proposed to address this issue. These methods only update a small subset of parameters for each downstream task, allowing the model retain knowledge and avoid catastrophic forgetting (Vu et al.) 2022]. Noticeably, these methods can match the accuracy of a fully finetuned model, while achieving better accuracy on out-of-domain data distributions [Lester al. 2021 Awadalla et al., 2022).
Unfortunately, standard parameter-efficiem. transfer learning methods only bring parameter efficiency, not inference efficiency. For example, while only a few small projection matrices are added into the pretrained model in the Adapter approach, all the model inputs (such as tokens) still use all parameters during inference. Therefore, the inference speed is the same (or slightly lower) with respect to the full finetuning method. Moreover, prior studies have shown that these parameter-effie -effiitte--fficinnt learning methods most effective when the size of the pretrained model is large [Lester al. 2021 making many advantages of these methods difficult to realize in practice.
In this paper, we propose Conditional Adapter (CoDA), parameter-efficient transfer learning method that offers both parameter and inference efficiency. CoDA is generalization of the adapter approach, built with the following intuition we treat the pretrained model as universal source of knowledge but only query against for necessary inputs. Figure compares CoDA with finetuning and standard adapter approaches. Similar to standard adapter approaches, our model adds and updates a small adapter in each layer, while fixing the pretrained Transformer blocks for downstream adaptation. Unlike previous approaches, however, CoDA assumes that many of input
"oorespondence: taoleics@gmail.com junvendgoogle.com
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

New param
MNLI (tex))
Acc 1
Speedup
P-Adapter
91.5
CoDA
0.4%
90.7
3.2x
New param
OCR-VQ (vision)
Speedup
P-Adapier CoDA
2.8%
67.5
1.0x
The
col
wapp
on
vee
The
2.8%
67.6
8.0%
Standard #instuninn
Stand Accepptr
Conditoonal Adapherr
New puram
Lbbrispeech (speech)
WER..
Speedup
Tunable paramitions
Frozon parameturs
G Computed plocks
Skipeed
P.Adapter
2.5%
1,4/277
1.0%
CoDA 2.5%
1,4/2.8
2.28
Comparison between different ways to pre- trained Transformer models, including (1) standard finetun- ing (left) where all parameters are tunable and computation is dense, (2) standard adapters (center) where a small set of new tunable parameters are added while the computation remains dense, and (3) CoDA (right) where the computation is sparsely activated.
Figure 1:
Table I: CODA significantly reduces the inference time compared to the
Parallel Adapter approach (He al. 2021]. while still maintaining param- cter efficiency.
token representations (of each layer) are not important for the prediction task and therefore do not require heavy computation. In such cases, the pretrained Transformer block can be skipped. Given that many tokens are not processed by the Transformer block, CoDA runs significantly faster than previous methods.
While conditional activation has clear speed benefits, CoDA must learn to select important tokens for heavy computation in order to maintain model accuracy. To this end, we introduce a soft top-k operation for computing the token selection decision. This soft top-k operation, which be seen as a generalization of softmax and relaxation of hard top-k, utilizes entropy-regularized optimization techniques similar to computational optimal transport [Cuturi 2013). As result, its output can be computed using fast and differentiable iterations, allowing token selection to be directly optimized for model performance.
We apply CoDA on encoder-heavy tasks and evaluate its effectiveness on three different domains natural language processing, computer vision and speech processing. Overall, CoDA achieves 2 to 8 times inference speed-up over standard adapter approach with moderate to no accuracy loss. Table [ showcases our results by selecting one of the best performing tasks each domain. We also conduct comprehensive ablation studies to analyze the effectiveness, efficiency and scalability of CODA. For example, we found that with just little such as T5 [Raffel et al. 22000) can be efficiently converted into CoDA models to gain both parameter efficiency and speed advantages.
router pretraining, existing dense pretrained models
2 Related Work
Parameter-efficient transfer learning methods Due to the ever-growing number of parameters in the pretrained Transformer models, various methods have been proposed for transfer learning with minimal parameter updates. Prompt tuning [Lester et al., 2021] and prefix tuning [Li and Liang 2021] introduce new virtual token embeddings that can be finetuned as model parameters. Adapter approaches [Houlsby et al. 2019 He et al., 2021) add small number of new, learnable parameters to each layer while keeping the pretrained parameters fixed. Another popular method, Low-Rank Adaptation [LoRA; Hu et al.) 2022]. injects learnale low-rank decomposition matrices into pretrained model parameters. In addition requiring less storage cost, parameter- eeffccceent methods have been shown to be more sample-efficient and achieve better out-of-domain generalization than standard finetuning. CoDA is adapter approach but be easily combined with other parameter-efficient methods such as LoRA to accelerate their inference.
Conditional computation The development sparsely and conditionally activated models has been very active research area. For example, Mixture-of-Experts (MoE) models (Shazeer et al.)

---

2017] and many recent advances (Du et al., 2022 Fedus et al. 2021] have been proposed to scale up the size of language models without increasing the computation cost. Many recent works have explored better token routing methods for MoE. models, for example using random hashing [Roller jet al. 202T], balanced assignment [Leiis et al. [02T] and expert-choosing router [Zhou et al. 2022] CODA applies conditional computation to both attention and feed-forward blocks of the model,
whereas MoE models only focus on sparse activation in the feed-forward blocks.
Similar to our approach, various recent methods have achieved computation efficiency by skipping
computation on subset of input tokens. However, the selection mechanism can be very different. such as using pooling [Nawrot et al. 2022], token merging [Bolya et al. 2023). token pruning [Rao al. 2021 Yin et al.] 2022]. J. learned sigmoid gates [Bapna et al. 2020 and early exiting (Schuster
et al. 2022 While most of the token merging and pruning methods have been proposed for vision tasks, we show that CoDA is applicable to multiple domains including text, vision and speech. In addition, token merging and our token selection method are built with different inductive biases and intuition. Token merging leverages redundancies in visual tokens, while token selection assumes a spike of token relevance. That is, only a few tokens are necessary for the prediction task. Another major difference is that CODA dynamically routes and updates token representations in each layer, whereas if token is pruned (or merged), it will never be re-used by subsequent layers. We believe our token routing mechanism is more suited for text and speech applications, such as question answering, where different tokens might play important roles in different layers, or given different input queries.
Finally, CoDA is closely related to concurrent work, CoLT5 [Ainslie al.) 2023]. which also utilizes conditional activation (token selection) for inference efficiency. The focus of CoL.T5 and CoDA are very different. CoLTS specifically tailors its model architecture for long text (e.g. over 16k tokens), for example, by combining local attention with routed attention. The CoLT5 models are pre-trained from scratch and all parameters are finetuned for downstream tasks. In comparison, CoDA is directly initialized and adapted from already pretrained dense model, and we optimize its performance on parameter-efficien transfer learning. The strengths of CoDA and CoLT5 can be combined for long text applications.
Efficient Transformer models Many efficient Transformer variants have been proposed to acceler-
ate model computation. Examples include creating fast attention variants [Wang et al.] 2020aa] Beltagy
et al. 2020 Guo et al., 2022 Hua al., 2022 , searching network architectures [Press al. 2019
So et al. 202 I SS et al. 2021 and utilizing non-attention neural modules for efficiency [Gulati et al.
2020
_ei 2021 CoDA utilizes conditional computation an orthogonal approach for efficiency.
Model compression Apart from building efficient model architectures. model compression meth-
ods such as pruning [Han et al. 2016 Zhu and Guptal 2017 Wang et al., 2020b Xia et al. 2022]
and distillation (Hinton et al. [2015]] Kim and Rush| [016 Ture et al. [2019] Lin et al.J [2220] can
be adopted to speed up model inference. Compared to these methods, CoDA retains all model
parameters of the pretrained large model, and therefore avoids retraining a new model from scratch
or knowledge forgetting caused by parameter removal. In addition, CoDA can be seen as a dynamic version of layer pruning because it can activate different Transformer layers for each token, and can be further combined with distillation to reduce the loss of accuracy caused by conditional computation.
3 Method
3.1 Architecture
Throughout this and the experiment section, we build CoDA on top of parallel adapters (He et al. 2021]. However, note that our method can be generalized to other types of adapters such as sequential adapters (Houlsby et al. 2019] and LoRA [Hu al.) 202T]. We present additional experimental results using LoRA in Appendix B.3, Figure 2 illustrates our architecture and shows how CODA computes its output by selecting only a small subset of input tokens to query against the pretrained model. When parallel adapters are used, CoDA introduces a small number of learnable parameters in the parallel branches, while the vast majority of model parameters (associated with the pretrained Transformer layers) remain fixed. In addition, CoDA only sends k [n/r]] tokens for heavy processing. We define r > as the reduction factor, a constant (such as 4) to control the computation saving.
3

---

Next, we briefly introduce our notations and describe the computation of CoDA in detail. We use F() denote parameterized neural network and the corresponding function defined by the network. For instance, a Transformer layer Vaswani et al. 2017] consists of an attention sub-layer (an() followed by a feed forward sub-layer Fmn(). Each layer also employs layer normalization [Ba et al., 2016).
namely an) and L.Nma (), before applying the attention and feed forward functions. We define
X E R as the input of Transformer encoder layer, where is the number of input tokens and d
the hidden size of the model.
Given layer input X, we first apply layer normalization, namely X nor I.N )) The normalized input will be processed by the adapter branch and the conditional Transformer branch. Their outputs then added and combined the final output of the layer.
Layer output
Pretrained
Adaperr branch Let Fadtperr() () denote the transformation function of the adapter branch. The output defined as
Expenstio
Tunakio Fast adaetar FFN )
Pretrained & Experssive ottenntion 1:(0)
Z mlapper
Fadapter X morm)
Similar to the previous approaches, Fadapter() is realized using a feed forward network with small hidden size such 64. As result, computing Zddaper only incurs a small number of floating point operations and its cost is often negligible compared the cost of the heavy Transformer branch. The adapter branch does not conditionally select tokens. In other words, Fadapte(() is applied all input tokens X 1Rd
all
tokens
Layer imput
Figure 2: Illustration of sin- gle CoDA layer with parallel adapter. tokens are selected and processed by the frozen pretrained Transformer layer, and all tokens are processed by the fast adapter layer.
Conditional branch The computation of the conditional branch takes three steps. First, each CoDA layer defines router function Froueer) to select k tokens for the conditional branch. The router
function in each layer returns two outputs
m, P
F Foutter norm
(2)
where P E {0, 1] kx* is matrix consisting of & one-hot vectors indicating the selection of tokens. Here Pūi,j] if and only the i-th selected token is the j-th input token from X. m E [0, 1]" is a weight mask in which m.j] is the selection weight for the j-th input token. m|j] = 0 if the token is not selected. We will describe how the router learns the selection more details later this section.
After the routing decision is made, the input representations of the selected tokens can be collected using a matrix multiplication,
Xrouted p.
where k rows in X are selected to construct the k-byy- matrix X Similar to standard Transformer layer, the conditional branch applies attention and feed forward transformations to the selected input:
Z Zrouted = Fam (
(4)
Zroused Fito (aNNaa(Souui + Zrouned)
(5)
where Zrouted: Zrouted E R denote the output of the attention network and the feed forward network respectively.
We consider two attention variants which differ in how they compute key-value vectors. One variant applies k--t--k attention using X as both the query vectors and key-value vectors. The other variant applies k-to-all attention using the entire input vectors X nnom as the attention keys and values. The k-to-all variant runs slower but obtains higher quality close to the full model. We compare the performance of the two variants in Section 5
The attention and beed-forward output Zrouted and Zyouted are combined and projected back to the same shape of the original input
Z Zoond P (2outed + Zrouted)
E
(6)

---

Finally Zood merges with the adapter output and the original input of the current layer to produce the output of the layer:
Y == X Z udapter +
Zeond
m @ Zoodd is an element-wise wwie multiplication that scales the rows of Zood using weight m. This operation be seen gating operation, where the hidden state Zood of the i-th token is weighted by the token selection score m [i] assigned by the router. This enables gradient propagation from m to the router parameters, such that the token selection can be jointly optimized with other model components during training.
Learned router An important ingredient of CoDA is the router function Froute() that is learned select subset of tokens for favorable model performance. Given the token representation X norm. our router first computes dot-product score w X X where E IRd parameter vector associated with the router in this layer. The dot-product score is further normalized by a function f0 : IR" + [0, 1]", and clipped produce the selection score m.
X f(s) m o Top(A, k)
(8)
R
Here Top(A, R-) f0, 1]"h is an indicator function which returns binary mask indicating the top-k values A. The oot matrix P defined in (2) can be created according to Top(>, k)). In short, the highest values of X will be selected by the router.
Function S() must remain differentiable with respect to its input (s in this case) such that we can update the router parameters w during training. One possible choice for s() is the sigmoid activation function which normalizes the values the constraint that we need to select & tokens from available tokens. Consider simple case where = 1, a natural choice for f( would be the softmax function. Since softmax provides global normalization over the input scores, gradient update to increase one of the scores would also decrease the other scores, a desirable effect for learning top-1 selection.
independently. However, this does not explicitly model
We hypothesize that soft top-k operator that generalizes softmax should be used for general k > 1. This is indeed possible by formalizing soft top-k as the following optimization problem:
((8) = arg max 8 T +H((A)
s.t. 1 1- A|I Vi 1,...,
(10)
Here H{) = -E - log A is a generalized entropy function (applied to any positive vector is small coefficient.
instead of distribution), and
This optimization problem is closely related to the softmax and top-k operation. Specifically, when = 0, it becomes linear program which returns Top(s, k) the solution. In addition, when & 1, it can be shown that its solution is softmax s/e). Broadly speaking, (10) will return a soft top-k mask and the smoothness is controlled by (and hence must be positive act
temperature).
Problem (T0) does not have closed-fone solution for an arbitrary >> 0 and i > 1, but its solution can be obtained using an iterative algorithm. Specifically, let E R and b E RR be two auxiliary variables (which can be initialized to zeros). The solution takes the form X exp( stbta) exp((+b+ta) values of and be obtained using the following iterative updates:
The
s
+ b(i]
a' cln(k) cIn
Σ exp
E
b' min(-s a' 0)
(11)
iterations and the function f(s) returns exp(#tbta) using a and from the last iteration. The function f(x) remain differentiable with respect to 8 using these iterative updates, so we can train the router jointly with other model parameters. We provide additional discussion and the derivation of the updates in Appendix $
In practice, we use
5

---

Large
Model
Reduction
MNLI
RTE
BoolQ
MNLI
RTE
BaolQ
MNLI
RTE.
BoolQ
A Avg
[w/o conditional computation)
Parallel Adapter
87.1
71.5
77.9
90.3
84.8
85.8
91.5
89.9
88.4
#0.0
CoDA, A-to-all attention
86.6
90.2
CoDA, & k-to-k atkention
86.3
72.2
76.2
89.8
87.0
83.7
91.4
89.5
88.2
-0.3
CoDA, A-to-all attention CoDA, k-to-k attention
86.0
82.5
70.8
75.4
88.1
87.0
81.8
89.9
87.7
84.8
-2.1
Table 2: Results of applying CoDA to T5 v1.1 models. CoDA achieves significant computation savings while retaining accuracy close to the dense baseline. We compare CoDA to corresponding parallel adapter method that processes all tokens without conditional computation. We report accuracy the development set on 3 tasks x 3 model sizes, and set the number of selected tokens &o [n/r]] The last column shows the change on average accuracy with respect to the parallel adapter method. We select the k-to-all version
default (shown in bold).
3.2 Training
CoDA can be directly initialized from an existing Transformer model. Given pretrained model such as T5 [Raffel et al. 2020], the Transformer layers are directly re-used and copied in the conditional branches of CoDA, and only the adapter and router parameters randomly initialized. Because pretraining a large dense model can be expensive, our method reduces the overall training cost.
The routers and neural network components in CoDA must co-operate and be optimized for accurate model predictions. When the available finetuning data is limited, a random initialization for the router (and adapter) parameters can be sub-optimal. We demonstrate that CoDA can be further pretrained using the same pretraining objective as the dense model, order to enhance downstream performance. Importantly, CoDA requires significantly fewer training steps during pretraining, since most of its parameters are taken from an already pretrained model. We show that the cost of CODA pretraining can be 10-30x lower than the pretraining of its original dense model. We present this analysis in Section 5
Finally, we train CoDA on downstream tasks by only updating the adapter, router and layer nor- malization parameters. The size of the adapters is small (e.g- 5M parameters), and each router and layer normalization block only introduces d parameters, where d is the model dimension. As result, CoDA remains parameter-effiient similar previous adapter and prompt-tunin tuning methods.
4 Experimental setup
CoDA is evaluated on three domains including natural language processing (NLP), computer vision and speech processing, and on a range of applications such as classification, question answering, sum- marization and speech recognition. The experiments are organized as follows: We first demonstrate the effectivenss of CoDA conduct analyses on its design choices using the publicly available T5 models ($5). In our final results ($6). we pretrain Transformer models from scratch and extend our evaluation to vision and speech domains.
Datasets We use the C4 corpus Raffel et al. 2020] for pretraining text models. For speech models,
we use the LibriLight corpus [Kahn et al. 2020 for pretraining. Our vision Transformer models use the same data and training procedure in Pix2Struee [Lee et al., 2022). Our finetuning datasets for
text models include the MNLI (Williams et al. 2018 RTE Dagan al.. 2005 Haim al. 20066 Giampiccolo et al. 2007 |entivvgli et al. 2209]] BoolQ Clark et 2019]. SQuAD [Rajpurkar et al. 2016] and XSm [arayan et al. 2018] datasets. The speech models are evaluated on
speech recognition task using the LibriSpeech dataset [Panayotov et al. 2015). Finally, we use the
OCR-VQA [Mshra al. 2019, DQA Mathhw et 20.1 and Screen2 Words [Wang et 202 datasets for vision models.

---

Large
Model
Reduction
MNLI
RTE
BoolQ
MNLI
RTE
BoolQ
MNLI
RTE.
BaslQ
A Avg
Soft top-)
86.3
72.2
76.2
89.8
87.0
83.7
91.4
89.5
88.2
=0.0
Sigmoid gate f($)
85.7
70.8
72.8
89.2
82.3
81.0
90.6
88.1
86.2
2.0
Truncation selecting
81.1
70.8
77.3
82.3
85.6
85.4
Soft top-k
82.5
70.8
75.4
88.1
87.0
81.8
89.9
87.7
84.8
=0.0
Sigmoid gate as f(a)
82.9
71.5
72.1
82.3
80.1
88.3
87.0
Truncation selecting lirst &
62.2
64.6
71.1
64.9
70.4
75.4
66.6
76.2
81.1
-12.9
Table 3: Ablation study on routing methods. We use CoDA k-to-k variant for a fair comparison with the truncation method. Better routing method delivers better accuracy on various tasks and model sizes tested. We use soft top-k as our default method.
Understanding and Analyzing CODA
Setup We present several analyses to validate the design choices of CoDA in this section. We initialize CoDA using the version 1.1 release of T5 checkpoint and perform CoDA pretraining using the same setting as the T5 models. During pretraining. we set routing capacity to k 192 given input sequence length the results of using different values in finetuning. We perform 100K gradient steps, which is 10% of the total number of steps used to train the T5 dense models. The overall computational cost is over 20x less than the full training of dense models, since CoDA only applies heavy computation on less than half of the tokens.
512. We do not tune the value of k for pretraining, but will report
For simplicity, we evaluate on classification tasks for various ablation studies of CoDA. Specifically, we report results on the MNLI, RTE and BoolQ datasets, and test three different model sizes including the Base, Large and XL size of T5. We will extend our evaluation to generation tasks such as question answering in the full result section (
Can CODA be fast and accurate? Table 2 presents the finetuning results of CODA. As a compar- ison, also report the results of Parallel Adapter, which is similar to CoDA except that it applies the expensive Transformer layers to all input tokens. This constitutes an upper- upper-bound, and is a strong baseline that has been reported as the best among range of adapter and prompt tuning methods (He) eet al. 2021]. As shown in Table 2 CODA can achieve 3-5x computation reduction (r = =,55) in the Transformer layers cost of less than 1.0 point drop on average accuracy. As expected, our k-to-all attention variant achieves consistently better accuracy than the k-to-k variant, since it can access the full attention context. On the other hand, the k-to-k attention variant runs faster in practice, which can be beneficial for tasks with very long inputs. We select the k-to-all version in the final result section ($6).
How many pretraining steps are needed? Figure 3 plots the finetuning accuracy by varying the number of pretraining steps for CoDA. Because CoDA can be initialized using pretrained dense models, it requires as few as 20K steps to obtain competitive finetuning results. Of course, using more pretraining steps can improve the downstream accuracy. The fact that CoDA can be quickly updated without repeating the expensive pretrain- ing will be very beneficial in real-world applications.
MYU
Bool
s5
25000
00600
75400
Figure Finetuning accuracy (y-axis) as function of CoDA pretraining steps (x- axis). We show results using 0. 20K, 50K and 100K pretraining steps, and for reduc- tion factor r and 5 respectively. CoDA requires as few as 20K steps to obtain competitive finetuning accuracy.
Does learned routing matter? We analyze the impact of learned routing in Table 3 by comparing our soft top-k router with other router implementations. We implement a variant that replaces soft top-k with the sigmoid activation function, each token activates on its own (without considering the capacity constraint). As shown in the table, this variant
the selection weight of
http:///ggttuub..ooggooeeee resear caasearch//eee wwwterccttteeteeeeeeeeeeeeeeeeee to- text- transfern-f---- hta.ae/rrttrraaffmmemeee//////la geleased_checkpoinee md#t511
7

---

Avg fire taning
Model
Tninell Paruns
Roduction
MNT.
A6c.
A4cc
ICFR
RealQ
SQUAI>>
F1
Refion
F1
XSmm
R2
76.2
Panild Adater
1004
91.5
91.0
68.5
94.8
91.4
21.9
300
76
ColDA
10M
91.3
90.3
87-9
94.1
89.1
70.6
CoDA
1034
90.7
89.5
87.3
93.5
57.6
2022
714
Prefis Turisp [Lind 2021
15M (2M)
20.5
Segurntal Auiuuz
88
Adaptzz MaarEn
IIM
3077
635
-Cod
P-Adtpprr
Base
Lorge
XL
Table 4: Comparison of CODA and parallel adapter on 6 language tasks. We report results the test set of XSum, and on the development set of other tasks. i indicates results taken from He et al. [222]]. J. and referenced results in bracket correspond to using 2M adapter parameters. Note that our Parallel Adapter numbers are stronger as our pretrained Transformer backbone uses more parameters than the model used in He et al. 2021
Figure 4: Average finetuning scores of Parallel Adapter and CoDA at different model sizes.
Speed v.s. Quality
Decoclng speed-up
Speed vss. Quality
CodA
3.7
Smuft
20
3
Large
2.5
Lorge
Lerge
24
14
2
Booe
1.0
X1
Small
89
8
Exsamples second
Reduction
Examples per second
Figure 5: The scaling of CoDA on the XSum and LibriSpeech dataset. Left: CoDA achieves better speed-quality trade-off than finetuning adapters with smaller models, on the XSum dataset. Middle: larger CoDA model achieves higher speed-ups. Right: CoDA achieves better speed-quality trade-off than the dense baseline on the LibriSpeech dataset.
achieves worse accuracy on almost all tasks and model sizes tested, getting 2.0 point worse on average. We also implement a "no-learning" baseline that simply selects the first & tokens, which is equivalent to truncating the input sequence This baseline performs much worse, resulting in more than 10 point decrease in accuracy for small k (and equivalently large r). This analysis confirms the importance of learning a good routing in order to retain strong model performance.
6
Full Results
Setup In this section, we apply our best training recipe to all tasks and application domains. We first pretrain dense Transformer models, followed by the CoDA training procedure in $3.2 Our speech models are pretrained using a masked language modeling (MLM) objective similar to BERT [Devlin et al., 2019), and random quantized output label space (Chiu et al., 2022 Our vision and text models use an encoder-decoder architecture similar to T5 but incorporate a few changes. Following PaLM [Chowdhery et al.) [022], we use multi-query attention [Shazeer 2019) that shares the same key and value projection for multiple query heads. We only use 6 decoder layers and increase the feed forward hidden size (to compensate for the decrease in the number of layers). These modifications have a neutral effect on model quality, but speed up auto-regressive decoding significantly. We will show CoDA is compatible with these changes and can further speed up inference by a considerably large factor. We provide more details of our experimental setup in Appendix A
NLP results In addition to the classification datasets used in Section S models on the SQuAD, ReCord and XSum datasets which require generating an answer or summary
we also evaluate our final
We always include the question text for BoolQ, achieve higher accuracy.
8

---

Model
Base
Large
XL
Model
OCRVQA
DocYQA
Screen2Words
clean
other
clean
other
clean
other
Speedup
ANLS
Speedup
CiDir
Speedup
w2y-BERT BEST-RQ
1.8
2.9
2.9
1.5
29
Aduper
Punilel
110.2
1x
P-Adapeer
1.5
3.5
1.4
3.0
1.4
2.7
CoDA
CODA
80x
4.x
80x
4,6
111.6
108.1
4.6%
CODA
2
1.6
3.5
14
3.0
1.4
2.8
CoDA
12.1x
12.5%
CODA
1.6
3,6
3.1
1.4
288
CoDA
194x
16.7%
104.2
17.8
Table 5: Comparison of CoDA and the parallel adapter baselines on Librispeech. We report the WER results on test-clean and test-other. More results can be found in B..2
Table 6: Comparison of CoDA and the paral- lel adapter applied to a pretrained Pix2Struct model [Lee et al. 2022] on 3 visually-situated language understanding tasks.
Original Image
Layer
Layer 8
Layer 17
Who the author of this book?
Who Buthor book?
FORMULA ONE
FORMULA
FORMULA
FORMULA ONE
ONE
ONE
THE ILLUSTRATED HISTORY
TLEILLUSTRTTER HISTORY
THE TRATED HISTOYY
THEILLUS TRATED HISTTYY
Visualization of routing preferences for CoDA model applied to the OCR- VQA task. Warmer and cooler colors represent higher and lower scores respectively. Router prefers diverse coverage in early layers, but converges selecting sparse and representative patches in later layers.
Figure 6:
given the input. Table 4 contains the finetuning results of XL models. Compared to the parallel adapter baseline that uses full computation, CoDA achieves 3x and 5x computation reduction with only 1.0 and 1.7 point loss average score.
Figure 4 and 5 highlight the scaling trend of CoDA. CoDA runs much faster with slightly worse quality than the parallel adapter baseline. This is expected because the baseline processes all tokens in every layer, whereas CoDA only selects 1/r of tokens for heavy processing. Importantly, this quality gap reduces as the model size increases (as shown in Figure 4 making CoDA a computationally efficient choice for large models. Indeed, CoDA can trade off quality for speed by varying the number of selected tokens. Figure (left) demonstrates that CoDA achieves much stronger speed-quality trade-off compared to dense models without conditional computation. The black line indicates the results of Parallel Adapter when the model size grows from Small to XI., and each blue line represents the speed-quality trade-off of CoDA using r = 1,3, 5. Moreover, Figure (middlee) shows that larger CoDA models exhibit higher inference speed-ups. These observations are consistent on other tasks.
We provide additional results in Appendix B
Speech recognition results We further validate the performance of CoDA in the speech domain. Our model uses a Transformer encoder and a 2-layer LSTM Transducer [Graves] 2012). Similar to NLP setups, test the performance of the speech model on scales Base, Large and XL (see Appendix for details). on word eгтor rate (WER) is consistently minimal on the test-clean and test-other sets of LibriSpeech across different model sizes (and on other sets in şB.2). Moreover, our results are comparable to the top-performing models, such as w2v-BERT [Chung al. 202T and BEST-RQ (Chiu al.) 2022). that are fully finetuned by updating all parameters. Figure 5 (right) highlight again that applying conditional computation leads better quality trade-off compared dense models.
that with sizable reduction ratios (r = 2, 4), the change
Vision results We extend our experiments to visual tasks that involves natural language within the image, such as documents and user interfaces. Our experiments are based on Pix2Struct (Lee et al.) 2022], where an mage-encooder-exx--eeeo
is pretrained by learning to predict simplified HT... from webpage screenshots. Table 6 shows the results on three tasks that were also evaluated in the

---

original Pix2Struct paper. In OCRVQA and Screen2Words, we observe relatively small drops in performance when reducing the number of routed tokens (i.e. patches). When the capacity is 1/16th of the original sequence length, leading to around 13x speedup, we only lose about point. We speculate that this is due to the high-level sparsity in the inputs for these two tasks. For Doc where there is comparatively more textual information, we observe steeper performance-speed trade-off but still achieve a 8x speedup with a 4-point drop.
To provide a more intuitive understanding why CODA works, we visualize the router behavior for the OCR- VQA model Figure 6 We show which patches the routers prefers the most (warmest colors) and least (coolest colors), for several layers. The first, immediately obvious, observation that router avoids cwwffrquency patches, i.e. patches likely to be "whitespace", since they can be adequately handled by the cheap adapter layers. The second, more subtle, observation is that the router progressively converges on a small number of key patches that we hypothesize serve as representations for larger regions. The visualization confirms that CoDA is able to select meaningful and representative patches that are useful for the prediction task.
Conclusion and Limitation
We present CoDA, parameter-efficiee adapter method that enables fast inference. CoDA relies on conditional computation to selectively activate model computation on important input units, providing novel way to balance model expressivity and efficiency.
In this work, we focus on encoder heavy applications such as summarization, speech recognition and visual question answering. by applying our method to the encoder. One limitation of CoDA is that the current routing mechanism (i.e. token selection to decoder-only models for auto-regressive token generation. Enabling fast token generation using conditional activation in decoder layers is an interesting direction we plan to explore in future work.
given sequence) is not directly applicable
Acknowledgements
We would like to thank Rama Pasumarthi, Hongkun Yu, Kelvin Guu, Zhuyun Dai, Timothy Dozat, Raphael Hoffmmann, Tao Wang, Tal Schuster, Ziwei Ji, Frederick Liu and Slav Petrov for helpful advice and discussion.
References
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning heterogeneous systems, 2015. URL http:t/s::/// h/hp::::/wwww... wwwsorfoo..oooooooooooooo. org/ Software available from tensorflow.org.
Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontañón, Siddhartha Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, et al. Colt5: Faster long-range transformers with conditional computation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023.
Anas Awadalla, Mitchell Wortsman, Gabriel Ilharco, Sewon Min, Ian Magnusson, Hannaneh Ha- jishirzi, and Ludwig Schmidt. Exploring the landscape of distributional robustness for question answering models. arXiv preprint arXiv:2210. 2022.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 1607,06450, 2016.
Ankur Bapna, Naveen Arivazhagan, and Orhan Firat. Controlling computation versus quality for neural sequence models. arXiv preprint arXiv:2002.07166, 2020.
10

---

Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. 055000 2020.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing
textual entailment challenge. In TAC, 2009. URL https://tac...... ciscc.nsst... patstt.o//uubicatioon. 2009/additionas papprs/RTES AT5 overview www.iiewwwooeeeings... pdf
Daniel Bolya, Cheng- Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your ViT but faster. In International Conference on Learning Represen- tations, 2023.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclau- rin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, et al. Jax: compos- able transformations of python+ numpy programs. Version 0.2, 5:14-24, 2018.
Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu. Self-supervised learning with sundom-projecting quantizer for speech recognition. In Kamalika Chaudhuri. Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine URL https:://proceedings
Learning Research, pages 3915-3924. PMLR, 17-23 Jul 2022.
mlr. wwt.press/vvv2//er htt16/2cchu2220 html
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:220.002311, 2022.
Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. In 202 / IEFE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 244-250. IEEE, 2021.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), 2019. URL https://aclarthologg org/ N199 1300
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment the PASCAL Challenges Workshop on Recognising Textual Entailment, glikkao/rtets/
challenge. In Proceedings the
2005. hatp://////ww. cs bwu.ac.i........ cec.il/gglkmaoo/roooooooo
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized lls.. arXiv preprint arXiv:2305. 2023.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805, 2019.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-expere PMLR, 2022.
International Conference on Machine Learning, pages 5547-5569.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. The third pascal rec- ognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, 2007. URL
Alex Graves. Sequence transduction with recurrent neural networks. arXiv preprint arXiv: 1211.311..711 2012.
11

---

Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: aaavoltion-auggeent transformer for speech recognition. arXiv preprint 200xx2200000/0000 1000 2020.
Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun- Sung, and Yinfei Yang. LongT5: Efficient text-to-text transformer for long sequences. In Findings of the Association for Computational Linguistics: NAACL 2022. Association for Computational Linguistics, 2022. URL
wwg/2022..c.oo....oo Sindings-naadee
R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. Proceedings of the Second PASCAL. Challenges Workshop on Recognising Textual Entailment, 2006. URL http://u. ctl.cs.biu.. biu, ac ie//inn///pp nwppp/TE/////// halp///E//Proceetee /tt////oooeeei/g/1......
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. International Conference on Learning Representations (ICLR), 2016.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.04366, 2021.
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee. Flax: neural network library and ecosystem for jax. Version 0.3, 3: 14-26, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1003003,,3. 2015.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 1790-2799 PMLR, 2019.
Edward Hu, Yelong Shen, Phil Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le. Transformer quality in linear time. In International Conference on Machine Learning, pages 9099-9117. PMLR, 2022.
J. Kahn, M. Rivière W. Zheng, E. Kharitonov, Q. Xu, P. E. Mazaré, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen. T. Likhomanenko, G. Synnaeve, A. Joulin, A. Mohamed, and E. Dupoux. Libri-light: benchmark for asr with limited or no supervision. In ICASSP 2020 2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 7669-7673, 2020.
Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods Natural Language Processing (EMNLP), 2016. URL https://aclanthollog
org.org///.. /D6g//D1-111 1139
Harold W Kuhn and Albert W Tucker. Nonlinear programming. In Traces and emergence of nonlinear programming, pages 247-258. Springer, 2014.
Kenton Lee, Mandar Joshi, Iulia Ture, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khan- delwal, Peter Shaw, Ming- Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing as pretraining for visual language understanding. arXiv preprint arXiv:2210003347 2022.
Tao Lei. When attention meets fast recurrence: Training language models with reduced compute. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2021. URL. http::ttt///tttt stttps:///allantoology org/ 2021
emnlp-main. 602
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficien prompt tuning. Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
org/ 2021.222..22 emnlp- main. 243
12

---

Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In International Conference on Machine Learning. pages 6265-6274, PMLR, 2021.
Xiang Lisa Li and Percy Liang. Prefix tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.000 000000,, 2021.
Alexander Lin, Jeremy Wohlwend, Howard Chen, and Tao Lei. Autoregressive knowledge distillation through imitation learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020. StRphttps://eet pttttps://aallanholoogg org/
org/2020. 20200.0 emn emnlp-main
494
Andre Martins and Ramon Astudillo. From softmax to sparsemax: A sparse model of attention and multi-label classification. In Proceedings of The 33rd International Conference on Machine Learning, 2016.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference applications of computer vision, pages 2200-2209, 2021.
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text images. In 2019 international conference on document analysis and recognition (ICDAR), pages 947-952. IEEE, 2019.
Shashi Narayan, Shay Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. ArXiv, abs/ 808 000800875, 2018.
Piotr Nawrot, Jan Chorowski, Adrian Lańcucki, and Edoardo M. Ponti. Efficient transformers with dynamic token pooling, 2022.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206-5210. IEFF, 2015.
Ofir Press, Noah A Smith, and Omer Levy. Improving transformer models by reordering their sublayers. arXiv preprint arXiv:1911.03864. 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 2020. URL http://jmmlr haamm.l.rrr///... www.oorg/p...eeeo..oooo. /apers/ v21/20-074 html
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ ques- tions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016. htttp://ts:// stanf edu/puba.
rajjpuraar2066squad pdf
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsich. Dynamicvit: Efficient vision transformers with dynamic token sparsification, In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowyez, Alex Salcianu, Mare van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee- Tee-Th Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy -heppn, Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. Scaling up models and data with t5x and seqio. arXiv preprint arXiv: /7 17189, 2022. URL htttst::::/// :tstts://arxivv //tp:xv..rx////////// abs bbs//220..m 17189
13

---

Stephen Roller, Sainbayar Sukhbaatar, Jason Weston, et al. Hash layers for large sparse models. Advances in Neural Information Processing Systems, 34:17555-17566 2021.
Bernhard Schmitzer. Stabilized sparse scaling algorithms for entropy regularized transport problems. SIAM Journal on Scientific Computing, 41(3):A1443-A1481 2019.
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. Advances in Neural Information Processing Systems, 2022.
Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202 2020.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le. Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely- gated mixture- nixture-of-experts layer. arXiv preprint arXiv:1701.06530 2017.
Jonathan Shen, Patrick Nguyen, Yonghui Wu, Zhifeng Chen, et al. Lingvo: modular and scalable framework for sequence-to-sequence modeling, 2019.
David So, Wojciech Mańke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoe V Le. Searching for efficient transformers for language modeling. Advances in Neural Information Processing Systems, 34:6010-6022, 2021.
Xiu Su, Shan You, Jiyang Xie, Mingkai Zheng, Fei Wang. Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Vision transformer architecture search. arXiv e-prints, pages arXiv-2106, 2021.
Kai Sheng Tai, Taipeng Tian, and Ser-Nam Lim. Spartan: Differentiable Sparsity via Regularized Transportation. In Advances in Neural Information Processing Systems, 2022.
Iulia Ture, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv: 10Xhv:1908.0896,, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
Tu Vu, Aditya Barua, Brian Lester, Daniel Cer, Mohit lyyer, and Noah Constant. Overcoming catastrophic forgetting in zero-shot cross-lingual generation. arXiv preprint 2rXiv:2205.1266 2022.
Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. Screen2words: Au- tomatic mobile ui summarization with multimodal learning. In The 34th Annual ACM Symposium on User Interface Software and Technology, UIST '21, page 498-510, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450386357. doi: 10.1145/3472449.4444666 3474765
URL https: org/ 10.
Sinong Wang. Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.0477 04768, 2020a.
Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2020b. URL https: pestps://acioe http:///cclatthoooggo org fogg/202..ao emnlp-maiileeee 496
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume (Long Papers), 2018.
www.oor-//oo........ 1101
Stephen J Wright. Coordinate descent algorithms. Mathematical Programming, 151(1):3-34, 2015.
14

---

Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruning learns compact and accurate models. In Association for Computational Linguistics (ACL), 2022.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10687-10698, 2020.
Hongxu Yin, Arash Vahdat, Jose M. Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-ViT: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 10809-108 18, June 2022.
Yu Zhang, Daniel S Park, Wei Han, James Qin, Anmol Gulati, Joel Shor, Aren Jansen, Yuanzhong Xu, Yanping Huang, Shibo Wang, et al. Bigssl: Exploring the frontier of large-scale semi-supervised learning for automatic speech recognition. IEEE Journal of Selected Topics in Signal Processing, 16(6):1519 1532, 2022.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon. Mixture-of-expert with expert choice routing, 2022. URL https://arxiv.. org/ wwwgg//bs/222..... 2ttp//222..099966
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy pruning for model compression, arXiv preprint arXiv:1710.01880 2017.
15

---

A Experimental details
Model implementation For our text and vision experiments, we implement our models using JAX BBradbury et al. 2018). Specifically, our training and model modules are built on top of the T5X, Flax and Flaxformer framework [Roberts al.) 2022 Heek et al. 2020 Following the T5 v1.1 implementation and PaLM [Chowdhery et al. [002]] our Transformer models use the GLU variant [Shazeer 2020] as the feed forward network and multi-query-attention [Shazeer [019] as the attention block. These modifications are shown to enhance modeling capacity and speed up decoding respectively.
For the speech experiments, we use TensorFlow [Abadi et al. 2013) and the Lingvo framework (Shen
al. 2019]. The state-of-the-an Transformer variant for speech recognition the Conformer architecture (uulai et al.] [0200 which additionally uses depth- wise convolution in each layer. Since the convolution operation is applied consecutive inputs and does not immediately support routing, we use the standard Transformer architecture |Vaswani et al. 2017) instead. Swish activation is used
the feed forward blocks, following Gulati et al. 2020 We provide the model configuration details
in Table
Model
Num of params
Layers
Num of heads
deoodel
dm
dheud
dady
Text Base
0.IB
12. 6
12
768
3072
128
64
Text Large
0.5B
24. 6
16
1024
4096
128
64
Text XL
2.1B
24.
32
2048
8192
128
Speech Base
0.2B
31, 2
8
768
3072
96
256
Speech Large
0.6B
32, 2
8
1280
5120
160
256
Speech XL
1.IB
32, 2
8
1664
6656
208
256
Vision
0.7B
18,
24
1536
3968
128
256
Table 7: Configuration of Transformer models used in $6 We show the total number of parameters (in billions). dmodel is the model hidden dimension, dun is the intermediate FFN hidden dimension, dread is the attention head dimension and dadpi is the adapter hidden dimension.
Dataset MNLI RTE
Input length
Batch size
Steps
Optimizer
Learning rate
128
256
BoolQ
384
128
300K
Adafactor
0.001, constant
SQuAD
512
ReCord
512
XSum
1664
LibriSpeech
3200
256
150K
Adafactor
0,001, inverse decay
OCR-VQA
4096
256
DocVQA
4096
256
20K
Adafactor
0.01, cosine decay
Screen2Words
4096
32
Table 8: Fine-tuning hyperparmaeters. We use a maximum of 300K steps for NLP tasks following T5 [Raffel et al. 2020). Pix2struct (Lee et al. 2022 uses 10K fine-tuning steps for vision tasks. We use 20K steps as CoDA takes longer to train.
We use the same data and procedure described in T5 [Raffel et al. 2020 BEST- RQ [Chiu et al. 2022 and Pix2struct [Lee et al. 2022] for pre-training the respective text, speech and vision models. .. We use the same training hyper-parametedt such as batch size, input sequence length, the number of pre-training steps and the choice of optimizer and learning rate scheduling. All models have been pre-trained using 128 or 256 TPUv3/TPUv4 chips.
Model training
We run CODA pre-training for text and vision models, using an additional 100K steps and 200K steps respectively. For text models, the input sequence length is 512 and we set the number of selected tokens l: = 192. For vision models, the input sequence contains n 4096 image patches and we set l 1024. CODA pre-training is not used for speech models because there are
16

---

BoolQ
SQuAD
XSum
3/7
X1
2,5
2/3
2
2
1.0
1.0
1.0
1
3
3
(a)
BoolQ
SQuAD
XSum
CoDA
Peichoter
F1 11osor
85
soore
82
Rouge-2
Large
2000
160
<5p
200
7000
Examples per second
Examples per second
Examples per second
(b)
Figure 7: Analyzing the speed and quality of CoDA. We select 3 representative tasks including BoolQ (classification), SQuAD (question answering) and XSum (summarization). (a) Relative decoding speed given different reduction factor r. (b) Speed-quality trade-off of CoDA applied different tasks and model sizes. CoDA achieves better quality than the dense Parallel Adapter baseline when running at a similar inference speed. The black line shows the performance of Parallel Adapter given Small to XL model size, Each blue line represents the performance of CoDA for reduction CoDA equivalent to the dense baseline.
r 3, 5 for a given model size. When
sufficient fine-tuning data. Following standard practice in speech, we use the 1K hour data from the LibriSpeech dataset [Panayotov et al. 2015 and another 30K hour data generated using the noisy
student self-training method [ie et al. 2020 Zhang et al. 2022
Table Aable&list lists the hyper- parameessssssrsss used for fine-tuning, including the sequence length, learning rate, batch size and the number of fine-tuning steps used. For NLP datasets, we set the maximum input length and decoding length to the 98th percentile of lengths in the training set. For vision datasets, we set the input length following the suggested values in Pix2struct. We also find that annealing the number routed tokens can achieve better finetuning results. Specifically, we decrease ie linearly from the sequence length n down to the target value rt/r using the first 10% to 20% of the finetuning steps.
B Additional results
B.1 NLP
Table contaains the complete fine-tuning results on the 6 language datasets. As discussed in $6 the gap between CoDA and its counterpart without conditional computation large at Base size. As model size increases, CoDA retains almost the same level of quality given 3x computation reduction (r = 3). The reduction leads to decoding speed-ups, as shown Figure 7 More importantly, we see that larger model benefits more from CoDA, achieving speed-up factor close to the reduction factor
17

---

Trainable Params
Reduction
MNLI Acc.
RTE Acc.
BoolQ Acc.
SQuAD F1
ReCord F1
XSum R2
Model
Base
Parallel Adapter
88.2
75.8
80.1
91.2
76.7
14.1
CoDA
2M
85.8
68.6
78.7
89.1
69.1
13.5
CoDA
82.8
60.3
78.0
87.3
61.8
13.1
Large
Parallel Adapter
90.5
90.6
86.7
93.9
18.3
CoDA
5M
90.0
84.8
84.3
93.1
84.6
17.0
CoDAA
5
89.4
88.5
83.9
92.2
81.2
16.7
XL
Parallel Adapter
91.5
91.0
88.5
94.8
91.4
22.3
CoDA
10M
91.2
90.3
87.5
94.1
89.3
21.2
CoDA
90.7
89.5
87.3
93.5
87.6
20.7
Table Fine-tuning results on 6 language tasks x model sizes. We report the best results on development set.
Base
Large
XI.
Model
dev
test
dev
test
dev
test
Avg
clean
other
clean
other
cleam
other
clean
other
clean
other
clean
other
w2v-BEKT BEST-RQ
2.9
1.6
3.5
1.7
3.5
1.5
2.8
1.6
2.9
2.7
1.4
2.7
Pamilel Adapter
1.5
3.5
1.6
3.5
1.4
3.0
1.4
3.0
1.4
2.7
1.4
2.7
+0.0
CoDA
CoDA
1.6
3.6
1.6
3.6
1.4
3.1
1.5
1.4
288
1.4
2.8
+0.07
Table 10: Comparison of CoDA and the parallel adapter baselines on all splits (dev-clean, dev-other, test-clean, test-other) of Librispeech.
These results highlight the potential of CoDA for large-scale models, which we plan to investigate in future work.
B.2 Speech
Table extendds Table 5 by including WER results dev-clean and dev-other splits. From the table, one can observe that XL with CoDA (r 2, 4) are consistently better than the Large parallel adapter model on each split, and the Large model with CoDA (r 2, 4) are also consistently better than the Base PA on each split. Given the inference speeds for CoDA models shown in Table larger CoDA models generally faster and better than smaller dense ones (even with PA) with regard to either time cost or computation GFLOPs. Therefore, it is likely for CoDA to help scale up ASR models with decent computation resources and time cost.
B.3 Combining CoDA and LoRA
CoDA can easily combined with other types of adapter methods. To see this, we additionally implemented variant that combines with Low-Rank Adapter [LoRA:[Hu et al.) [02T], which is another parameter-efficient transfer learning method that recently became the most popular choice for LLMs. We incorporate the latest development suggested in the QLoRA paper [Dettmers et al. 2023). which adds low-rank adapters to every linear projection matrix in the Transtormer layers. This is found to obtain better fine-tuning performance than the original implementation. Our CoDA variant with LoRA simply removes the parallel adapter branches and instead adds low-rank adapters to the projection matrices of the pretrained layers.
Table Il shows the finetuning results. The new LoRA baseline achieves stronger accuracy than the Parallel Adapter baseline (84.0 v.s. 82.9 on average), highlighting the effectiveness of recent development on LoRA. In addition, our CoDA variant using LoRA still achieves very close accuracy compared to its dense counterpart (84.0 v.s. 84.0 or 83.7 average). We believe the additional
18

---

Speed Quality
GFLOPs
2.0
1200
Dene
2.6
WER
2.4
400
2.2
2.0
Large
Examples per second
Dense
CODA (1=2)
COCA (1:4)
Figure 8: The scaling of CoDA on Librispeech. Left: Larger models with CoDA can achieve much better performance with the same or faster speed compared to smaller models. Right: GFLOPS reduction is significant as increases.
Base
Large
Model
Reduction y MNLJI
RTE
BoolQ
MNL.
RTE
BoolQ
Avg on
Parallel Adapter (PA)
87.1
71.5
77.9
90.3
84.8
85.8
82.9
CoDA w/ PA
3
86.6
72.6
76.6
90.2
85.9
85.1
82.8
CoDA w/ PA
5
86.0
70.8
76.0
89.7
85.2
84.3
82.0
Low-rank Adapter (LoRA)
88.0
73.7
80.3
90.7
85.2
86.3
84.0
CoDA w/ LoRA
86.2
76.9
78.4
90.3
86.3
85.8
84.0
CoDA w/ LoRA
86.0
76.9
78.3
89.8
86.3
84.7
83.7
Table 1l: Results of applying CoDA with Parallel Adapter v.s. Low-Rank Adapter. We report results on the development sets and on the Base and Large model size.
results strengthen our claims - that CoDA enables strong trade-off between accuracy and efficiency using conditional activation, and this technique can combined with other developments PETL.
C
Soft top-k algorithm
C.1 Derivation of the iterative updates
We present the derivation of iterative updates 1I for solving the soft top-k problem (0) in Section 3 The soft top-k operation is defined maximization problem (10). For the derivation. rewrite as an equivalent minimization problem:
max *+
min
s cH(A)
min 8 X - ell (A) - -el
(12)
s.t. 1 T X = k, A E [0. 1]. 1 1,...,n. n.
Note the term c1 X will be constant e x k. but we include it in the minimization object to make our derivation simpler later.
Now, let E I and b E Rmmh be the Lagrangian variables corresponding to the linear constraints 1 and A[i < 1 Vi
The minimization problem is equivalent to its Lagrangian expression:
min A
max aER
T (H) <1 a(k 1' b
(13)
A 2 1 is already implied by the term ( = Σ. -Aii log A in the objective, due to the use of
log A
19

---

The objective function [2) is strongly convex and the solution space of À is a convex set. As result, strong duality holds so we can instead solve the dual problem. The dual problem exchanges the min and max operators in 13)
max min -8 T eH - T a a(k- R, AER"
(14)
The optimal solution (a, b, A) must have the Marush-Kuhn-Bae Tucker (KKT) conditions hold [uuhn and Tucker
2014 namely
a cH(A) + cl T + a(k AT + (1 21)
8 +
[ + bii]
A exp
[[]
exp
Vi 1,...,. n
Substituting À using the above equation in
14
the dual problem now has simple form:
max a. 1 exp ( 8 1s+a+l
aER,b50
We can solve this problem using coordinate descent WWright 2015 by successively maximizing the function with either a or fixed. That is, we find the optimal that maximizes the dual objective given a fixed b, and vice versa. This leads to the iterative updates [1I described in Section
8 + bi ))
a' = ln(k)) - -eln Σ exp
b' = mim(in (-8 -'800
In short, we obtain the iterative updates of the soft top-k problem (10) by solving its dual problem and by performing coordinate decent the dual variables a and b. The iterative updates are fact the coordinate decent steps.
C.2 The e-scaling trick
The iterations of a and will converge but the number of iterations needed can be very large for small In practice, we only perform small number of iterations and return the corresponding X, which may be close but not the exact solution to (2).. In order to improve the convergence given small number of iterations, 2019]. Let E denote the value at the f-th iteration. We initialize en to a larger value and gradually reduce e, to the target e. Specifically,
apply empirical trick called the e-scaling heuristic [Schmitzer]
set max(Be;-1,6 at the t-th iteration, using scaling constant | E (0, 1). We use Gu = 4 throughout our experiments, c = 0.03 and /3 0.7 for text and vision models and 1.0 and 3 = 0.85 for speech models. Using larger number of iterations leads to better convergence but we found = 20 sufficient for our experiments.
C.3 Overhead of soft top-k iterations
The soft top-k iterations are performed for every routed Transformer layer. Although this seems computationally expensive, the actual overhead is very small compared to the overall decoding latency. The complexity only scales linearly with the number of layers and the sequence length, and does not depend on the model dimension Table [2soeccsssowcceee the latency numbers on the BoolQ and XSum datasets, when performing batched decoding using single TPUv4 chip. We observe that the cost of iterations is less than 2% of the total decoding latency. Moreover, the relative cost decreases dramatically the model size increases, since it does not depend on the model dimension.
C.4 Additional discussion
This iterative algorithm is closely related to the Sinkhorn algorithm of Optimal Transport (OT). Specifically, the Sinkhorn algorithm solves the entropy- regula-regulrrizee version of Optimal Transport (Cu turi 2013 However, our work concerns an different optimization instance. While OT solves transportation problem where the solution space is defined with the marginal constraints over the rows and columns transportation matrix, our optimization problem constrained with a total budget (E, X k)) and upper bounds (A, < 1 Vi). This leads to different iterative updates.
20

---

Task
Model size
Iteration latency (ms)
Total latency (ms)
Base
0.9
48.0
BoolQ
Large
1.7
105.3
1.7
297.6
Base
1.0
513.9
XSum
Large
2.0
1076.2
XI.
2.0
2426.0
Table 12: Latency (in milliseconds) of the soft top-k iteration and the total decoding time per batch. We use a single TPUv4 chip and 128 sequences per batch. The maximum iteration overhead less than 2% of the total latency.
Concurrent to our work, Tai et al. 2022 have used similar linear program (LP) formulation for soft top-k operation, and have applied the operator for learning sparse neural networks (i.e. model pruning). Compared to our formulation (12), they first reduce the LP to an equivalent instance of optimal transport problem, before introducing the entropy term. As a result, the derived updates are different. In addition, Tai et al. [2022] have introduced an initialization for the dual variables to improve the convergence their algorithm, whereas we use scaling instead. Their implementation can be explored for CoDA well.
Besides formulating soft top-k using entropy-regularized optimization, there other possible variants for trainable sparsity. One example is sparsemax [Martins and Astudillo, 2016) that can learn sparse multi-label probabilities. We believe that the sparsemax formulation can generalize from the top-1 to top-k case, but it is beyond the scope of this work. We use the current soft top-k implementation because it natural extension of softmax (see discussions in $3). and because can be solved using simple iterative updates.
D Author Contributions
authors have contributed to running experiments discussing research ideas. Tao leads project, developed the conditional architecture, designed the experiments and analyses. Kenton, Yu and Ming- Wei proposed the idea of applying conditional computation for large model adaptation. Joshua demonstrated the conditional architecture is applicable to attention, and implemented the initial version of conditional attention block. Tao, Yanqi, Nan, Vincent, Yuexin, Ming-Was Wei and Yu conducted the NLP experiments including model pre-training, fine-tuning and various ablation analyses. Siddhartha conducted the majority of the vision experiments. Kenton conducted the vision analysis and advised the vision experiments. Junwen conducted the majority of the speech experiments. Bo and Yu assisted in trouble-shooting the speech models, ran the model pre-training and provided guidance the speech experiments. Finally, Tao, Ming-Wee Wei, Junwen and Kenton made the primary contributions to the writing of the paper.
21


Reviews:
[{"review_id": "xGBgWVCWhw", "reviewer": "Reviewer_yqit", "summary": "The paper introduces a novel technique called Conditional Adapters (CODA), which aims to enhance transfer learning and improve inference efficiency. CODA surpasses the state-of-the-art Adapter approach by achieving a remarkable 2x to 8x acceleration in inference speed while maintaining moderate accuracy loss and identical parameter efficiency. The paper provides examples of language, vision, and speech tasks where CODA has been extensively evaluated, showcasing the promising results obtained through its application.\n", "questions": "See weakness.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper addresses an intriguing problem of accelerating inference speed through efficient parameter tuning.\n2. It is commendable that the proposed method is evaluated across three different modalities, indicating its versatility.\n3. The model's simplicity and effectiveness are notable. The overall design is straightforward, and the method is technically sound.\n4. The paper is well-written and easy to follow.\n", "weaknesses": "1. The availability of the source code is unclear, and it would be beneficial to know if it will be made publicly available for further exploration and reproducibility.\n3. Can not be applied to current main-stream auto-regressive models.\n2. While the paper covers text and speech modalities, for the visual modality, it would be valuable to benchmark the proposed method on tasks such as image recognition and object detection, in addition to OCR-VQA and Doc-VQA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and comments!\n\n&nbsp;  \n**Open sourcing and reproducibility:**\n\nWe will request to make our code available. Open sourcing the code has to go through our employer’s approval process. In addition, we provide detailed descriptions of our implementation and experimental setup in the supplementary materials for better reproducibility of the work. We will be happy to provide more details if needed.\n\n&nbsp;  \n**Limited applicability to auto-regressive decoder:**\n\nWe agree with the reviewer and have mentioned this as our limitation in Section 7. CODA selects tokens in a given sequence, but during autoregressive decoding the output sequences are generated on the fly. \n\nAlthough CODA’s token selection does not apply to autoregressive decoding, the general algorithm of using a light-weight adapter and learning to activate the pretrained model can be useful for future work. While we cannot select tokens along the sequence axis, the soft top-k router is generic and as a result other selection mechanisms can be explored for decoder models, for example:\n- By selecting tokens within the decoding batch\n- By selecting parameters or sub-layers of the pretrained model for decoding\n\nThese are future work we are working on.\n\n\n&nbsp;  \n**Evaluation on other vision tasks:**\n\nWe agree that more traditional vision experiments would be useful. Due to space limitations we decided to focus on the setting that would highlight our strengths. In the case of language understanding within images, the input length is long due to the need for high resolution images. In addition, the sparsity pattern is expected and known a priori because (1) only certain regions of the input image are related to the question, and (2) there is a lot of whitespace (background) in the image. Our VQA experiments can serve as a visualization-friendly sanity check that CODA is able to learn the correct sparsity patterns when they occur in the data, which are indeed observed in Figure 6.\n"}, {"review_id": "SJfSyaJvdz", "reviewer": "Reviewer_XqSH", "summary": "This paper proposed Conditional Adapter (CODA), which can achieve good trade-offs between speed and accuracy. CODA achieves 2x to 8x inference speed-up compared to the state-of-the-art Adapter approach while maintaining moderate to no accuracy loss and the same parameter efficiency across language, vision, and speech tasks.", "questions": "See weakness", "limitations": "See weakness", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method can greatly speed up inference, while keeping competitive performance.\n2. The writing of the paper is clear and easy to understand.\n3. The experiments are extensive, which can also well support the arguments of this paper.", "weaknesses": "1.  This paper proposes a new adapter and claims to accelerate model inference. However,  there is no direct relationship between the adapter and the inference acceleration. The token pruning approach appears more like an independent module. Therefore,  this paper is more like a combination of token pruning and traditional adapters, and both approaches are not new in the community.  \n2. The token pruning  is not new, as there have been many methods applied to ViT. Some methods can even achieve inference acceleration without fine-tuning. The author should disscuss the differences with these methods.\n\nToken Merging: Your ViT But Faster, ICLR, 2023\nDynamicvit: Efficient vision transformers with dynamic token sparsification. NeurIPS, 2021\nA-ViT: Adaptive tokens for efficient vision transformer, CVPR, 2022\n\n3. Based on 1, I'm not quite clear whether this article proposes an adapter method or a pruning method. From the author's descriptions, i lean towards the former. However, in the experimental section, the author lacks a comparison with newly proposed  parameter-efficient transferring learning (PETL) methods, such as   AdaMix, MAM adapter, and so on.\n\nAdamix: Mixture-of-adapter for parameter-efficient tuning of large language models, EMNLP, 2022\nTOWARDS A UNIFIED VIEW OF PARAMETER-EFFICIENT TRANSFER LEARNING, ICLR, 2022\n\n4. It is unclear whether this method can be applied to the currently popular autoregressive models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and comments! Please see our response to the weakness points.\n\n\n&nbsp;  \n**Contributions:**\n\nOne of our major contributions is the insight that the combination of adapters and token pruning is a natural fit. While token pruning is effective for faster inference, it is a destructive process that must be compensated for. Adapter methods introduce a minimal addition that can approximately repair the damage from token pruning. Therefore, our evaluation isn’t more accurate adapters, but having a better trade-off between accuracy, efficiency, and tuned parameters for transfer learning.\n\nWe believe our contributions are the following:\n - We are the first to apply conditional activation to adapter (and PETL) methods to obtain inference efficiency in addition to parameter efficiency.\n - We provide an evaluation on three modalities, validating and highlighting the general applicability of our method.\n - We designed an effective algorithm to improve learning efficiency and model quality, including the soft top-k router and up-training CODA from existing dense models.\n\nWe will discuss the differences with existing token pruning methods in the paper. Some differences include:\n - The suggested token pruning works focus on vision tasks, while CODA is applied on text, speech and vision domains. For instance, CODA is shown to preserve the state-of-the-art WER and accelerate inference on the mainstream speech recognition benchmark.\n - Token merging and top token selection are built with different inductive bias and intuition. Token merging leverages redundancies in visual tokens, while token selection assumes there is a spike of relevance. That is, only a fraction of tokens are most necessary for the prediction task. We believe the latter is more suited for tasks such as question answering, where the information is dense but only a subset of input is relevant to the given question.\n - CODA introduces a new routing scheme. In previous token dropping works such as DynamicViT and ToMe, if a token is dropped (merged), it will never be reused in subsequent blocks. In contrast, CODA dynamically updates the token representations. Even if the token is not selected for updating in one transformer block, it may still get updated in the later blocks. In text and speech tasks, different tokens might play important roles in different layers. CODA can avoid dropping tokens completely for such scenarios.\n\n\n&nbsp;  \n**Evaluation and comparisons:**\n\nThank you for suggesting the PETL methods!\n\nPlease note that our evaluation and choice of baseline are indeed heavily inspired by the suggested work (He et al., 2022; Towards a unified view of parameter-efficient transfer learning). Specifically, He et al. has unified and compared existing PETL methods such as prompt tuning, LoRA, sequential adapters and parallel adapters etc., and found parallel adapters as the best approach. The choice of adapter type in CODA is flexible. Given the reported results from He et al.,  we implement CODA on top of parallel adapters, and select parallel adapters as our baseline for a direct apple-to-apple comparison of dense v.s. conditional models. We will clarify this choice in the paper.\n\nMoreover, we’d like to emphasize that we are proposing a new trade-off between accuracy, speed, and parameter efficiency that existing work has not focused on. Our claim is that CODA enables a new “sweet-spot” along these dimensions. It is possible that using CODA with other adapter / PETL methods (e.g. AdaMix and MAM that use a mixture of multiple adapters) would lead to higher accuracy, but that difference would be orthogonal to our contribution.\n\n\n&nbsp;  \n**Limited applicability to auto-regressive decoder:**\n\nWe agree with the reviewer and have mentioned this as our limitation in Section 7. CODA selects tokens in a given sequence, but during autoregressive decoding the output sequences are generated on the fly. \n\nAlthough CODA’s token selection does not apply to autoregressive decoders, the general algorithm of using a light-weight adapter and learning to activate the pretrained model can be useful for future work. While we cannot select tokens along the sequence axis, the soft top-k router is generic and as a result other selection mechanisms can be explored for decoder models, for example:  \n - By selecting tokens within the decoding batch\n - By selecting parameters or sub-layers of the pretrained model for decoding\n\nThese are future work we are working on.\n"}, {"review_id": "8yDiwWipOT", "reviewer": "Reviewer_2t3i", "summary": "This paper proposes a unique modification over parameter-efficient transfer learning by employing a conditional adapter parallel to the pre-trained model. In doing so, a dynamic token selection mechanism is proposed for each block. The selected tokens undergo heavy pre-trained operations, while all the tokens undergo light-weight adapter operations. This greatly improves the inference efficiency of the target dataset. Experiments from language, vision and speech recognition are shown to test the efficacy of the proposed method.", "questions": "1. Is there a simple softmax baseline instead of the top-soft-k operation?\n2. Is the iterative process for soft top-k done for each forward pass and each layer? If yes, how much overhead does it add?\n3. What is the parameter overhead of CoDA and absolute latency numbers for different tasks?\n4. There are a variety of standard benchmarks for parameter efficient fine-tuning - VTAB-1k (vision), GLUE (language), among others; why did the authors choose specific benchmarks presented in the paper? If possible, can the authors present results on these standard benchmarks?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Wide variety of experiments provided show the superiority of CoDA as compared to parallel-adapter among others.\n2. The overall idea of improving the inference speed of large models during the transfer learning stage has huge practical value.\n3. A large chuck of questions have been addressed in the main text including - iteration wise performance, the importance of routing, etc. ", "weaknesses": "1. The presentation of the work can be improved; after section 3.2, the paper seems unorganised, and the scale and location of tables and plots also require some work.\n2. Thorough comparison with existing works is missing. There are two streams of comparison possible - [1] Parameter-efficient fine-tuning methods [2] Dynamic/conditional inference methods. Even though experiments are shown on a variety of tasks, it is still unclear how this work compares with existing methods.\n3. Final latency with respect to the baseline model is not thoroughly presented, also additional parameters due to parallel branches are not given.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review and suggestions!\n\n&nbsp;  \n**Presentation:**  \nWe agree that the scale and location of the tables and plots need additional work. Due to the page limit, we have to shrink the size of tables and plots and move some results to the supplementary materials. We will improve the presentation in the next version. Thank you for your comment! \n\n&nbsp;  \n**Evaluation and comparisons:**  \nWe would like to give more context on how we select the baseline. Because there is no prior work on applying conditional computation to adapter / PETL methods, our evaluation focused on comparing CODA against the dense parallel adapter baseline. This baseline is very competitive. Recent work (He et al., 2022; Towards a unified view of parameter-efficient transfer learning) has compared existing PETL methods such as prompt tuning, LoRA, sequential adapters and parallel adapters etc., and found parallel adapters as the best approach. The choice of adapter type in CODA is flexible. Given the reported results from He et al., we implement CODA on top of parallel adapters, and select parallel adapters as our baseline for a direct, apple-to-apple comparison of dense v.s. conditional models. \n\nPlease note that we are proposing a new trade-off between accuracy, speed, and parameter efficiency that existing work has not focused on. Our claim is that CODA enables a new “sweet-spot” along these dimensions. We choose to use parallel adapters because it was shown to be the state of the art at the time. It is possible that using CODA with other adapter / PETL methods would lead to higher accuracy, but that difference would be orthogonal to our contribution. We will clarify our choice of baseline in the paper, and are more than happy to hear your suggestions!\n\n&nbsp;  \n**Latency and trainable parameters:**  \nOur current version of the paper only reports speed and trainable parameters in Table 4, Figure 5 and in the supplementary materials. We will report additional numbers in other tables (e.g. Table 5 and 6). Thank you!  \nPlease also see our answers to your questions related to latency and trainable parameters.\n\n&nbsp;  \n### Answers to other questions:\n\n**1. “Is there a simple softmax baseline instead of the top-soft-k operation?”**  \nA: We implemented and trained a softmax baseline for the rebuttal. This baseline performs much worse than the soft-top-k version. We believe using softmax normalization has an issue as the total contribution (i.e. the mask weight) of all selected tokens is at most 1. Each selected token will receive a small weight, and the output distribution will be shifted (scaled down) and be sensitive to the change of sequence length.  \n\nThe table below compares softmax v.s. soft top-k methods for a Large model:  \n\n| Method | Reduction r | MNLI | RTE | BoolQ |\n| --- | :---: | :---: | :---: | :---: |\n| Soft top-k | 3 | 90.2 | 85.9 | 85.1 |\n| Softmax | 3 | 87.2 | 71.8 | 66.8 |\n| Soft top-k | 5 | 89.7 | 85.2 | 84.3 |\n| Softmax | 5 | 84.7 | 70.4 | 66.8 |\n\n&nbsp;  \n**2. “Is the iterative process for soft top-k done for each forward pass and each layer? If yes, how much overhead does it add?”**  \nA: Yes, the soft top-k iterations are done for each forward pass and each layer. The overhead of the soft top-k iterations is very small compared to the total decoding latency. Specifically, the complexity is O(TLN) which does not depend on the model dimension d (T is the number of iterations, L is the sequence length and N is the number of layers).\n\nThe table below showcases the latency (in milliseconds) of CODA and soft top-k iterations, when decoding using a single TPUv4 core and a batch size of 128. The maximum overhead is less than 2% of the total latency, and the relative overhead decreases dramatically as the model size increases.\n\n| Model size | Task | Reduction r | Total latency (ms) | Soft top-k iteration latency (ms) |\n| --- | :---: | :---: | :---: | :---: |\n| CODA Base  | BoolQ | 5 | 48.0 | 0.9 |\n| CODA Base | BoolQ | 3 | 68.0 | 0.9 |\n| CODA Large | BoolQ | 5 | 105.3 | 1.7 |\n| CODA Large | BoolQ | 3 | 164.6 | 1.7 |\n| CODA XL | BoolQ | 5 | 297.6 | 1.7 |\n| CODA XL | BoolQ | 3 | 466.2 | 1.7 |\n| CODA Base  | Xsum | 5 | 513.9 | 1.0 |\n| CODA Base | Xsum | 3 | 736.3 | 1.0 |\n| CODA Large | Xsum | 5 | 1076.2 | 2.0 |\n| CODA Large | Xsum | 3 | 1534.1 | 2.0 |\n| CODA XL | Xsum | 5 | 2426.0 | 2.0 |\n| CODA XL | Xsum | 3 | 3612.5 | 2.0 |\n\n&nbsp;  \n**3. “What is the parameter overhead of CoDA and absolute latency numbers for different tasks?”**  \nA: The parameter overhead of CODA is 2M for Base size, 5M for Large size and 10M for XL size. The parallel adapter baseline uses the same amount of trainable parameters. We report these numbers in Table 9 in the supplementary materials. Please see the absolute latency numbers in the table above.\n\n&nbsp;  \n**4. “Why did the authors choose specific benchmarks presented in the paper? If possible, can the authors present results on these standard benchmarks?”**  \nA: The 4 language understanding tasks we used are indeed standard tasks that are selected in the SuperGLUE and GLUE benchmark. We additionally add SQuAD and XSum to evaluate on generation tasks. Both SQuAD and XSum are commonly used benchmarks as well. We did not evaluate on every single task in GLUE and SuperGLUE, as we have many model variants to evaluate (3 model sizes x 2 reduction rates x 2-4 routing variants).\n"}, {"review_id": "WtgXiiZYBw", "reviewer": "Reviewer_TS35", "summary": "The paper proposes Conditional Adapter (CODA), which not only enables parameter-efficient fine-tuning but is also beneficial for the inference speedup. The main idea of CODA is to pass all tokens to a lightweight routing adapter path while selectively passing only certain tokens to the original path. The selection process, named the “Soft Top-K” operation, is conducted by solving an optimization problem that utilizes a generalized entropy function and an iterative searching algorithm. Experimental results on Encoder-Decoder Transformers in NLP, Vision, and Speech applications demonstrate significant inference acceleration with only minimal performance degradation.", "questions": "* How much is the burden of the iterative searching process (T=20) during training? In other words, how does the training time/performance change as iteration steps T are reduced?\n* Regarding Figure 5(b), how could the decoding be accelerated? As I understand, CODA does not remove the number of tokens, so the input #tokens to the cross-attention would also not be reduced. Additionally, the readability could be improved if Figure 5 is enlarged in the manuscript (minor issue).\n* (Suggestion) I expect CODA can also reduce memory consumption for both training and inference. It would strengthen the paper’s value if there were a comparison regarding memory usage.\n* (Suggestion) CODA currently fixes the reduction ratio r for all layers. I understand that it is a simple and structured way, but how about setting different r for different layers? For example, small/large r for low/high layers. This could provide additional insights.\n", "limitations": "The authors have addressed the limitations in Section 7.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "* The idea of CODA is simple yet powerful. The authors support the effectiveness of the proposed method through sufficient experiments. Although the baseline (Parallel Adapter) achieves higher performance than CODA, I believe the inference speedup can compensate for the performance loss.\n* The derivation and implementation of the “Soft Top-K” operation are particularly unique. This operation outperforms simple sigmoid gating or Top-K truncation. Furthermore, this differentiable function has the potential for a broader range of neural network research.\n", "weaknesses": "* Although the authors briefly mentioned CoLT5, it would be better to discuss more the differences between CoLT5 and CODA. The current explanation (L78-79) is somewhat insufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and suggestions!\n\n&nbsp;  \n**Differences between CoLT5 and CODA:**\n\nWe will add a better explanation in the next version of our paper. While both CODA and CoLT5 use conditional activation (token selection) to enhance inference efficiency, the focus of the two works are very different. CoLT5 focuses on optimizing its architecture for very long text (e.g. over 16k tokens), for example, by combining a local attention module with a routed attention module in each layer. The CoLT5 models have to be pre-trained from scratch, and all parameters are fine-tuned on downstream tasks. In comparison, CODA focuses on parameter-efficient transfer learning. CODA models are directly initialized and adapted from an already pretrained dense model, and only the router and small FFN adapters (containing only 5M parameters for example) are trainable in downstream tasks. The strengths of CODA and CoLT5 can be combined for long text tasks, for example, by using local-attention adapters for transfer learning, and by learning CoLT5 models from dense models.\n\n&nbsp;  \n### Answers to other questions:\n\n**1. “How much is the burden of the iterative searching process (T=20) during training? In other words, how does the training time/performance change as iteration steps T are reduced?”**  \nA: Please also see our answer to Reviewer 2t3i who asked the same question about soft top-k iteration overhead. The soft top-k iterations (T=20) add about 2-5% overhead during training, and 0.1-2% overhead during inference (decoding). The cost is linear with respect to the number of iterations T. So using T=10 for example will reduce the overhead by another 50%. It is also worth noting that the relative overhead becomes smaller for larger models, since the cost of iterations doesn’t depend on the model dimension.\n\n&nbsp;  \n**2. “Regarding Figure 5(b), how could the decoding be accelerated? As I understand, CODA does not remove the number of tokens, so the input #tokens to the cross-attention would also not be reduced. Additionally, the readability could be improved if Figure 5 is enlarged in the manuscript (minor issue).”**  \nA: CODA focuses on encoder-heavy tasks such as classification, summarization, question answering and image understanding. The overall decoding latency is dominated by encoding the input sequence. As a result, decoding can be accelerated since CODA makes input encoding significantly faster than the dense baseline. We will improve the readability and add clarity to Figure 5.  \nFor decoder-heavy tasks, CODA will have limited impact on speed. We listed this limitation in Section 7 and discussed possible future research in our response.\n\n&nbsp;  \n**3. “(Suggestion) I expect CODA can also reduce memory consumption for both training and inference. It would strengthen the paper’s value if there were a comparison regarding memory usage.”**  \nA: Thank you for your suggestion! You are absolutely correct that CODA indeed reduces memory usage. For example, CODA XL model can decode with a batch size of 512 using a single TPUv4 chip for the summarization task. In contrast, the dense parallel adapter model already runs into out-of-memory (OOM) issues with a batch size of 128.\n\n&nbsp;  \n**4. “(Suggestion) CODA currently fixes the reduction ratio r for all layers. I understand that it is a simple and structured way, but how about setting different r for different layers? For example, small/large r for low/high layers. This could provide additional insights.”**  \nA: We agree with your suggestion / intuition. We kept a fixed r for simplicity and for an easy, structured implementation of our model. Using (and even learning) different r for different layers is an interesting and promising direction to further improve the accuracy of CODA.\n"}]

Author rebuttal:
We thank all reviewers for their insightful comments and feedback.

Most reviewers agree that there is an thorough evaluation and analysis in the paper (R1, R2, R3, R4), the tackled problem and proposed technique are important and have a broad practical value (R3, R4), and the paper is very-well presented and written (R1, R2, R4).

The main concerns of the paper include comparison with existing work (R2, R3), and the applicability to auto-regressive decoding (R1, R2).


&nbsp;  
**1. Evaluation and comparisons with existing work:**

We’d like to emphasize that we are proposing a new trade-off between accuracy, speed, and parameter efficiency that existing work has not focused on. CODA is not intended / claimed to be the most accurate adapter (which recent work has made a lot of contributions on). Our claim is that CODA enables a new “sweet-spot” along these dimensions -- that is, adding conditional activation to adapter methods offers a nice trade-off between accuracy and efficiency.

We selected parallel adapter as our baseline because it is reported to be the strongest among many PETL methods (e.g. prompt tuning, LoRA, sequential adapters and parallel adapters) in (He et a., 2022; towards a unified view of parameter efficient transfer learning). It is possible that using CODA with newly proposed adapter / PETL methods (e.g. AdaMix and MAM that use a mixture of multiple adapters) would lead to higher accuracy, but that difference would be orthogonal to our contribution. 

&nbsp;  
**2. Limited applicability to auto-regressive decoding:**

We agree with the reviewers and have discussed this as our limitation in Section 7. CODA selects tokens in a given sequence, but during autoregressive decoding the output sequences are generated on the fly (and therefore not immediately available).

Although CODA is not directly applicable to autoregressive decoder, the general algorithm of using a light-weight adapter and learning to activate the pretrained model can be useful for future work. While we cannot select tokens along the sequence axis, the soft top-k router is generic (R4) and as a result other selection mechanisms can be explored for decoder models, for example:
 - By selecting tokens within the decoding batch
 - By selecting parameters or sub-layers of the pretrained model for decoding

These are future work we are working on.

&nbsp;  
**3. Answers to additional questions:**

In addition to these concerns, reviewers suggested us to compare with another softmax routing variant, report the overhead of soft top-k iterations and also discuss the differences with other token dropping works. We provided the detailed answers to these questions / suggestions in the separate response to each reviewer. We thank the reviewers again and are more than happy to hear additional suggestions.


Meta-review: This paper proposes a unique modification over parameter-efficient transfer learning by employing a conditional adapter parallel to the pre-trained model. The paper introduces a novel technique called Conditional Adapters (CODA) to enhance transfer learning and improve inference efficiency. CODA surpasses the state-of-the-art Adapter approach by achieving a remarkable 2x to 8x acceleration in inference speed while maintaining moderate accuracy loss and identical parameter efficiency. The paper provides examples of language, vision, and speech tasks where CODA has been extensively evaluated, showcasing the promising results obtained through its application.
Decision: Accept (poster)