SHARCS: Efficient Transformers Through Routing with Dynamic Width Sub-networks
Authors: Mohammadreza Salehi
Sachin Mehta
Aditya Kusupati
Ali Farhadi
Hannaneh Hajishirzi
Venue: EMNLP 2023 Findings (2023)
Keywords: Efficiency
Routing
hardness
TL;DR: Our method makes inference more efficient by learning a router on any transformer network, enabling the network to direct different samples to sub-networks with varying widths.

Abstract: We introduce SHARCS for adaptive inference that takes into account the hardness of input samples. SHARCS can train a router on any transformer network, enabling the model to direct different samples to sub-networks with varying widths. Our experiments demonstrate that: (1) SHARCS outperforms or complements existing per-sample adaptive inference methods across various classification tasks in terms of accuracy vs. FLOPs; (2) SHARCS generalizes across different architectures and can be even applied to compressed and efficient transformer encoders to further improve their efficiency; (3) SHARCS can provide a 2 times inference speed up at an insignificant drop in accuracy.

SHARCS: Efficient Transformers through Routing with Dynamic Width Sub-networks
Mohammadreza Salehi
Sachin Mehta Aditya Kusupati
Ali Farhadi
Hannaneh Hajishirzi
Allen Institute for Artificial Intelligence (mrsalehi, sacmehta, kusupati, ali, hannaneh]@cs. washington, edu
University of Washington
Abstract
and token dropping. The former incorporates inter- nal classifiers into intermediate layers of the model. Various techniques have been explored for early exiting, including using confidence scores or en- tropy of internal classifier predictions (Liu et al.. 2020; Xin et al., 2020), using a module that pre- dicts whether layer should exit early (Xin et al., 2021), or implementing patience- based method to adjust these internal predictions (Zhou al., 2020). The other category of sumple-adapll inference methods, token dropping, enhances efficiency by progressively decreasing sequence length as the forward pass proceeds layer by layer (Goyal et al., 2020; Guan et al., 2022).
We introduce SHARCS for adaptive inference that takes into account the hardness of imput samples. SHARCS can train router on any trans- former network, enabling the model to direct different samples to sub-networks with varying widths. Our experiments demonstrate that: (1) SHARCS outperforms or complements existing per-sample adaptive inference methods across various classification tasks terms of accuracy vs. FLOPs: (2) SHARCS generalizes across dif- ferent architectures and can be even applied to compressed and efficient transformer encoders to further improve their efficiency; (3) SHARCS can provide 2 inference speed up at an in- significant drop in accuracy.
In this paper, we propose Sample Hardness Aware Routing based on Confidence Scores (SHARCS!), which is novel category within the efficient sample adaptive inference domain. Our approach introduces training a light-weight router. The router dynamically assigns input samples, based on their hardness (Ethayarajh et al., 2021), to one of the sub-networks with varying widths. Due to the lack of ground truth notion of hardness, we estimate sample hardness heuristically based on the network's 's prediction history during training for each sample. These estimates then used labels train the router.
1 Introduction
Web-scale pretrained models, including Large Lan- guage Models (LLMs), are widely used in vari- upplications (Devlin et al., 2018; Liu et al., 2019a; Brown al., 2020). However, their compu- tational resource requirements can be problematic, especially environments with limited resources. To address this issue, more efficient methods are needed, particularly those that can run on-device with efficient inference (Sanh et al., 2019).
Several methods (e.g.. knowledge distillation (Hinton al., 2015), pruning (Lagunas et al., 2021: Xia al., 2022), and quantization (Shen al., 2019)) have been proposed to improve the inference efficiency of transformer-basee models. While these methods are promising, one drawback is that the resulting model is static. This raises con- cerns about whether the model is too complex for simple samples and not complex enough for more challenging ones. To tackle this problem, previous work have investigated sample-adaptive inference methods that use varying amount of compute to process different input samples (Kaya and Dumi- tras, 2018). Two predominant approaches exist in the field of sample adaptive inference: early-exiting
We make the following contributions:
1. SHARCS introduces router that predicts the hardness sample and can be trained on any transformer network. It enables dynamic in- put processing using sub-networks of varying widths, determined by the predicted hardness.
2. SHARCS delivers substantial efficiency improve- ments in terms of accuracy vs. FLOPs trade-off across different datasets transformer encoders. For example, on QQP (Wang et al., 2017), it reduces the FLOPs of RoBERTapase by 2.75x with only 1% accuracy drop. Compared to other 'pronounced sharks.

---

Emboddings Rdo.
2.1 Estimating Hardness of a Sample
Non-adaptive
x
Our objective is to learn sample hardness aware router, enabling us dynamically direct the input sample to one of the sub-networks with varying ca- pacities (including the main network) for efficient inference. As there are no ground-trute labels that represent hardness, we leverage network's 's predic- tion history during training, We assume that there are M possible hardness levels for sample, with level being the easiest and M being the hardest. Our goal is to assign hardness label ò E {0, 1] MM to samples in the training set and train the router with them. Toward this end, we employ a heuristic procedure which is illustrated Figure 1b: If the model predicts the ground truth class for a training sample with a probability within confidence in- terval [T Tight then the ith entry in the label would be 1; otherwise it would be zero. Here, ranges from 1 to M, and I are hyperparameters associated with hardness level i.
11 E Ruudd
Router
Pooler 0.5
h2 E Rladdax
Adaptive 0.5
x(L-K)
(0.25 0.5) 1.0) reduction factor
his E B
UnPaoler x 0.5
14 Iaaaa
Classifier
Inference.
Input
predited probablly
Conl. scores look up table
Reduction
Cont. intervel
0.43
0.25
[0.4, 1.0]
00.0. 0.5]
w Epocha
0.47
Compare with threshälds
Router labols for epoch We1
Input
Because of the stochastic nature of training, it is possible that the samples denoted as easy ear- lier will potentially be denoted as hard after that, posing instabilities while training the router. To mitigate such randomness, the ith entry is 1 only if the predicted ground truth probability is within the interval for moving window of last W epochs. The assigned hardness labels will be used as labels to train the router in the next epoch. We do not train the router during the first W epochs and just train the network and record the network's predictions. Please see appendix E.0.5 for more details on the role of window size.
f
0.40
(b) Generating hardness labels to train the router.
Figure 1: (a) At inference the router selects the reduc- tion factor. Red parts denote the changes enforced by the router. (b) Training the router. The confidence scores the last epochs for each input sample is recorded. Then they compared with confidence thresholds and labels for training the router are assigned accordingly.
sample adaptive inference techniques, SHARCS either outperforms them or can be paired with them to achieve even greater efficiency.
2.2 Training Sample Hardness Aware Router
3. The gains from SHARCS can be realized real- world deployment with significant latency reduc- tion in CPU-based inference. On QQP, SHARCS can speed up BERT hase more than 2 with less than 1% drop in accuracy.
We split the main transformer network with L lay- ers into non-adaptive and adaptive parts, and in- corporate the router between these networks (see Figure la). The non-adaptive network is comprised of the first 0 0< K < L- layers while the adaptive part consists of the remaining L K layers. The adaptive component shared network that con- sists of sub-networks with varying widths, where the width cach sub-network a fraction of the width of the network.
2 Method
We introduce SHARCS for adaptive inference that takes into account the hardness of input samples. Our approach has three main steps: (1) Obtaining labels that represent sample hardness ($2.1); (2) Training a router that can predict sample hardness ($2.2); (3) Adjusting the network's 's inference ca- pacity according to the predicted hardness ($2.3).
More formally, sub-networks associated with given set of reduction factors (r, [r:),s rs S where the width of i-th sub-network is 1/r;x smaller than that of main network. We map the hard- ness level to the width reduction factor r;. During training, for each input, we sample one of the re-

---

SHARCS
PABEE
DeaBERT
FaSIBERT
BERxIT
Roberta
SST-2
QNLI
QQP
34.8
92.8
91.8
90.0
87.6
$5.0
Accurety
62.5
80.0
75.0
77.5
70.0
75.0
68.0
85.0
72.5
0.25 0.6
0.75
1.0 1.25 Flops
0.0
6.0
10.0
16.0
20.0
24.5
0.0
20.0
40,0
00.0
1023
Flops
Tera Flops
Figure 2: Results the three of the sub-tasks in the GLUE benchmark. For the full set of plots 8 tasks please refer to the appendix D.0.1, Best viewed in color.
duction factors with entry in the router label for that input and do the forward and backward pass with just the network associated with that reduc- tion factor. During inference, given the output of non-adaptive network for input sample x, the ob- jective of the router is to determine the width of sub-network in the adaptive module to process We train the sub-network and the router with the following objective:
detailed description of width reduction of different components in transformer layer.
3 Experimental setup
Datasets. We evaluate SHARCS on 8 classifica-
tion tasks in the standard GLUE benchmark (Wang et al., 2018); MNLI-m (Williams et al., 2018), QNLI (Wang et al., 201 8; Rajpurkar et al., 2016), QQP (Wang et al., 2017), SST-2 (Socher et al., 2013), MRPC (Dolan and Brockett, 2005), RTE (Dagan et al., 2005), CoLA (Warstadt et al., 2019).
C >task Ltask + Arouter Crouter
(1)
WNLI (Levesque al., 2012).
where Lroater a binary cross-entropy loss be- tween predicted and ground-truth hardness label, Lrask is a task-specific loss, and Atask and Arouter are loss weights which are hyper-parameters
Evaluation metrics. Following previous work, report the accuracy on the validation set, with exception CoLA for which we report Matthews correlation. We measure the total FLOPs on the validation set as is invariant to the run time envi- ronment (Liu et al., 2020).
2.3 Reducing Width of Adaptive Module
The basic building block both multi-hcad at- tention (MHA) module and feed forward network (FFN) in transformers is the linear layer (Vaswani et al., 2017). Given the fully-connected nature of linear layers, to reduce their width, we retain the leftmost r dmodel neurons in both input and out- put (Kusupati al., 2022). It is worth noting that as we reduce the input and output dimensions of matrix multiplications by a factor 1/r, the flops will be reduced by a factor of 1 1/y²_ We follow a similar procedure for reducing the width of affine parameters in LayerNorm (Ba et al., 2016). Also in our setup, we do not change the head dimensions in MHA and instead decrease the number of heads by factor of 1/r. As Figure Ia illustrates, down-project the input hidden states to the adap- tive layers using a pooler module and up-project back before feeding to the single classifier for all the sub-networks. Please see Appendix B for a
Training details. We train network with Adam W optimizer (Loshchilov and Hutter, 2019). We choose the number of epochs in {5, 10, 15) and use Icarning rate in {2e - 5, 5e - 5 in our experi- ments. Please see Appendix C for more details.
Results
4.1 SHARCS is Better
We compare SHARCS with existing sample adaptive inference methods: RTJ (Schwartz et al., 2020), DeeBERT (Xin et al., 2020), FastBERT (Liu et al., 2020), BERxiT (Xin et al., 2021), and PABEE (Zhou et al., 2020). Figure 2 shows the accuracy- FLOPs trade-off plots for different sample adap- tive approaches with RoBERTabas model on three GLUE subtasks (See appendix D.0.1 for the full set of plots). Our results show that SHARCS signif- icantly outperforms other methods, especially

---

Table 1: Comparison of different adaptive inference methods on MNLI-m dataset for different FLOP ranges. For the full table, please see Appendix D.0.2
4.3 SHARCS is Fast
We compare the latency of SHARCS applied BERT with the baseline adaptive approaches on the QQP (Wang et al., 2017) dataset. The latency measurements are conducted on an Intel Xeon Gold 6230 CPU with a batch size of 1. We use two reduc- tion factors {0.25, 1.0} and place the router after layer 2 or 4 to get different accuracy-specd up trade offs. We keep the speed up between to 3 and re- port the best accuracy for each method in this range. Table 2 shows that our method obtains higher comparable accuracy in this speed up range. In- terestingly, SHARCS achieves performance on par with DistilBERT, which is trained with distillation on a much larger dataset, but with higher speed up.
FLOPS range (Tera FLAPS)
Time
SHARCS
1 1
FLOPS 1
Acc.
FLOPS
Roberta (Ace: 87.6, FLOPs: 33,5)
0-10
61,83
9.56
76.37
4.91
84.19
85.93
20-30
87.53
29.56
$7.38
28.35
BERT (Ace: 84.8, FL,PP;: 33)
0-10
48.87
8.76
72.7
8.94
10-20
72.50
83:27
83.04
22.76
DistIBERT (Ace: $2.1, FLOPs: 16.57)
0.5
48.57
4.7
64.72
3,30
5-10
64.44
9.95
76.02
8.87
$1.61
DERBERRT 0.25 width (Ace: 83.9, FLOPss: 8.26)
0-1
65.61
3,78
78,92
3.99
4-6
76.01
5.42
$1.48
5.21
83.86
7.92
4.4 SHARCS Paired with Token Dropping
the low-FLOPs regime. This can suggest that by substantially reducing the width of deeper layers in the model, SHARCS can achieve significant reduc- tion in FLOPs. Furthermore, accuracy better compared to fully skipping deeper layers as commonly done in early exiting methods.
As discussed in Section 1, in addition to early- exiting, token dropping another well-known approach to sample-adaptive inference efficiency (Goyal et al., 2020; Guan et al., 2022). Token drop- ping enhances efficiency by progressively dropping tokens layer by layer, thereby decreasing the hid- den states sequence length as the forward pass pro- ceeds. In contrast, SHARCS improves efficiency via reducing the transformer block's 's width. Therefore token dropping and SHARCS should not interfere with each other and in fact, they can be paired to- gether to bring more efficiency. To showcase this, we combine SHARCS with a RoBERTapua. network that has already been made more efficient using Transkimmer token dropping. Table 3 shows that on QNLI dataset, SHARCS reduces the FLOPs of RoBERTa + Transkimmer by 40% with negligi- ble drop in accuracy.
maintain the
4.2 SHARCS Model Agnostic
SHARCS can be seamlessly integrated with any state- of-the-art non-efficient and efficient transformer- based encoders to further improve their efficiency. We SHARCS and the baseline sample adaptive inference methods in 154.1) with two standard non- and
efficient (RoBERTatase and efficient (DistilBERT (Sanh et al., 2019) and Dyn- aBERT (Hou et al., 2020)) models. Table shows that SHARCS consistently improves the efficiency of different models while maintaining the accuracy better than other baseline sample adaptive methods, Note that, for brevity, we have selected the highest accuracy among all the sample adaptive baselines for each FLOPs range; please see Table in Ap- pendix D.0.2 for the full set of results. As an exam- ple, SHARCS can reduce the FLOPSs of BERT base to half with near 3% reduction in accuracy whereas in other methods the drop is more than 12%. Inter- estingly, SHARCS can further improve the efficiency of already optimized networks. For instance, infer- ence with SHARCS on DynaBERT 0.25 width takes 10-15% less FLOPs with less than 1% drop in ac- curacy. More accuracy vs. FLOPPs trade-off results for DynaBERT 0.25 and DistilBERT can be found in Figure 10 and 11.
Table 2: Inference speed up results on the QQP dataset on CPU for various adaptive inference tech- niques applied to BERT. For SHARCS we place the router after layers and 4 and report two numbers.
Accuracy (%)
Speed up (x)
BERT
90.90
DistiBERT
88.50
BERxiT
85.60
2.04
FastBERT
83,99
2.17
DecBERT
83,82
2.14
PABEE
89.09
2,06
RTJ
84.44
2
SHARCS (layer 2)
88.43
2.57
SHARCS (layer 4)
90.05
2.05

---

Table 3: SHARCS can be applied to RoBERTa in addi- tion to Transkimmer to bring more efficiency. The decrease in FLOPs and Accuracy in parantheses are with respect to RoBERTa Transkimmer.
widely used classes of models for which we plan integrate SHARCS in the future work. It should be noted that although our approach can be applied to regression tasks with minor adjustments, this work does not include any results for regression tasks.
QNLI Ace.
ToraFLOPs
RoBERTa Base
92.8%
24.5
Acknowledgements
Transkimmer
89.45%
14.08
-ranskimmer SHARCS
88.83% (-0.62%)
8.56 (~40%%.
We thank members of RAIVN and H2Lab for help- ful discussion and feedback, and Hyak cluster team at the University of Washington for infrastructure support. This research was supported by NSF IIS- 2044660, ONR MURI N00014- 18-1-2670, and gifts from AI2, Google and Apple.
4.5
Ablating the Router
To show the efficacy of the router and its training strategy in SHARCS, we replace it with the routing strategy used in BERxiT (Xin et al., 2021): while training two sub-networks with reduction factors (0.25, 1.0]. we feed the training sample to both of them: if sub-network correctly classifies the sample, the router label for that sub-network in that iteration would be 1. Otherwise, it would be zero. The backward pass is then done with all the losses of sub-networks and the router loss. Table 4 shows the area under the normalized accuracy FLOPs curve (AUC) for both methods on MNLI- m. Please find more detailed ablation results and experiments in Appendix E.
References
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hin- ton. 2016. Layer normalization.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. CoRR, 0as/2005.14166
Table 4: While using adaptive width with RoBERTagss on MNLI-m, SHARCS router outperforms BERxiT router.
Router on RoBER
AUC
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment chal- lenge. In Proceedings of the PASCAL Challenges Workshop on Recognising Textual Entailment.
SHARCS
0.78
BERxiT
0.73
5 Conclusion
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lec, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language under- standing. CoRR, 1 10.04805.
We presented SHARCS new sample adaptive in- ference approach that can improve any network's 's inference efficiency. SHARCS incorporated a light- weight router which is trained with a novel ap- proach using the confidence of the network predic- tions during training. Our experiments showed the superiority or complementary role of SHARCS com- pared to other sample adaptive inference methods across various datasets and backbones.
Devvrit, Sneha Kudugunta, Aditya Kusupati. Tim Dettmers, Kaifeng Chen, Inderjit Dhillon, Yulia Tsvetkov, Hajishirzi Hannaneh, Sham Kakade, Ali Farhadi, and Prateck Jain. 2023. Matformer: Nested transformer for elastic inference. arXiv preprint 231.v2310..77077.
William Dolan and Chris Brockett. 2005. Automati- cally constructing a corpus of sentential paraphrases. In P Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Limitations
While the router and training approach in SHARCS are general purpose, limitation of this paper is its focus solely on studying the impact on trans- former encoders. Nonetheless, decoder-only (Rad- ford et al., 2018, 2019) and encoder-decoder (Raf- fel et al., 2019; Lewis et al., 2020) models are
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. 2021. Informatiin--hhorrtiic measures of dataset difficulty. CoRR, abs/21 10.08420.
Saurabh Goyal, Anamitra R. Choudhury, Saurabh M. Raje, Venkatesan T. Chakaravarthy, Yogish Sabhar- wal, and Ashish Verma. 2020. Power-bert: Accel-

---

crating bert inference via progressive word-vector elimination.
Weijie Liu, Peng Zhou, Zhiruo Wang. Zhe Zhao, Haotang Deng, and Qi Ju. 2020. FastBERT: self- distilling BERT with adaptive inference time, In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 6035- 6044, Online. Association for Computational Lin- guistics.
Yue Guan, Zhengyi Li, Jingwen Leng, Zhouhan Lin, and Minyi Guo. 2022. Transkimmer: Transformer learns to layer-wiss wise skim. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics VVolume 1: Long Papers), pages 7275- 7286, Dublin, Ireland. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyamov. 2019a. Roberta: A robustly optimized BERT pretraining approach. CoRR,
Dan Hendrycks and Kevin Gimpel. 2016. Bridging non- linearities and stochastic regularizers with gaussian error linear units. CoRR, abs/1600.004155.
Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang. Kwang-Ting (Tim) Cheng, and Jian Sun. 2019b. Metapruning: Meta learning for au- tomatic neural network channel pruning. CoRR, abs/1903.10258.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network.
Lu Hou, Zhiqi Huang, Lifeng Shang. Xin Jiang, Xiao Chen, and Qun Liu. 2020. Dynabert: Dynamic bert with adaptive width and depth. In Advances in Neural Information Processing Systems, volume 33, pages 9782-9793. Curran Associates, Inc.
Ilya Loshchilov and Frank Hutter. 2019, Decoupled weight decay regularization.
Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? In Ad- vances Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019. Tinyhert: Distilling BERT for natural language un- derstanding. CoRR, ats//999..oom 10351.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language under- standing by generative pre-training.
Yigitcan Kaya and Tudor Dumitras. 2018. How to stop off-the-shelff deep neural networks from overthinking. CoRR, 0to//881000055..
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Sehoon Kim, Amir Gholami. Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. 2021. 1-bert: Integer- only bert quantization. International Conference on Machine Learning (Accepted).
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text trans- former. CoRR, abs/1910.10683.
Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanu- jan, William -nwdrr-Snyder. Kaifeng Chen, Sham Kakade, Prateek Jain, et al. 2022. Matryoshka repre- sentation learning. Advances in Neural Information Processing Systems, 35:30233-30249.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods Natu- ral Language Processing, pages 2383-2383 2392,, Austin, Texas. Association for Computational Linguistics.
François Lagunas, Ella Charlaix, Victor Sanh, and Alexander Rush. 2021. Block pruning for faster transformers. CoRR, abs/2
Hector J. Levesque, Ernest Davis, and Leora Morgen- stern. 2012. The Winograd Schema Challenge. In Proceedings of the Thirteenth International Confer- ence on Principles of Knowledge Representation and Reasoning, KR 12. pages 552 551. AAAI Press, Rome, Italy.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR. abs/1910.01108.
Victor Sanh, Thomas Wolf, and Alexander M. Rush. 2020. Movement pruning: Adaptive sparsity by fine- tuning. CoRR, abs/2005.07683.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sameeneett-seqqence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online, Association for Computa- tional Linguistics.
Roy Schwartz, Gabriel Stanovsky, Swabha Swayamdipta, Jesse Dodge, and Noah A. Smith. 2020. The right tool for the job: Matching model and instance complexities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6640-6651, Online. Association for Computational Linguistics.

---

Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. 2019. Q-BERT: hessian based ul- tra low precision quantization of BERT. CoRR, abs/1909.05840.
(Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
Mengzhou Xia, Zexuan Zhong, and Dangi Chen. 2022. Structured pruning learns compact and accurate mod- els.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631-1642. Seattle, Washington, USA. Association for Computational Linguistics.
Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. 2020. DeeBERT: Dynamic early exiting for accelerating BERT inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2251,, Online. Association for Computational Linguisties.
Tianxiang Sun, Xiangyang Liu, Wei Zhu, Zhichao Geng, Lingling Wu, Yilong He, Yuan Ni, Guotong Xie, Xu- anjing Huang, and Xipeng Qiu. 2022. A simple hash-based early exiting approach for language un- derstanding and generation.
Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin. 2021. BERxiT: Early exiting for BERT with better fine-tuning and extension to regression. In Proceed- ings of the 16th Conference of the European Chap- ter of the Association for Computational Linguistics: Main Volume, pages 91-104, Online. Association for Computational Linguistics.
Zhiqing Sun, Hongkun Yu, Xiaodan Song. Renjie Liu, Yiming Yang, and Denny Zhou. 2020. MobileBERT: a compact task-agnostic BERT for resource- devices. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 2158-2170, Online. Association for Computa- tional Linguisties.
Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAAley, Ke Xu, and Furu Wei. 2020. Bert loses patience: Fast and robust inference with early exit. In Advances in Neural Information Processing Systems, volume 33, pages 98330-1834) Curran Associates, Inc.
Surat Teerapittayanon, Bradley McDanel, and H. T, Kung. 2017. Branchynet: Fast inference via early exiting from deep neural networks. CoRR, abs/1709.01663.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, ukasz Kaiser, and Miaaa Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sen- nrich, and Ivan Titov, 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. CoRR,
Zhiguo Wang, Wael Hamza, and Radu Florian. 2017. Bilateral pulti-perspective matching for natural lan- guage sentences. CoRR, abs// 702.03814.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bow- man. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625 641.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A proer-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume

---

A Background and Related Work
isis H, Obead E R.dhest and WK, Wo. Wy R Xdeoad are the key, query, and value projec- tion matrices, and dhead = dmodet/ The out- puts from different heads are concatenated into Oheads E Rdmmll and projected with another ma- trix Wo E Rd Xduoaaxddmaa and passed through a layer norm (Ba et al., 2016) to get the output: OMHA LayerNorm(z + Wooheads).
Efficient non-adaptive inference In the context of non-adaptive methods, numerous studies in the literature have improved the inference efficiency of transformers through various techniques such as knowledge distillation (Hinton et al., 2015) into smaller models (Sanh et al., 2019; Sun et al., 2020; Hou et al., 2020; Jiao et al., 2019), pruning unim- portant weights of the model (Xia et al., 2022; La- gunas et al., 2021; Sanh et al., 2020: Liu et al., 2019b), and weight quantization (Shen al., 2019; Kim al., 2021) to store weights of network with lower precision values. The aforementioned approaches result smaller and more efficient al- beit fixed and static models.
FFN consists of two feed-forward layers Wi and W2 with GeLU (Hendrycks and Gimpel, 2016) non-linearity and takes the output of MHA module and computes LayerNorm(oMIa + GeLU (0MM)))))))
B.0.2 Reducing Width of a Transformer Layer
Efficient adaptive inference. Another line of work have proposed adaptive inference methods that allow the network to allocate varying amounts of compute for each sample. The predominant tech- nique in this area is early exiting via adding internal classifiers to intermediate layers (Schwartz al., 2020; Kaya and Dumitras, 201 8; Xin et al., 2020; Tecrapitayanoon et al., 2017; Zhou et al., 2020; Liu et al., 2020): To early exit, prior work either use the confidence score of internal classifiers' predictions (Schwartz et al., 2020; Xin et al., 2020); the entropy of these predictions (Liu et al., 2020); a module that predicts whether layer should early exit or not (Xin et al., (021);& patience based change in these internal prediction (Zhou et al., 2020); hash based mechanism do token level early exiting (Sun et al., 2022).
Figure 3 illustrates our approach in reducing the pacity of transformer network based on the given reduction factor. We leave the first K layers (aka non-adaptive module) of the model intact, where K is hyperparameter. Given an input sequence 2, after computing the intermediate representation h E Rixdmatel by the non-adaptive module, and the reduction factor r by the router, we reduce the width of different components which we will de- scribe in detail next.
Before passing h to adaptive layers, we pass it through a pooler module that reduces its dimension- ality to dmoueel X r. For example, if dmoded = 768 and r r= 0.25, the input to the adaptive module will be 192 dimensional. Although we experimented with affine transformations for the pooler, in prac- tice we observed that simpler pooling operation such as selecting just the first dmodet X r dimen- sions works as good. Therefore, as this pooler is simpler and importantly adds no extra FLOPs to in- ferece, we opted for that. For the unpooler, we used an affine transformation that transforms the hidden state with reduced dimensionality (i.e. r dmodel) to dmodet- Note that for the unpooler layer, the dimen- sionality of the input can change while the output dimensionality if always dioodl--
Devvrit et al. (2023) recently proposed Mat- Former that can enable adaptive compute based on the resource constraints but does not utilize dy- namic token-based routing making SHARCS com- plementary to it.
B Implementation Details
B.0.1 Transformer Networks
Transformer networks (Vaswani et al., 2017) are composed of stack of L layers and each layer has two main components: Multi-head attention (MHA) and Feed-forward network (FFN).
Throughout this paper. whenever we decrease a dimension of a component (e.g. input vectors or weights) from d to d', we exclusively use the first d' dimensions and disregards the remainder in the computations. This is illustrated schemati- cally in Figure 4 for FFN and MHA components in transformer models.
MHA consists of nig heads where each head computes the attention operation (Vaswani al.. 2017) on the projection of a sequence of tokens x = (I1, ..., ..., 2ι) into key, query, and values: o i = Attention(W,, 2x Wqr, W Wvr), wherein
In what follows we describe how we reduce the width of different components by the reduction

---

Embeidings c
alter the sequence length of the input (i.e. 1) or any of the hidden states.
Non-adaptive
k
Reducing FEN width. Similar to the output pro- jection W, of MHA, the imput and output dimen- sions of the feed-forward layers W1 W1< and W2 in the FFN are reduced by factor of /r. Therefore, the output dimension of all the adaptive layersare reduced by factor of to Doodel X
h, Tee
Router
R -aa
is important to note that majority of operations in FFN and MHA components are comprised of matrix multiplications and reducing the input and output dimensions of these multiplications by a factor of rj will reduce their flops by a factor of r).
key
Query
Value
(0:25, 0,5, 1.03 reduction factor
Out projection
Reducing layernorms width. We also reduce the width of the layer norm parameters by factor of r. In our experiments, we find beneficial initialize the layernorm parameters with the corresponding first dimensions of original layernorm parameters instead of training them from scratch.
Ads Aoom
(L-K)
700
W2
Add S.Nom
B.0.3 Inference
In contrast to training where the width of the adap- tive layers doing forward pass enforced by the router labels, at inference, the router predicts the reduction factor. More formally, given the input sample and router logits W E RMM, we select the reduction factor rj to do the forward pass where is arg max(W)
133
Classifier
Figure 3: Detailed schematic of reducing with of dif- ferent components in a transformer model based on the prediction of the router.
C Training Details
factor which is also depicted in detail in Figure 3.
Similar (Hou al., 2020), reorder the heads based on their importance before fine-tuning on the down-stream task. We set Atask to I and Arouter to 0.5 in our approach and use a set of two reduction factors {0.25, 1.0}. We choose the batch size in [16, 32) depending on the model and dataset. We do grid search over the lower and upper bound on the confidence thresholds and do the forward pass of each reduction factor on a separate GPU. We choose the window size values in {3, 5). We set Atask to I and Arowter to 0.5 in our approach and use a set of two reduction factors (0.25, 1.0). We do a grid search over the lower and upper bound on the confidence thresholds and the hyperparamters of the baselines and do the forward pass of each reduction factor separate GPU. Similar to pre- vious work (Hou al., 2020), we pad the input sequences in batch 128 tokens while training. To train SHARCS with Transkimer on RoBERTa, we first train RoBERTa+ + Transkimmer with skim co- efficient of 0.3. We then start from this model as
Reducing MHA Width. We reduce the width of
MHA using the following steps:
1. The input dimension d'oudel of self-attention projection matrices W K. Wo. and W, is de- creased to dmodel r. We do not change the output dimension dhead of the linear projections.
The adaptive module only computes the output of the first n4 heads and disregards the other heads. Therefore the dimensionality of Ohends is de- creased to dmodet- Note that we could also reduce the dimensionality of each head instead, however, we built on previous findings in the literature that many of the heads in transformers are redundant and can be pruned without hurting the performance (Michel et al., 2019; Voita et al., 2019).
3. For the output projection W. of MHA, the input and output dimensions, which are both equal dmootel. are reduced diwslel r.
Throughout all the changes above, we do not

---

Input
Weight
blss
Inpar
Heed projection weights heads, Head projection besees
12
D. 12
@ D
D
heed
Prad
0.5 reduction
6.25 rofluciin
3
@
+
D
D
horl
(a) FFN
(b) MHA
Figure Reducing the width FFN and MHA adaptive layers.
an initial checkpoint and train for 10 epochs using both our loss and Transkimmer loss. Here, train- ing the router is done with with 0.25 and 1.0 as reduction factors.
85
DERaiT louter SHARCS Rouflr
D Results Details
MIL Accuracy
D.0.1 GLUE Results
62
We do a thorough hyperparameter search with dif- ferent sample adaptive inference methods to get the accuracy of the RoBERTabase model across dif- ferent FLOPs values. We report the total number of FLOPs the validation set and do not pad the input sequences before feeding to the model. Fig- ure 12 shows the plots for 8 of the sub-tasks in the GLUE benchmark.
80-
76-
74
Figure 5: Accuracy vs. FLOPs plot for Adaptive width sub-networks trained with SHARCS router (orange) and BERxiT (Xin et al., 2021) router (blue).
D.0.2 Different Backbone Results
Table shows the comparison of different ple adaptive inference methods applied different backbones.
F.0.2 Number of Reduction Factors
E Ablations and Discussion
We change the number of reduction factors (or hardness levels) M to three and four and use the reduction factors {0.25, 0.5, 1.0] and {0.25, 0.5, 0.75, 1.0) respectively. Figure 6 shows the results of changing the number of reduction fac- tors with RoBERTabae model MNLI-m dataset. The model trained with two reduction factors out- perform the other cases for FLOPs values above 15 teraFLOPs significantly and four reduction factors can get better results the range 10-15 Tera FLOPs.
E.0.1
Router Ablation
Figure 5 shows the accuracy FLOPs plot for two methods: 1) The router introduced in SHARCS with adaptive width sub-networks. 2) A router similar the early-exiting router used in BERxiT (Xin al., 2021) with adaptive-width withh sub-networks. We report the AUC for similar range of FLOPS for both methods and scale the accuracy and FLOPs values to [0, 1).

---

Table 5: Comparison of different adaptive inference methods on MNLI-m dataset for different FLOP ranges.
FLOIS runge (Tera Flope)
KTJ
DedBEK
FABLE
FaSBERT
BERST
SHARCS
L
Ace.
Reberts
0:10
54,27
9.59
9.56
2.84
8.43
76.37
4.91
10-20
K2.29
19.77
99.79
85.13
77.24
BERT
0-10
47.53
9:46
42.32
9.97
42.97
48. K7
8.76
66.85
9.88
72.7
8.94
70.25
DistIBERT (Acc: 82.2, ELOPS: 16.57
0:5
41.92
2.76
43.98
2.76
41.45
2.76
48.87
4.7
41.47
4.077
64.72
3,80
5:10
52.72
9.55
72.93
13,90
14.12
70.72
11.58
DYnABERT 0.25 width (Acc: 83.9, FLOPS: 8.26)
0-4
60.95
3.98
56.13
3.38
56.09
3,97
65.61
3.78
56.19
3.55
78.12
3,99
6-6
74.55
5.82
15.26
69.81
5.90
76.01
5.62
61.48
5.21
83.43
7,60
83.40
32530
carfisnnce threcoo covRsonce Brratted
225,0/11
66
225 02..0...77
02
corBsance
24
84
KNN
Figure 6: Results different number of networks (or reduction factors) with RoBERTakus model on MNLI-m.
Figure 7: Results of changing confidence thresholds with RoBERTa model MNLI-m dataset. We use reduction factors (0.25, 1.0) and confidence thresh- olds [0. for reduction factor 0.25 and confidence thresholds [x, 1] for reduction factor 1, where r e {0.5, 0.7.0.93.
E.0.3 Confidence Thresholds
We study the effect of changing confidence thresh- olds results with simple experiment: With reduction factors {0.25, 1.0} we use the fol- lowing confidence score lower and upper bounds: x for full network and [ar, 1.0) for 0.25 net- work, where æ E {0.5, 0.9. We place the router after layer (i.e. = 1). Figure shows the results of RoBERTabase model on MNLI-m dataset. According to the figure, higher value of x leads to better acuracies for fixed number of FLOPs. Furthermore, we decrease x, can reach lower values of FLOPs. This intuitive as decreasing the lower bound smallest reduction factor makes more samples be routed to that which reduces the overall FLOPs.
E.0.4 Using Entropy or Confidence-basee Hardness Labels to Train Router
Similar to sample adaptive inference in carly ex- iting methods (Xin al., 2020; Liu et al., 2020), one get hardness labels by defining thresholds entropy of the network predictions instead of the confidence. We use the following formula compute the entropy of network's prediction:
C
=
Σ
ps log
wherein C' denotes the number of classes of the classifier, and p; is the softamx probablity that the

---

PLEEPIO EXIT
Intropy Confidence
REOR 20 SUYT
85-
83.5
PLSPP 1:4.391
23.0
MNJ.m Accuracy
PLLAN VIRAUT
20-
82.5
POOPIRO SENT
20-
DynOBERT
adaptive nutting
DytаBERT
50
10
30
6.75
7.00
7.25
Tera Floss
3.50
7.75
8.00
TeraFLOPs
Figure 10: Accuracy FLOPS plot for SHARCS applied DynaBERT 0.25 W (Hou et al., 2020) model. We set reduction factors to {2/3, 1.0)- By placing router different layers and changing the confidence thresh- olds, we can get different points in the plot. Note that the FLOPs each sub-network in each point is also reported.
Figure 8: Comparing entropy and confidence based hardness labels for training router.
Windw 3
Widow
15.0
Window
confidence thresholds [0.0, 0.8] for reduction fac- tor and [0.8, 1.0] for reduction factor 0.25. Fig- ure 9 shows the accuracy FLOPs trade offs that different checkpoints with different window sizes get throughout training the validation set. Ac- cording to the figure, the network can reach higher accuracy when trained with window size 3 or 5. We did not any significant improvements by using window size larger than
12.5
60.0
17.5
12.5
15
20
TeraTLCPe
Figure 9: Effect of changing history window size in SHARCS MNLI-m Accuracy FLOOPs trade-off.
classifier assigns class i. Figure 8 shows the ac- curacy FLOPs trade-off for both of these metries MNLI-m dataset with RoBERTa Tabase model. Given that the confidence score is a simpler approach and does not require any additional computation, we opted for using that in our method.
E.0.5
Changing History Window Size in Training the Router
As mentioned in ($2.1), having a larger window size helps stabilizing training the router as the hardness label for training sample might change throughout training. To illustrate this effect, train SHARCS on RoBERTabass model with three different window sizes (1,3,5) 5) for epochs on MNLI-m dataset. We place the router at layer 2, use two reduction factors (0.25, 1.0), and set the

---

82
FLOPH 114.1
FL009 1SHAIST
MNL INL.m
78
Der sisptiee muiles
13.0
13.5
14.0
14.5
15.0
15.5
16.0
16.5
Figure 11: Accuracy FLOPs plot SHARCS applied (Sah al., 2019) model. We set the reduction factors to (2/3, 1.0} or {0.5, 1.0}. By placing router different layers and changing the confidence thresholds, can get different points in the plot. Note that the FLOPss for each sub-network in each point is also reported.

---

SHARCS
RTJ
PABEE
DeeBERT
FasiBERT
BERxiT
Roberta base
MNLI-m
CNLI
QQP
87.0
02.8
90.0
90.0
80.0
85.0
Aocurac3
70.0
Accuracy
Accurace
00.0
63.0
16.0
80.0
50.0
70.0
40 8.00
6.00
10.00
16.00
20.00
25.00
30,0033.56 56
65.0
6.0
10.0
15.0
25.0
24.5
72
0.0
20.0
40.0
60.0
80.0
105.5
Tera Flops
Tera Flops
Tera Flops
SST-2
MRPC
CoLA
63.6
90.2
90.0
Metthews Correaalion
00.0
85.0
Aocuraiy
Accuracy
62.0
00.0
80.0
30.0
73.0
70.0
0.00
0.25
0.50
0.76
1.00
1.25
1.50
1.87
00
0.66
0.00
0.76
1.25
1.50
1.701.00
0820
0.30
0.40
0.50
0.50
0.70
0.80
0.00
1.03
Tera Flops
Tera
Flops
Tera
Flops
WNLI
78.7
58.00
58.33
Accuracy
70.0
Accuracy
56.00
30.00
00.0
48.00
00.0
46000
0.20
Flops
Flops
Figure 12: GLUE benchmark results. Best viewed in color.


Reviews:
[{"review_id": "TnUfAh7sWT", "reviewer": "Reviewer_sLft", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their time and valuable feedback. Please read our comment on positioning our paper in the literature which is available [here](https://openreview.net/forum?id=M1GRz46Ahz&noteId=GlysqKUbZV).\n\nWe address the concerns and questions of the reviewer as follows:\n\n**A. The paper is motivated by the expectation that dynamic subnetwork selection leads to better efficiency. However, the paper does not compare the proposed model with static compression methods comprehensively, such as structured pruning, leaving the motivation unconvincing.**  \n\nWe acknowledge that structured pruning is an effective method for improving inference efficiency of models. Nonetheless, it is *static* and the main drawback of static compression methods is that for each compute constraint a new model should be trained. \nDynamic approaches, including ours, enable us to attain a broader span of efficiency-accuracy trade-offs. These approaches have the capability to restore full accuracy by directing samples towards the biggest sub-network or to maximize efficiency by routing them to the most efficient sub-network dynamically based on the deployment constraints. Furthermore, in this work as a short paper, we maintained a focused exploration within the particular category of dynamic and sample adaptive inference methods.\n\n\n**B. Moreover, existing baselines for comparison need to be updated including [2].**\n\nWe are introducing SHARCS as a new class of adaptive inference methods which uses **width reduction** and a **novel sample hardness-based routing**. To demonstrate its potential, we conduct a comparison between SHARCS and various existing adaptive depth inference methods. It is important to recognize that our adaptive width approach can be used in conjunction with adaptive depth and token reduction methods. Please refer to the rebuttal (first table in the rebuttal available at this [link](https://openreview.net/forum?id=M1GRz46Ahz&noteId=kN68g9XLd3)) where we show that SHARCS can reduce the inference FLOPS of Transkimer by near 40% with the negligible accuracy drop of only ~0.6%.\nRegarding PCEE-BERT, unfortunately, the source code is not available and reproducing their numbers and comparing with them given the short span of rebuttal period does not seem feasible.\n\n**C. The reasons of design choices, such as hardness, adaptation methods, and pooler-unpooler structures, are unclear due to the lack of theoretical or empirical evidences.**\n\n**Hardness**: There are two main approaches in measuring hardness in the literature: Confidence-based hardness score [3] and entropy based hardness score [4]. We did an ablation on deciding between these metrics in appendix E.0.4 (line 710) and figure 8. The figure suggests that confidence based hardness gets better accuracies on MNLI dataset for FLOPs values smaller than 20 TeraFLOPs while on FLOPs values larger than 20 TeraFlops both of the metrics get comparable results.\n\n**Pooler-unpooler**: Pooler can be any module that reduces the dimensionality of the hidden state by the chosen reduction factor. Here are the main pooler modules one can consider:\n\n1. Taking the first $ r * d $ dimensions of the input where $r$ is the reduction factor and $d$ is model dimension (e.g. if \n$ r=0.25 $ \nand $d=768$, we take the first $ 768 * 0.25 = 192 $ dimensions of the hidden state as the input to the adaptive component). This is the approach we take in the paper.\n\n2. Using some other segment from the input. For example, the last dimensions or any random subset of dimensions in the input. We expect that this approach gets the same accuracy as the first approach. Previous related work in the literature observed that information is uniformly diffused in different segments of the hidden states whether they are drawn from the last, random, or initial dimensions [1, 2]. This is also intuitive as there is no inductive bias in the learning process that make different dimensions encode information differently and the model is not aware of the order of dimensions.\n\n3. Projecting the input into a vector with a smaller dimensionality using a linear layer. We opt this one out as it adds to the computational complexity which goes against our goal of maintaining computational efficiency within the model.\n\n4. One other option that we considered while designing the pooler was splitting the hidden state vector into sub-vectors with equal sizes and compute a weighted sum of them. For example, if the reduction factor is $0.25$ and model dimension is $768$, we chunk the hidden states into four $192$-d vectors $v_1$ to $v_4$ and take the sum $ w_1 . v_1 + … + w_4.v_4 $ where $ (w_1, w_2, w_3, w_4) $ is the weight parameter. This will only add four parameters to the method which is much more efficient than using a linear layer. The following table shows the results of comparison between SHARCS that uses first $ r * d$ dimensions and SHARCS that takes a weighted sum of chunks of hidden state. We average the results for threshold values from $0.1$ to $0.9$ for Roberta Base on QNLI while placing router after the third layer\n| Method | QNLI Avg. Acc. |  TeraFLOPs|\n|----------|----------|----------|\n| SHARCS (weighted sum of chunks)    | $82.09$   | $10.41$  |\n| SHARCS (first dimensions)    | $82.45$   | $9.97$  |\n| Roberta base   | $92.8$   | $24.5$ |\n\nThe table suggests that SHARCS that uses the initial dimensions outperforms SHARCS that takes the weighted sum of the chunks in terms of accuracy FLOPs trade-off.\n\nNote that the unpooler is the linear layer before the classifier layer. In fact, it is common to have one layer which operates on CLS token of BERT based models before the classifier layer [5]. We reduce the input dimension of this pre-existing layer according to the chosen reduction factor and do not change the output dimension (i.e. the output dimension remains 768 which is the dimensionality of base models in the BERT family). \n\n[1] Kusupati, A., Bhatt, G., Rege, A., Wallingford, M., Sinha, A., Ramanujan, V., Howard-Snyder, W., Chen, K., Kakade, S., Jain, P., & others (2022). Matryoshka Representation Learning. In Advances in Neural Information Processing Systems.\n\n[2] Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, & Nathan Srebro. (2022). The Implicit Bias of Gradient Descent on Separable Data.\n\n[3] Roy Schwartz, Gabriel Stanovsky, Swabha Swayamdipta, Jesse Dodge, & Noah A. Smith. (2020). The Right Tool for the Job: Matching Model and Instance Complexities.\n\n[4] Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, & Jimmy Lin. (2020). DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference.\n\n[5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, & Kristina Toutanova (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. CoRR, abs/1810.04805.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposes a novel approach based on dynamic pruning of hidden dimensions to reduce the model size. Experimental results show improvements in efficiency compared with some compressed models.", "reasons_to_reject": "A. The paper is motivated by the expectation that dynamic subnetwork selection leads to better efficiency. However, the paper does not compare the proposed model with static compression methods comprehensively, such as structured pruning [1], leaving the motivation unconvincing.\n\nB. Moreover, existing baselines for comparison need to be updated including [2].\n\nC. The reasons of design choices, such as hardness, adaptation methods, and pooler-unpooler structures, are unclear due to the lack of theoretical or empirical evidences.\n\n[1] Xia et al., Structured Pruning Learns Compact and Accurate Models, ACL 2022.\n\n[2] Zhang et al., PCEE-BERT: Accelerating BERT Inference via Patient and Confident Early Exiting, NAACL Finding 2022.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addresses the problem of a high computational cost of pre-trained language models. To reduce the cost, the authors propose a dynamic width transformer that routes subnetworks with different lengths based on the hardness of inputs. The experimental results in the paper show that the proposed method shows higher accuracy with commensurate cost compared with various compression methods.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5sChIEiMr7", "reviewer": "Reviewer_gQht", "summary": "", "questions": "Q1) In Table 2, why SHARCS (layer-2) shows better speedup with better accuracy than SHARCS (layer-4)?\n\nQ2) The labels for the router training are based on the confidence of the training models for the  training samples. However, as the training proceeds, the trained models tend to reveal over-confidence and low calibration, which can assign the most examples into the smallest subnetwork . How did the authors deal with such over-confidence? Moreover, it is great to see the real-examples about which samples are assigned to which groups.\n\nQ3) Which factors makes the proposed method special or achieve better performance compared to existing methods?Can you explain the rationale behind the proposed method?\n\nQ4) Why does the SHARCS use the leftmost dimensions for the reduction? How sensitive is the model when selecting rightmost or any directions?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their time and valuable feedback. Please read our comment on positioning our paper in the literature which is available [here](https://openreview.net/forum?id=M1GRz46Ahz&noteId=GlysqKUbZV).\n\nWe address the concerns and questions of the reviewer as follows:\n\n\n**(i) The outperformed results (in terms of speedup) can be seen only when the accuracy is significantly decreased (trade-off plots in Figure 2).**\n\nWe disagree that the SHARCS only outperform when the accuracy is significantly decreased. For example in MNLI-m and QNLI plots in figure 12, our method outperforms other methods even in the range of 0-5% accuracy drop of the base model. Furthermore, while we understand that the high compute regime is important, the performance should drop gracefully in the low-flops regime. This is especially important for low-resource scenarios wherein the compute constraints are harsh and the model needs to serve in a restricted setting. \n\n**(ii) As far as I understand, the proposed method is not the early-exiting approach. However, the paper mainly compares SHARCS with early-exiting methods. In the field of inference efficiency, there are a number of active studies that reduce the width (in this case, the number of tokens processed within the transformer), please refer to [1-3]. Moreover, the paper compare the outdated static compression method (distillbert, proposed in 2019).**\n\nIt is important to note that we are introducing a new class of efficient models which leverages **adaptive width** and a **novel sample hardness-based router**. We do not alter the sequence length. This makes our approach complementary to existing approaches such as token reduction methods. To demonstrate this, we combined SHARCS with the Transkimer [4].\n\nTo get the results we used Transkimer’s code and trained a model with skim coefficient 0.3. We then start from this model as initial checkpoint and train for 10 epochs using both our loss and Transkimer loss. We used two reduction factors 0.25 and 1.0. The table below shows that using SHARCS with Transkimer can reduce the FLOPS by near 40% while the accuracy drops only ~0.6%.\n\n| Model | QNLI Acc. | TeraFLOPs |\n|----------|----------|----------|\n| Roberta Base    | $92.8$   | $24.5$   |\n| Transkimer    | $89.45$   | $14.08$   |\n| **Transkimer + SHARCS**   | $88.83$ **(-0.62%)**   |  **$8.56$ (40% FLOPs reduction)**  |\n\n\n**(iii) The last thing is the lack of 'why' part. Why does the proposed method reveal better speedup than existing early exit methods?**\n\nWe believe that there are a few important reasons why our approach outperforms the previous methods:\n1. SHARCS router is effective in learning the hardness of input samples. This allows input samples to be processed with appropriate amounts of compute which in turn results in inference efficiency. The ablation at section 4.4 line 257 in the paper shows that replacing SHARCS router with a router that is used in an early exiting methods result in worse accuracy FLOPs trade-off. \n2. The accuracy drops are more steep in adaptive depth methods compared to adaptive width. This is mainly due to the fact that skipping an entire layer can significantly hurt the performance. In contrast, our method does not fully skip deeper layers; rather it substantially improves their efficiency by decreasing input and output dimensions of all projection matrices. For example, the first projection weight of the FFN component in each layer of BERT has dimensions of $768 * 3072$. In a $0.25$-width sub-network in SHARCS, these dimensions will be decreased to $ 192 * 768 $. Therefore the projection computation requires $16$ times less FLOPs. This also holds true for other components in the transformer layer as they are all essentially projection layers.  \n3. While SHARCS is effective in reducing the FLOPs by a large factor, table 2 of the paper shows that it can also result in a significant speed up. In fact, the substantial reduction in the FLOPs translates to better latency – although not 16 times faster, as FLOPs and latency are not linearly correlated.\n\n**Q1) In Table 2, why SHARCS (layer-2) shows better speedup with better accuracy than SHARCS (layer-4)?**\n\nThis is an oversight and the labels should be flipped. Thanks for pointing this out.\n\n**Q2) The labels for the router training are based on the confidence of the training models for the training samples. However, as the training proceeds, the trained models tend to reveal over-confidence and low calibration, which can assign the most examples into the smallest subnetwork . How did the authors deal with such over-confidence? Moreover, it is great to see the real-examples about which samples are assigned to which groups.**\n\nWe believe that If the networks are trained for too many epochs then this problem can happen. However by setting the hyperparameters (e.g. learning rate and its scheduler and number of epochs) and thresholds correctly, we observed that this can be prevented. Let’s assume a scenario wherein window size is W and there are two sub-networks: 0.25 width and full width. Here is the training dynamics we observed in most of our experiments:\n1. During the first window epochs, only the full width network is trained and confidence scores are recorded.\n2. Before epoch W+1, a large percentage of the samples will be assigned to the smallest sub-network. At this point we start training both sub-networks. Many of the samples assigned to the smallest sub-network start bouncing back to the full width-network as they are too hard for this slimmest sub-network.\n3. After training for a reasonable number of epochs by using early stopping, in many of our experiments we observed that near 60% of the samples were routed to the smallest sub-network with a minimal drop in accuracy compared to the original model.\n\n**Q3) Which factors makes the proposed method special or achieve better performance compared to existing methods?Can you explain the rationale behind the proposed method?**\nPlease refer to our response to point **(iii)** and our general comment (available [here](https://openreview.net/forum?id=M1GRz46Ahz&noteId=GlysqKUbZV)) which describes our rationale for our adaptive width method and sample hardness-based router.\n\n\n**Q4) Why does the SHARCS use the leftmost dimensions for the reduction? How sensitive is the model when selecting rightmost or any directions?**\n\nPooler can be any module that reduces the dimensionality of the hidden state by the chosen reduction factor. Here are the main pooler modules one can consider:\n\n1. Taking the first $ r * d $ dimensions of the input where $r$ is the reduction factor and $d$ is model dimension (e.g. if \n$ r=0.25 $ \nand $d=768$, we take the first $ 768 * 0.25 = 192 $ dimensions of the hidden state as the input to the adaptive component). This is the approach we take in the paper.\n\n2. Using some other segment from the input. For example, the last dimensions or any random subset of dimensions in the input. We expect that this approach gets the same accuracy as the first approach. Previous related work in the literature observed that information is uniformly diffused in different segments of the hidden states whether they are drawn from the last, random, or initial dimensions [1, 2]. This is also intuitive as there is no inductive bias in the learning process that make different dimensions encode information differently and the model is not aware of the order of dimensions.\n\n3. Projecting the input into a vector with a smaller dimensionality using a linear layer. We opt this one out as it adds to the computational complexity which goes against our goal of maintaining computational efficiency within the model.\n\n4. One other option that we considered while designing the pooler was splitting the hidden state vector into sub-vectors with equal sizes and compute a weighted sum of them. For example, if the reduction factor is $0.25$ and model dimension is $768$, we chunk the hidden states into four $192$-d vectors $v_1$ to $v_4$ and take the sum $ w_1 . v_1 + … + w_4.v_4 $ where $ (w_1, w_2, w_3, w_4) $ is the weight parameter. This will only add four parameters to the method which is much more efficient than using a linear layer. The following table shows the results of comparison between SHARCS that uses first $ r * d$ dimensions and SHARCS that takes a weighted sum of chunks of hidden state. We average the results for threshold values from $0.1$ to $0.9$ for Roberta Base on QNLI while placing router after the third layer\n| Method | QNLI Avg. Acc. |  TeraFLOPs|\n|----------|----------|----------|\n| SHARCS (weighted sum of chunks)    | $82.09$   | $10.41$  |\n| SHARCS (first dimensions)    | $82.45$   | $9.97$  |\n| Roberta base   | $92.8$   | $24.5$ |\n\nThe table suggests that SHARCS that uses the initial dimensions outperforms SHARCS that takes the weighted sum of the chunks in terms of accuracy FLOPs trade-off.\n\nNote that the unpooler is the linear layer before the classifier layer. In fact, it is common to have one layer which operates on CLS token of BERT based models before the classifier layer [3]. We reduce the input dimension of this pre-existing layer according to the chosen reduction factor and do not change the output dimension (i.e. the output dimension remains 768 which is the dimensionality of base models in the BERT family). \n\n[1] Kusupati, A., Bhatt, G., Rege, A., Wallingford, M., Sinha, A., Ramanujan, V., Howard-Snyder, W., Chen, K., Kakade, S., Jain, P., & others (2022). Matryoshka Representation Learning. In Advances in Neural Information Processing Systems.\n\n[2] Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, & Nathan Srebro. (2022). The Implicit Bias of Gradient Descent on Separable Data.\n\n[3] Jacob Devlin, Ming-Wei Chang, Kenton Lee, & Kristina Toutanova (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.\n\n[4] Guan, Y., Li, Z., Leng, J., Lin, Z., & Guo, M. (2022). Transkimmer: Transformer Learns to Layer-wise Skim.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strengths of this paper are as follows:\n\n(i) The proposed method is a new and has quite different perspective compared to existing early exiting methods, which can facilitate future research in NLP community.\n\n(ii) Outperformed results on various NLP tasks and backbones.", "reasons_to_reject": "The weaknesses of this paper are as follows:\n\n(i) The outperformed results (in terms of speedup) can be seen only when the accuracy is significantly decreased (trade-off plots in Figure 2).\n\n(ii) As far as I understand, the proposed method is not the early-exiting approach. However, the paper mainly compares SHARCS with early-exiting methods. In the field of inference efficiency, there are a number of active studies that reduce the width (in this case, the number of tokens processed within the transformer), please refer to [1-3]. Moreover, the paper compare the outdated static compression method (distillbert, proposed in 2019).\n\n(iii) The last thing is the lack of 'why' part. Why does the proposed method reveal better speedup than existing early exit methods? \n\n[1] PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination, ICML'20\n\n[2] AdapLeR: Speeding up Inference by Adaptive Length Reduction, ACL'22\n\n[3] Transkimmer: Transformer Learns to Layer-wise Skim, ACL'22", "questions_to_authors": "Q1) In Table 2, why SHARCS (layer-2) shows better speedup with better accuracy than SHARCS (layer-4)?\n\nQ2) The labels for the router training are based on the confidence of the training models for the  training samples. However, as the training proceeds, the trained models tend to reveal over-confidence and low calibration, which can assign the most examples into the smallest subnetwork . How did the authors deal with such over-confidence? Moreover, it is great to see the real-examples about which samples are assigned to which groups.\n\nQ3) Which factors makes the proposed method special or achieve better performance compared to existing methods?Can you explain the rationale behind the proposed method?\n\nQ4) Why does the SHARCS use the leftmost dimensions for the reduction? How sensitive is the model when selecting rightmost or any directions?", "paper_topic_and_main_contributions": "The paper considers inference inefficiency in current pre-trained transformer models. To this end, the authors propose SHARCS, a width-adaptive models that first identifies the level of difficulty of the samples and then assigns them into different layers with varying widths. To realize such mechanism, the paper introduces a trainable router that predicts the level of difficulty based on the moving average of the confidence. Evaluation results show that the proposed mechanism show better speed-up compared to existing method while achieving comparable accuracy. \n\nIn the reviewer's viewpoint, the contributions of the paper are: (i) a new mechanism for efficient inference method (ii) better performance on wide-range of NLP tasks. \n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Please refer to the relation to the existing width-wise compression methods (token removing approach in Reasons to Reject [1-3])"}, "extra_text": {}}, {"review_id": "bMT4NH8fmy", "reviewer": "Reviewer_cRFX", "summary": "", "questions": "Q1: As shown in Eq. (1) and Line156, the router only is trained by the binary cross-entropy loss, which does not involve information about the degree of hardness of samples. So, why does it have discriminative ability (i.e. the router can assign the reduction factor based on the hardness level of a sample) during inference?\n\nQ2: When making inference, the hardness of the samples within the same batch may be quite different, and therefore the reduction factor assigned should be diverse. In this case, how to perform the parallel computation?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their time and valuable feedback. Please read our comment on positioning our paper in the literature which is available [here](https://openreview.net/forum?id=M1GRz46Ahz&noteId=GlysqKUbZV).\n\nWe address the concerns and questions of the reviewer as follows:\n\n**Poor writing quality: although this paper is easy to understand, a thorough proofreading of the paper is still required (e.g., expression).**\n\nWe are glad the reviewer found the paper easy to follow and understand. We shall proof-read the paper further to improve the writing; however, we would be grateful if the reviewer could point us to the specific parts that require proof-reading so that we can improve it.\n\n**Lack of novelty: the paper presents a method that is not sufficiently novel or innovative, and the paper fails to provide a clear and compelling rationale for why the proposed method is necessary and how it differs from existing methods.**\n\nWe believe that there are a few important reasons why our approach outperforms the previous methods:\n1. SHARCS router is effective in learning the hardness of input samples. This allows samples to be processed with appropriate amounts of compute which in turn results in inference efficiency. The ablation at section 4.4 line 257 in the paper shows that replacing SHARCS router with a router that is used in an early exiting methods result in worse accuracy FLOPs trade-off. \n2. The accuracy drops are more steep in adaptive depth methods compared to adaptive width. This is mainly due to the fact that skipping an entire layer can significantly hurt the performance. In contrast, our method does not fully skip deeper layers; rather it substantially improves their efficiency by decreasing input and output dimensions of all projection matrices. For example, the first projection weight of the FFN component in each layer of BERT has dimensions of \n$ 768 * 3072 $. \nIn a $0.25$-width sub-network in SHARCS, these dimensions will be decreased to $ 192 * 768 $. Therefore the projection computation requires 16 times less FLOPs. This also holds true for other components in the transformer layer as they are all essentially projection layers.\n3. While SHARCS is effective in reducing the FLOPs by a large factor, table 2 of the paper shows that it can also result in a significant speed up. In fact, the substantial reduction in the FLOPs translates to better latency – although not 16 times faster, as FLOPs and latency are not linearly correlated.\n\n**Insufficient experimental comparison: (i) the comparison results on the test set are more convincing than those on the validation set; (ii) ablation experiments are necessary on the number of non-adaptive layers (i.e. K).**\n\n(i) As the labels for GLUE test set are not publicly accessible, following  most existing work that evaluate on GLUE dataset, we do not report any numbers on the test set and use the validation set as our test set. \n\n(ii) Regarding ablations, we have done ablations on number of sub-networks (appendix E.0.2 line 682), confidence thresholds (appendix E.0.3 line 694), using confidence or entropy as the measure of hardness (appendix E.0.4 line 710) and the window size during training (appendix E.0.5 line 721). We also did an extra ablation on $k$ on MNLI dataset for the rebuttal (i.e. number of non-adaptive layers) and the following table shows the results:\n\n| $k$ (# of non-adaptive layers) | MNLI-m Acc. | TeraFLOPs |\n|----------|----------|----------|\n| $1$    | $72.35$   | $6.35$ |\n| $3$   |  $79.95$  | $16.31$   |\n| $5$   |  $83.67$  | $18.71$  |\n| $7$   |  $86.51$  | $21.67$  |\n| $9$   |  $86.59$  | $26.40$   |\n\nThe table shows that having more non-adaptive layers lead to better accuracy yet at the cost of higher FLOPs.\n\n**Q1: As shown in Eq. (1) and Line156, the router only is trained by the binary cross-entropy loss, which does not involve information about the degree of hardness of samples. So, why does it have discriminative ability (i.e. the router can assign the reduction factor based on the hardness level of a sample) during inference?**\n\nTo make the sub-networks robust to router’s noise, we allow the hardness labels to be multi-hot labels which makes training the router a multi-label classification problem. During inference the reduction factor that has been assigned the highest logit by the router is selected and will be applied to the adaptive part. This will induce the necessary discriminative ability.\n\n**Q2: When making inference, the hardness of the samples within the same batch may be quite different, and therefore the reduction factor assigned should be diverse. In this case, how to perform the parallel computation?**\n\nIt is important to note that in many user-centric applications (e.g. models on edge devices or streaming settings) batch size of 1 is often used. In case of models hosted on cloud, if the batch size is large enough (e.g. 256 or larger), there shall be enough examples for each reduction factor to run adaptive layers for them in parallel through batching.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well organized and easy to follow.\n- The paper presents the approach that addresses an important problem (i.e. efficient inference) in application.\n- The paper conducts extensive experiments to demonstrate the effectiveness of the proposed method.", "reasons_to_reject": "- Poor writing quality: although this paper is easy to understand, a thorough proofreading of the paper is still required (e.g., expression).\n\n- Lack of novelty: the paper presents a method that is not sufficiently novel or innovative, and the paper fails to provide a clear and compelling rationale for why the proposed method is necessary and how it differs from existing methods.\n\n- Insufficient experimental comparison: (i) the comparison results on the test set are more convincing than those on the validation set; (ii) ablation experiments are necessary on the number of non-adaptive layers (i.e. K).", "questions_to_authors": "Q1: As shown in Eq. (1) and Line156, the router only is trained by the binary cross-entropy loss, which does not involve information about the degree of hardness of samples. So, why does it have discriminative ability (i.e. the router can assign the reduction factor based on the hardness level of a sample) during inference?\n\nQ2: When making inference, the hardness of the samples within the same batch may be quite different, and therefore the reduction factor assigned should be diverse. In this case, how to perform the parallel computation?", "paper_topic_and_main_contributions": "This paper proposes a method, SHARCS, to accelerate the inference of Transformer-based models by introducing a router. To train the router, authors first design a heuristical method to estimate the hardness of samples, then finetune the sub-networks and router, simultaneously. The experiments show that SHARCS outperforms prior methods and improves the inference efficiency.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents SHARCS, a method to accelerate Transformer inference using a mechanism that adaptively adjusts the width of Transformer layers based on the perceived difficulty of the input samples.

There was a general concern raised by multiple reviewers that the approach was compared against only a subset of dynamic inference techniques (early-existing approaches) which was considered strange as SHARCS is not an early-exiting approach. The authors provided a general note in the rebuttal that they are focusing on just adaptive approaches (not static methods like pruning). However, the paper would have been better positioned if the initial comparison was against other (more recent) adaptive approaches. In the rebuttal, the authors provide additional results against one token reduction technique (Transkimer).

Another concern raised by the reviewers was a general explanation of why this approach achieves a good trade-off. In the rebuttal, the authors explain that the improvement is due to the introduction of routing based on sample hardness. Additional experiment settings (or ablation studies beyond 4.4) on this main contribution would make the key novelty more clear.
Decision: Accept-Findings