Adapter Pruning using Tropical Characterization
Authors: Rishabh Bhardwaj
Tushar Vaidya
Soujanya Poria
Venue: EMNLP 2023 Findings (2023)
Keywords: Pruning
Adapters
Parameter Efficient Transfer Learning
Domain Adaptation

Abstract: Adapters are widely popular parameter-efficient transfer learning approaches in natural language processing that insert trainable modules in between layers of a pre-trained language model. Apart from several heuristics, however, there has been a lack of studies analyzing the optimal number of adapter parameters needed for downstream applications. Thus, we propose an adapter pruning approach by studying the tropical characteristics of trainable modules. We cast it as an optimization problem that aims to prune parameters from the adapter layers without changing the orientation of underlying tropical hypersurfaces. Our experiments on five NLP datasets show that tropical geometry tends to identify more relevant parameters to prune when compared with the magnitude-based baseline, while a combined approach works best across the tasks.

Adapter Pruning using Tropical Characterization
Rishabh Bhardwaj Tushar Vaidya Soujanya Poria
Singapore University of Technology and Design, Singapore
Nanyang Technological University, Singapore
rishabh_bbaardwajj@yymaaal sutd. edu. sg, tushar. vaidyaentu. edu.
Abstract
In this work, we propose novel approach pruning adapter layers without any iterative fine- tuning of the model parameters on downstream tasks. Using tropical algebra, we study the (duals of) hypersurfaces generated by adapter modules in the high-dimensional space. As pruning objec- tive, we aim to minimize the magnitude of adapter weights while constraining the change in hypersur- face geometry be small.
Adapters are widely popular parameter- efficient transfer learning approaches in nat- ural language processing that insert trainable modules in between layers of pre-trained lan- guage model. Apart from several heuristics, however, there has been lack studies analyz- ing the optimal number of adapter parameters needed for downstream applications. In this paper, we propose an adapter pruning approach by studying the tropical characteristics of train- able modules. We cast it as an optimization problem that aims prune parameters from the adapter layers without changing the orientation underlying tropical hypersurfaces. Our ex- periments five NLP datasets show that trop- ical geometry tends to identify more relevant parameters to prune when compared with the magnitude-based baseline, while combined approach works best across the tasks.
Related works include adapters pruning using lottery ticket hypothesis (Wu et al., 2022: Frankle and Carbin, 2018) that performs itcrative pruning- a few gradient steps, prune, and reset the parame- ters to initial weights. Rücklé et al. (2020) drops adapter from lower transformer layers. While these works are interesting, we provide more concrete angle prune adapter layers-prune by preserving the hypersurface geometry. We extend an insightful analysis of tropical geometry of neural networks (Zhang et al., 2018; Alfarra et al., 2022) to adapters.
1 Introduction
With the increase in network sizes, we are observ- ing an ever-increasing space and computational demand for models needed to solve given task. To tackle this, model compression (Cheng et al., 2017) techniques are becoming continuously popu- lar which retain the most important learning from the full model while reducing the size of the net- work either by pruning distillation.
Background
Adapter Operations. We use the adapter setup proposed by Pfeiffer et al. (2020) that inserts small modules after FFN add and layer norm sub-layer.
- h fech-+Jhhee S(hWa) W.
(1)
consists of down-projection Wa E Rdxr, up- projection W. E RR a ReLU activation func- tion f()), where typically < d.
Transfer learning approaches, such as adapters al-
(Houlsby al., 2019), are a parameter- ternative to full model fine-tuning which obviates the need to maintain a task-specific copy of the base language model (LM). Adapters insert simple modules in between layers of an LM to adapt the pre-trained representation for given downstream NLP task. However, there is a lack of research in pruning adapter modules to further enhance their parameter efficiency. We hypothesize that adapter weights can be pruned significantly by not compro- mising the performance observed with unpruned states, this motivates the proposed approach.
Tropical Arithmetic. Tropical algebra is vari- ant of classical algebra where basic arithmetic op- erations are redefined. The tropical sum @ of two numbers represents their maximum and the tropical product O represents a classical addition' Thus,
I + y = max {I, y}
x C y +
'The tropical addition can be defined as a t b = min(a, b) or max(a,bl b), we focus on the latter as we ana- lyze ReL.U-based adapter network.

---

For e.g., 2 @ 5 5 and 2 о 5 7. Axioms and order of arithmetic operations in tropical algebra follow the classical, thus addition is commutative and multiplication is distributive over addition. We relegate detailed discussions about tropical algebra, polynomials, and hypersurfaces to the Appendix A.
Without the loss of generality, we assume down- projection bias-free?, d()) can be expressed in terms of generator matrices G of zonotopes ob- tained from A and B. To find sparse A, B, we introduce sparse regularization terms in the opti- mization function. Thus, finding adapter weights that preserve the hypersurface geometry can be cast as the following optimization problem:
Notations used: Henceforth, we denote Wa. Wa. h by A, B. and x, respectively; B := max (B,0): B == max[ - B,0); b, denotes 4sh row of B; b++:=max 0}, b' maa Diag(u] arranges in diagonal matrix; ||G||,,1=~% (((,) 11-11e
min A,
- 2
G
G.
||
+ 2 c G2 - Gz |
+ A1
G 111
+ X2 G2 1,1
(2)
denotes Frobenius Norm.
3 Tropical Adapter Pruning
where
G = Diag b² b111'1111 []A; G2 = Diag (b* JA,
G Diag b |A; G2 Diag(B (b' JA,
Given a frozen language model adapted to a spe- cific task using adapter layers, we divide our ap- proach into two steps: 1) Finding adapter weights P. that are crucial to preserving the tropical adapter hypersurface by solving simple optimiza- tion problem; 2) Pruning of adapter weights with least magnitudes that do not lie Pr. Next, we de- scribe step-1 which is core to the pruning method: A bottleneck adapter block can be expressed by f(x) B max Ax. 0). Since [x) in it- self is not tropical polynomial and thus does not form tropical surface, we rewrite in terms of the difference between two tropical polynomials f(x = H ((xx)-Q(x) following the analysis of tropical rational function by Alfarra et al. (2022).
We provide a derivation of the above function in Appendix B. It is important to note that in the prun- ing phase, we do not iteratively fine-tune adapter or LM parameters on the downstream task.
Algorithm 1: Tropical Adapter Pruning
Initialize: T. \1. d2:
Return: Á, B;
B < B B < B
for in I,.., T do
for in 1,..., r do
if is even then
Thus we focus on a relatively lenient problem i.e. identifying weights that preserve tropical hyper- surfaces defined by H (x) and Q(x). Let H(x) and @xx) be the respective hypersurfaces, one can choose sparse set of Å, B that belongs to the set of matrices obtained by solving the following optimization problem
G = Diag (b 1J;;
loss || G 2
loss |G
(= 0.5 loss1 + X1 loss2;
else
G = Diag =
loss = G G 2:
min A.B
d((x)) + d(O(x), ((x))
loss |/
l = 0.5 * loss1 + X2 los82;
end
Where d(-) defines the distance between two geo- metric objects: H and Q are hypersurfaces obtained by substituting A and B with A and B in (x). In place of preserving the orientation of Hxx) and (xx), we aim to preserve the orientation of their respective dual objects denoted by 8( (H (x)) and 8()(x)). Thus.
check convergence of combined loss
À + À - 17* a(A)
8 C;
+ B - 17 a(33
à l.
end
end
min ((8(H(()) ((((x)) + +=(((2(x)) A.B
WW merge hias term b with the down-projection matrix,
thus x + |x, 1] and A 4- [A; b..

---

Task-I (MELD)
1.4%
2.9%
5.7%
8.6%
11.5%
14.5%
17.4%
20.3%
23.3%
Standard
33.56
35.00
41.02"
39.65
43.72
49.38
50.18
55.05
57.06
Tropical
34.06"
37.37"
36.79
49.59"
52.46
52.28'
56.34
58.04"
58.09"
60.71
Combined
33.56
37.37
41.02
49.59
52.46
52.28
56.34
58.04
58.09
Task-2 (SNLT)
1.4%
2.9%
4.3%
11.5%
13.0%
14.5
26.0%
32.8%
39.4%
Standard
40.31
34.41
34.74
77.91*
79.96
82.06'
85.77
86.01
85.77
Tropical
41,08"
46.33"
46,94"
75.82
77.93
78.17
85,96
86.17
85,96"
86.45
Combined
41.08
46.33
46.94
77.91
79.96
82.06
85.96
86.01
85,96
Task-3 (RT)
1.4%
2.7%
4.2%
5.8%
7.3%
8.9%
11.1% 1%
12.1 1
15.0%
FM
Standard
77.49
74.67
69.23
82.65
83.86
86.49
83.02
85.18
87.80"
Tropical
79.17"
82.55
83.68
84.05
86.12"
87.05"
87.24"
88.37
87.71
87.99
Combined ned
75.17
82.35
83.68
84.05
86. 2
86.49
87.24
88.37
87.71
14
(IMDB)
1.4%
2.9%
5.7%
8.5%
11.5 %
14.4%
17.4%
20.3%
26.3%
FM
Standard
74.55
71.70
69.82
59.46
77.04
80.85
80.32
83.75
84.22
falpical
75.37"
79.14"
82,22'
75.89"
75.11
83.79*
83.78'
85.41
85.15"
87.61
Combined
74.55
79.14
82.22
75.89
77.04
83.79
83.78
85,41
85.15
Task-5 (TREC)
Method
1.4%
3.0%
5.0%
11.5%
14.5
16.1
25.9%
30.4%
44.5%
Standard
24.0
42.2
56,0
63.0
64,6
70.2
96.6"
96.8
97.4"
Tropical 30.8"
45.8"
64.6"
71.8"
75.8"
73.0"
96.4
96.4
97.2
97.2
Combined
30.8
45.8
64.6
71.8
75.8
73.0
96.6
96.8
97.4
Table 1: Percentage of retained parameters (100- - 9 vs Test Accuracy/F1, FM refers to the full model, i.e..
unpruned adapter states. Superscript *** refers to better performing setting out of Standard and Tropical.
Given an adapter module, Algorithml finds the minimizers Á and B by performing gradient descent-based updates² over two loss terms ex- pressed in terms of generators G and Gg. T, denote the maximum gradient steps and the num- ber of rows in A and columns in B. n) R is step size and λ1 >2 E R+ indicate the importance of pruning over the shift in generators. We em- ploy laser-wise pruning of the network without any iterative fine-tuning on downstream tasks. We find p% parameters with the smallest magnitude in {A. B} and (A, B) separately, denoted by Ps and Pr. We denote tropical adapter pruning by Tropical that prunes only those parameters in Pr which also present in the set Ps. The final percentage of pruned parameters decreases p%. We compare the approach with the baseline that prunes 5% of the smallest magnitude parameters from the layer. We denote this setting by Standard. Combined chooses one of Tropical or Standard whichever gives better results on the development set. We omit the comparison with Drop
method as even at 50% pruning. the method shows a significant drop in the performance. Standard inherently tests the validity of magnitude-basee pruning via lottery ticket hypothesis (Wu et al., 2022) but without iterative retraining of adapter parameters. We do not iteratively fine-tune adapter parameters the downstream task. The proposed method is agnostic downstream tasks, models, and the learning algorithm used to train it. Thus, the framework related to but not directly com- parable to model Lo sparsification (Louizos et al.. 2017) and low-rank compression (Idelbayev and Carreira-Perpinán, 2020).
4 Experiments
We set up a RoBERTa-base (Liu et al., 2019) with one adapter module inserted in each layer after add and layer norm sub-layer. We follow the adapter configuration from Pfciffer et al. (2020). For prun- ing analysis, consider three tasks-Emotion Recognition in Conversations (ERC), Natural Lan- guage Inference (NLI), and Text Classification (TC). For ERC, we use MELD, the task is to classify the emotion of an utterance given past utterances.
"Not to confuse with gradient descent used to learn model parameters, Here, it is used to solve the optimization prohlem in Equation (2).

---

1.4%
2.7%
5.8%
8.9%
12.0%
15.0%
S-CN
71.76
68.48
65.29
84.52
85.55
86.96
T-ON
76.27
79.55
78.42
80.11
86.49
87.24
S-CU
77.49
74.67
82.65
86.49
85.18
87.80
Loss
T-CU
79.17
82.55
84.05
87.05
88.37
87.71
S-CB
69.89
774.39"
58.82
76.I7
"85.74"
87.90
T-CB
73.73
50.00
66.79
84.05
86.87
86.68
200
Table 2: Accuracy scores on RT task. Comparing node- wise (CN), layer-wise (CU), and pruning all modules gether (CB). S and T denote Standard and Tropical. respectively.
10
15
20
25
Iteration
Figure Value of pruning function (loss) with itera- tions.
parameters (considering both weights and biases). As shown in Table in the Standard settings S-CN/ S-CU/ S-CB, we observe layer-wise S-CU pruning works best in four out of six different fractions of parameters retained. the Tropical pruning settings T-CN/ T-CU/ T-CB, layer-wise prun- ing T-CU performs best amongst all the considered pruning fractions. Moreover, T-CU works best un- der each pruning fraction category.
Keeping the current utterance first, we append the past seven utterances in reverse order (Bhardwaj al., 2022b). For NLI, use SNLI dataset (Bow- man al., 2015). We append the premise and hypothesis separated by the special token 
. For TC task, we use three datasets: IMDB (Maas et al., 2011), Rotten Tomatoes RT (Pang and Lee, 2005), and TREC (Li and Roth, 2002). Separately, we pre- train adapters on downstream tasks with batch size 32, LR of 0.001, and 1000 steps with evaluation at every 100 steps using development set. The eval- uation metric for ERC is macro Fl and accuracy for all the other tasks. We set pruning percentage p E (98%, 96%. .... 2%).. Table shows the test performance of networks with the percentage of adapter parameters retained, i.e., (100 - 9)%. this is represented in black-bold fonts. We observe that both Standard and Tropical can be effective in pruning more than 60% of the adapter parameters with small drop in performance with respect to the full module performance (FM). Moreover, we notice Tropical outperforms Standard in eight out of nine pruned model states on MELD, six out of nine on SNLI, eight out of nine pruned adapter states on RT and IMDB, and six out of nine states on Trec. Across the 45 combinations of tasks and ob-
Figure 1 shows the Objective function in Equa- tion (2) quickly converges the minimum. This observation corroborates the claim of convexity by (A (Alfarra al., 2022). The plot in Figure 2 shows the change in zonotope structure before and after optimization on SNLI. The black polytope is ob- tained from generators A, and the red polytope shows the polytope obtained after optimization. i.e., zonotope obtained from Å. B. We observe the optimization preserves the geometry of zonotopes while enforcing the rows of the down-projection matrices to be much sparse possible, i.e., many points in the zonotope come close zero, keeping necessary boundary points to preserve the geometry. These zonotopes are dual to adapter hy- persurfaces, thus preserving one structure enforces the other's orientation to remain preserved. Hence, one can prune adapters yet maintain their charac- teristic properties.
pruning fractions, except for two settings, serve tropical geometry- baase combined approach outperforms the other two, denoted in red font.
Conclusion
Next, we study tropical pruning in different scenarios- caaeriio--llass-biee class-bind, class-unin uniform, and node- wise (See et al., 2016). In class-blind (CB), all the parameters of adapters are considered for pruning p% of the smallest magnitude weights and biases. class-uniform (CU), we prune p% of the smallest magnitude parameters of each adapter layer sepa- rately. We also refer to it as layer-wise pruning. In node-wise pruning, we prune p% of the node-wise
We proposed novel approach for adapter prun- ing by studying their tropical characteristics. We formulated it as optimization problem that aims to identify row-sparse projection matrices while minimizing the distance between tropical hypersur- faces before and after pruning. We demonstrated the advantages tropical characterization on five NLP datasets reformulated as classification.

---

transport for solving NLP tasks. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4801-4820, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. arXiv preprint (.K3::/508.05326
-20
Zonotope
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. 2017. A survey of model compression and accel- eration for deep neural networks. arXiv preprint arXi::1710.0922..
Figure 2: Zonotope defined by adapters before (red) and after the pruning (blue) via Algorithm 1.
Narsingh Deo. 2017. Graph theory with applications to engineering and computer science. Courier Dover Publications.
6 Limitations
As our focus is on adapter-based architectures, the proposed approach can not be directly adapted to other parameter-efficient approaches such as soft prompt tuning (Lester et al., 2021; Bhardwaj et al., 2022a) which do not have explicit dense connec- tions and activation. Another limitation comes from ReLU activation function. Since it fits in min-max (Tropical) algebra, we could reformulate the problem in terms of tropical polynomials. How- ever, for other non-linear activation functions such as Tanh, one has to reformulate and likely resort to approximations as there is straightforward way to cast them in a tropical algebraic expression.
Jonathan Frankle and Michael Carbin. 2018. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: /803.0060.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo. Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In- ternational Conference on Machine Learning, pages 2790-2799, PMLR.
Yerlan Idelbayev and Miguel A Carreira- Perpinán. 2020. Low-rank compression of neural nets: Learning the rank of each layer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 8049-8059.
Acknowledgement
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.
The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 3045-3059, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
We thank the anonymous reviewers for their con- structive feedback. This project is supported by the AcRF MoE Tier-2 grant (Project no. T2M0E2008. and Grantor reference no. MOE-T2EP20220- 0017) titled: "CSK-NLP: Leveraging Common- sense Knowledge for NLP", and the SRG grant id: TISRIS19149 titled "An Affective Multimodal Dialogue System".
Xin Li and Dan Roth. 2002. Learning question clas- sifiers. In COLING 2002: The 19th International Conference on Computational Linguistics.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint 1907.1199
References
Motasem Alfarra, Adel Bibi, Hasan Hammoud, Mo- hamed Gaafar, and Bernard Ghanem. 2022. On the decision boundaries of neural networks: A tropical geometry perspective. IEEE Transactions Patlern Analysis and Machine Intelligence.
Christos Louizos, Max Welling, and Diederik P Kingma. 2017. Learning sparse neural networks through i_0 regularization. arXiv preprint 17120:111..01312.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham. Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Lin- guistics.
Rishabh Bhardwaj. Amrita Saha, and Steven CH Hoi. 2022a. Vector-quantized inputcoontettualzze soft prompts for natural language understanding. arXiv preprint arXiv:2205.0 1024.
Rishabh Bhardwaj. Tushar Vaidya, and Soujanya Poria. 2022b. KNOT: Knowledge distillation using optimal

---

Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the ACI.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2020. Adapterfusion: Non-destructive task composition for transfer learning. arXiv preprint arXiv:2005.0024:
Andreas Rückléé Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2020. Adapterdrop: On the effi- ciency of adapters in transformers. arXiv preprint WrXiv:2010.11918
Abigail See, Minh-Thang Luong, and Christopher Manning. 2016. Compression of neural machine translation models via pruning. arXiv preprint 1.Xi::1606.09274.
Jiarun Wu, Qingliang Chen, Zeguan Xiao, Yuliang Gu, and Mengsi Sun. 2022. Pruming adatperfusion with lottery ticket hypothesis. In Findings of the Associ- ation for Computational Linguistics: NAACI. 2022, pages 1632-1646, Seattle, United States. Association for Computational Linguistics.
Liwen Zhang, Gregory Naitzat, and Lek-Heng Lim. 2018. Tropical geometry of deep neural networks. In International Conference on Machine Learning. pages 1824-5833 PMLR.

---

A Tropical Algebra and Geometry
x
2
(0,2)
To motivate our approach we first provide back- ground on tropical algebra and geometry.
x1
Tropical Arithmetic. Tropical algebra is a vari- ant of classical algebra where basic arithmetic op- erations redefined. The tropical sum of two numbers represents their maximum and the tropical product @ represents classical addition*. Thus,
X2
(0.1)
(1.1)
X
10,0)
(1.0)
(2,0)
2
X1
y max {a.yl
Figure 3: Tropical curve F(p) (orange) and dual sub-
② y = x y
division of Newton polytope ō(p) (black) of f(x) 1@ a @
1 1 2 + @ 2,22 @ 2 21
eerpaa220 2 22 22
For instance, 1 5 = 5 and 2 3
Axioms and order of arithmetic operations in trop- ical algebra follows the classical, thus addition is commutative and multiplication is distributive over addition:
Tropical powers, monomials and polynomials are basic building blocks of the algorithm we pro- pose for adapter pruning.
x ① y
y x O y +
(commutative)
A.1 Tropical Hypersurfaces.
2)
(distributive)
Tropical hypersurfaces are analogues to classical algebraic surfaces and key objects for us study for adapter pruning. Given a tropical polynomial f(x) = c1x @... c, x",, its tropical hypersur- face is set of points where is attained by two or more constituting monomials, thus
From these properties, it can be inferred that -o is the additive identity as multiplicative identity @ x. Elements under the tropical arithmetic in the space of real numbers (with -o)) are said to form semiring T denoted by a triplet (R U (- O)
and O is
F(p) = {x E Rd Gx = c,xx,
Tropical Power and Monomial. For any vari- able C T, the tropical power can be defined as on a.r. where E N (a natural number). For simplicity of notations, will write x² in place of Sa tropical monomial expressed in the form
for some a, # az}.
Here we mention a few provable facts F divides the domain of p into convex regions (or cells). Poly- nomial p is non-linear at x if and only if x lies on F. Similar to algebraic polynomials, we can identify Newton polytopes associated to tropical polynomi- als.
xxx c0
12
where E RU{ - and a, N. For convenience, we will write tropical monomial by e x where x- -_1,,...,,. E Td and a=
Newton Polytopes. For a given polynomial f(x) = cixºI @ .... @ cxx"M, its newton poly- tope is defined by the convex hull of the exponents ag Nd. The points a and polytope lies in a d-dimensional plane (RRd). Thus
ax) E Nd.
Tropical Polynomial. A d-variable tropical poly- nomial fx) can be represented by a finite sum of tropical monomials
f(x) c1 E x2 ....@ x
(pp) = ConvHull(fa; e : )in
where the a, # aj when i # j. coefficients C
The tropical polynomial determines the dual subdivision 8(p) of newton polytope. The tropical hypersurface F(p) is dual graph to this 8()), i.e., vertices of F(p) are regions of 8(p) and edges rep- resent two adjacent regions in 8(p)3. Each vertex in 8(p) corresponds to one "cell" in I² where p
R U (-0),, 0g- (a:1, 0:2, .., aid) Nd and exponents a; are integers. Ignoring 0o for ease, is important to note that p has a mapping Rd + R. both x and a are d-dimensional vectors.
"The tropical addition can be defined as a @ b min(a, b) or max(a, b}, we focus on the latter as we ama- lyze a Rel.U-based adapter network.
Reader can read more about dual graphs in (Deo, 2017)

---

is linear. Since d(p) is in one-to-one correspon- dence with the tropical hypersurface, we study the adapter unnaaterrrssisss-udddiiiiiinn hypersurfaces F(p))-b studying the orientation of the primal graph (p)). To determine (()), we use the fact that when the model is bias-free, 8(p) = Ap) (Zhang et al., 2018; Alfarra al., 2022). Figure 3 provides an illustration of F() adn 6p)) for specific p.
Thus, f(x) [[() - Qx).. Let f denote the first output from adapter block, b' B :] (i.e. 5th row of B). We use H and @ to denote tropical hypersurfaces of H and Q at node
O
P
H =
x
x
x
j=1
Computing dual subdivision
Zonotopes. The zonotope formed by VI.... Vm E R is defined as (vv...... Vm) {E Av, 0 S X, 11.
8(HL) - (b'
(ba ConvHull(a),,, ,q
+ +113=100 [)]_
Minkowski sum. Given two sets P1 and P2 in Rd the Minkowski is defined
(P-1)
[/-1 ConvHull(a) 97 07.
p P = (vi V2: V, P3]
+ +[ (0) [5)_a)]] shift (P-2)
Property-1. The Minkowski sum of two poly- topes is the convex hull of their vertex sets. Let, ((P) be the vertex sets of polytope P. then
(b') ConvHull(a,, 0)) + shift
Similarly, we compute dual subdivision of q²
Pi P2 ConvHull (V(P))+((P))
6(2) (b- ConvHull(a,, 01) + shift
Under bias-free assumption,
Note that convex hull of aj and 0 is line seg- ment. Thus, 5(h') defines a Minkowski sum over line segments which is zonotope. Following Al- farra et al. (2022), and ognoting the shifts, one can straightaway obtain zonotope generators G1 and G2 for 8(H) and (()), respectibely.
Property-2. nomials, then
Let p1 and p2 be the tropical poly-
p(p ④ p2) = =(p1)
B Pruning Objective
B.1 Notations Used
We denote Wa Wu, h by A, B, and x, respec-
tively; B+ (aammax(BBBB 0); = max(-B.
by denotes 4zh row of B: bi+:= max 0).
bi max{-b', 0): Diag(u) [u] arranges u in di-
agonal matrix; || GG.:)) ||F
denotes Frobenius Norm.
B.2 Derivation of Pruning Objective
Let f(x) B max(Ax, 0), then
f(x) = ( + BB max A
x.
A x}
A
B* + max A + x. A x + A
B max A x,
A x
+ -
Thus define (xx) and Qxx)
H(x)= B max A x, x) B A
Q(x)= = B max A x. x + B B


Reviews:
[{"review_id": "QXMNwEMoBy", "reviewer": "Reviewer_2vRX", "summary": "", "questions": "The code repository has expired.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Clarification-1]: This study exclusively applies its method to RoBERTa, which results in a lack of generalizability validation for the proposed approach.**\n\nWe perform new experiments on DeBERTa [1] and AlBERT [2] to show that the proposed approach is generalizable across models. We observe the model to outperform the Standard baseline in 12 out of 21 settings and perform equally well in the rest of the settings.\n\n|Model                        |   1% |  3%      | 5% | 9% | 15% | 17% | 20% | FM|\n|------------------------|-------|----------|------|-----|-------|-------|-------|-----| \n|RoBERTa (Standard)| 33.56 | 35.00|41.02|39.65|49.38|50.18|55.05|60.71|\n|RoBERTa (Tropical)  | 33.56 | **37.37**|41.02|**49.59**|**52.28**|**56.34**|**58.04**|60.71|\n|DeBERTa (Standard)|34.37|40.06|40.99|45.62|48.93|49.64|62.82|62.44|\n|DeBERTa (Tropical). |34.37|40.06|**42.78**|**47.17**|**51.18**|**61.69**|62.82|62.44|\n|ALBERT   (Standard) |19.58|25.07|34.75|42.41|44.21|45.78|45.78|45.78|45.78|\n|ALBERT   (Tropical)   |19.58|25.07|**40.66**|**45.78**|**45.78**|45.78|45.78|45.78|45.78|\n\n_Table 1: Results of preserving only x% of adapter parameters trained on MELD. We approximate values to the nearest percentage of pruning as opposed to Table 1 in the paper showing the exact percentage of pruned parameters. Fraction of pruned parameters = (1-x)%._\n\nWhile we believe the proposed approach is strong enough to be generalizable across models and tasks, if the reviewers feel these experiments should be included in the main draft, we would do so. To prove generalizability further, we include experiments with different bottleneck dimensions of adapters in Tables 1 and 2 of reviewer-rtfk. Additionally, to make the claim even stronger, we add another baseline as random pruning in Table-3 of reviewer-rtfk.\n\n=======================\n\n**[Clarification-2]: Normally, pruning is employed to enhance model training or inference efficiency or to improve model generalization. Where does adapter pruning help?**\n\nThanks for asking! Our method contributes to the second factor i.e. **inference efficiency**. Inference efficiency carries two aspects, one can contribute to any of the following or both:\n\n1- Reduced computational resources used by the model for inference.\n\n2- Increased speed of inference.\n\n_Notably, we contribute in aspect 1 i.e. **reduction of computational resources utilized by the model**_. This is particularly important for deploying models on resource-constrained devices or systems. We further elaborate on resource efficiency in Table 2.\n\n=======================\n\n**[Clarification-3]: Timing before and after pruning.**\n\nAs elaborated in clarification-1, we don't contribute to the speed of inference rather we contribute to the computational resources required by the model which is proportional to the number of network parameters pruned. Since adapter pruning does not change the depth of the computational graph, it makes timing analysis less interesting since it stays the same as the standard model. Instead of timing analysis, we show the percentage reduction of the computational resources required by adapters as compared to the magnitude-based pruning:\n\n| Task | % Reduction  (Computational Resources) |\n|------|--------------------|\n| MELD | 5.9                |\n| SNLI | 13.4               |\n| RT   | 6.1                |\n| IMDB | 8.9                |\n| TREC | 14.1               |\n\n_Table2: Percentage reduction in computational resources required by the model to achieve the same pruning performance as magnitude-based pruning._\n\nIf the reviewer feels that the above analysis should be a part of the main draft, we would be happy to do so.\n\n=======================\n\n**[Clarification-4]: Discussion on hyperparameter lambda.**\n\nThe $\\lambda$ terms denote the importance of regularizers to control the sparsity of the network in the number of line segments in zonotopes. Essentially, too big $\\lambda$ values can lead to shift or distortion in hypersurfaces leading to noisy pruning. Too small $\\lambda$ values will make the problem equivalent to finding another set of adapter weights but without any insights into which parameters to prune. Thus, $\\lambda$ balances set the trade-off between the importance of hypersurface geometry to be preserved and sparsity in the network. Thanks for the suggestion, we will elaborate on it in the paper.\n\n=======================\n\n**[Clarification-5]: Code repository.**\n\nThanks for pointing it out! We have made it live again.\n\n=======================\n\nReferences:\n- [1] He, Pengcheng, et al. \"Deberta: Decoding-enhanced bert with disentangled attention.\" arXiv preprint arXiv:2006.03654 (2020).\n- [2] Lan, Zhenzhong, et al. \"Albert: A lite bert for self-supervised learning of language representations.\" arXiv preprint arXiv:1909.11942 (2019).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The experimental results suggest that the proposed approach could lead to enhanced performance for Pre-trained Language Model (PLM) in specific tasks.\n- This study assesses the performance of PLM in relation to varying percentages of retained adapter parameters.", "reasons_to_reject": "- The new experimental results reveal that the Computational Resources have not achieved a substantial reduction.\n- The experiments inadequately discuss crucial hyperparameters, such as lambda.", "questions_to_authors": "The code repository has expired.", "paper_topic_and_main_contributions": "This study presents an adapter pruning approach through an examination of the tropical characteristics of trainable modules.However, I find myself somewhat puzzled by the author's reasoning behind the pruning analysis. Normally, pruning is employed to enhance model training or inference efficiency, or to improve model generalization. Nevertheless, the article does not delve into the impact of pruning on model efficiency, and the proposed method did not surpass the full model's performance in most tasks.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "26ulDnmwoe", "reviewer": "Reviewer_28EZ", "summary": "", "questions": "1. Why standard deviation is missing in Table 1?\n2. Could you explain Fig 2 better?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Clarification-1]: Motivation.**\n\n- (Problem formulation): Find parameters that are least important to the network, such that removing them does not lead to significant performance degradation on the task.\n\n- (Why classical pruning is insufficient): Classical magnitude-based pruning is prone to errors as low magnitude weights can still have a huge impact on the adapter's output. Thus pruning them from the network can potentially lead to a significant drop in the model's performance. We posit that the magnitude of parameters is not sufficient to convey their importance to the network.\n\n- (Hypersurfaces to rescue): The hypersurfaces tell us about the characteristics of the function defined by the network, here the network is adapter modules. Thus, preserving the adapter's hypersurface is equivalent to preserving parameters that are important to the function, irrespective of their magnitudes. This analysis would provide extra insights into parameter importance to the network.\n\n- (How to find hypersurfaces): Because of the nonlinearity of the function, it is challenging to find exact hypersurfaces formed by the ReLU-based adapter. Thus, we adopt the zonotopes, a dual analysis that helps us identify important weights with more ease. During magnitude-based pruning, we find that preserving weights that are important for zonotope geometry retained more performance from the model than alone magnitude-based pruning of the same fraction of parameters disregarding their importance to hypersurfaces.\n\nAs per the reviewer's suggestion, we plan to streamline the discussion about the paper's motivation and proposed approach in the provided extra space after acceptance.\n\n==\n\n**[Clarification-2]: Undefined terms.**\n\nWhile we tried to cover all the terms in the paper, if the reviewer feels the following table can enhance the readability, we are willing to include it in the paper.\n\nNotation|Definition|\n|-|-|\n|$T, r$|the maximum gradient steps and the number of rows in A and columns in B (line 150)|\n|$\\eta$|step size (line 152)|\n|$\\lambda_1$ and $\\lambda_2$|the importance of pruning over the shift in generators (line 153)|\n|$G_1, \\hat{G}_1, G_2, \\hat{G}_2$|Generator matrices (line 140-141)|\n|$A, B, B^+, B^-, b^{i+}, b^{i-}, \\|\\|G\\|\\|_{1,1}$ | Specific paragraph covering common notations (line 087)|\n$\\hat{A}, \\hat{B}, \\hat{b}^{i+}$| Defined in algorithm 1\n\n_Table1: Notations used in Algorithm 1_\n\n==\n\n**[Clarification-3]: Why standard deviation is missing in Table 1?**\n\nReason: There is no randomness in the process of pruning. Randomness can come from two aspects of the proposed approach:\n- Model training: Since we focus on off-the-shelf adapter-based models, we don't analyze randomness in their training.\n- Pruning: Algorithm-1 is a convex optimization problem, it has exact solutions while obtaining a new set of adapter weights. Thus, there is no randomness in the optimizing algorithm, it always converges to the global minima.\n\nHowever, to obtain the off-the-shelf models for pruning, we performed in-house training. We found the performance varies from 1-1.5\\% across the adapter-based models and datasets. While the analysis would be less interesting to the paper as training a task-specific model is not the primary contribution, if the reviewer feels, we would be happy to include randomness in obtaining the unpruned (off-the-shelf) model.\n\n==\n\n**[Clarification-4]: Could you explain Fig 2 better?**\n\n(Simplified explanation of how preserving zonotopes structure tends to reflect more effective pruning):\n\nHypersurfaces tell the characteristics of the function defined by the adapters. Since directly analyzing hypersurfaces is a complex task due to highly non-linear operations, we analyze the duals of hypersurfaces using tropical algebra. Broadly, zonotopes define the dual to the hypersurfaces. Preserving zonotope structures during pruning tends to preserve the structure of the (primal) hypersurfaces, which in turn preserves the adapter properties. The plot in Figure 2 shows the change in zonotope structure before and after optimization on SNLI. The black polytope shows zonotopes before pruning and the red polytope shows the polytope obtained after pruning (Algorithm-1). These polytopes are obtained by the convex hull of the points of the corresponding color. Each point represents a row in the projection matrices A and B. We show the optimization preserves the geometry of zonotopes while decreasing the magnitude of parameters that are least important to the structure. The parameters that are not part of defining the polytope come close to zero i.e. tend to be pruned more.\n\nIf the reviewer feels the discussion should be extended, we would do so in the paper.\n\n==\n\n**[Clarification-5]: For the experiments, authors only compared their method with \"standard\" - are there more relevant baselines? If no - that's ok, but needs to be discussed.**\n\nThanks for asking this. There are several related works such as L0 sparsification [1], low-rank compression [2], and lottery-ticket hypothesis [3][4]. We believe these are not direct baselines as they involve a critical step of tuning model parameters after compression, whereas the proposed adapter pruning, and the considered baselines directly act on an off-the-shelf trained model without further (or iterative) training of the pruned model. We discuss some of these baselines in lines 177-181. Since the reviewer pointed it out, we will include a separate section in the paper comparing the approach of adapter pruning with the relevant baselines. We extended our experiments to compare with another baseline i.e. random pruning of weights. We discuss it in Clarification-3 (Table 3) of reviewer rtfk.\n\n==\n\nReferences:\n\n- [1] Louizos, Christos, Max Welling, and Diederik P. Kingma. \"Learning sparse neural networks through $ L_0 $ regularization.\" arXiv preprint arXiv:1712.01312 (2017).\n- [2] Idelbayev, Yerlan, and Miguel A. Carreira-Perpinán. \"Low-rank compression of neural nets: Learning the rank of each layer.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n- [3] Wu, Jiarun, and Qingliang Chen. \"Pruning adapters with lottery ticket.\" Algorithms 15.2 (2022): 63.\n- [4] Wu, Jiarun, et al. \"Pruning Adatperfusion with Lottery Ticket Hypothesis.\" Findings of the Association for Computational Linguistics: NAACL 2022. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The paper tackles an important problem in NLP research - parameter-efficient learning, specifically the authors try to determine a minimal set of adapter parameters required for strong performance.\n2) The proposed approach is elegant and motivated from linear algebra standpoint - preserving the tropical adapter hypersurface.\n", "reasons_to_reject": "1) The mathematical motivation should be made more clear; Algo 1 box is not very well explained, some notations are not defined.\n2) For the experiments, authors only compared their method with \"standard\" - are there more relevant baselines? If no - that's ok, but needs to be discussed. ", "questions_to_authors": "1. Why standard deviation is missing in Table 1?\n2. Could you explain Fig 2 better?", "paper_topic_and_main_contributions": "The paper proposed a novel approach for adapter pruning using their tropical characteristics. The proposed method outperforms several existing baselines across multiple datasets.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "S7riuzox8O", "reviewer": "Reviewer_rtfk", "summary": "", "questions": "Why is it important to find adapter parameters to preserve the hypersurface geometry? Any comparison between this idea with other pruning ideas?  Is it this method to limited to adapter pruning? If not, how does this method compared to other pruning methods on other tasks? \n\nWhat is the bottleneck dimension value used in this paper? Do you find any relationship between dimension and pruning ratio? \n\nHow does this method compared to the setting of finetuning the similar amount of parameters which are randomly selected?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Clarification 1]: Why find adapter parameters that preserve hypersurface geometry**\n\n- (Problem formulation): Find parameters that are least important to the network, such that removing them does not lead to significant performance degradation on the task.\n\n- (Why classical pruning is insufficient): Classical magnitude-based pruning is prone to errors as low magnitude weights can still have a huge impact on the adapter's output. Thus pruning them from the network can potentially lead to a significant drop in the model's performance. We posit that the magnitude of parameters is not sufficient to convey their importance to the network.\n\n- (Hypersurfaces to rescue): The hypersurfaces tell us about the characteristics of the function defined by the network, here the network is adapter modules. Thus, preserving the adapter's hypersurface is equivalent to preserving parameters that are important to the function, irrespective of their magnitudes. This analysis would provide extra insights into parameter importance to the network.\n\n- (How to find hypersurfaces): Because of the nonlinearity of the function, it is challenging to find exact hypersurfaces formed by the ReLU-based adapter. Thus, we adopt the zonotopes, a dual analysis that helps us identify important weights with more ease. During magnitude-based pruning, we find that preserving weights that are important for zonotope geometry retained more performance from the model than alone magnitude-based pruning of the same fraction of parameters disregarding their importance to hypersurfaces.\n\nThus, finding parameters that are important for adapter hypersurfaces (function expressed by adapters) and preserving them during the magnitude-based pruning is highly effective at retaining model performance. As per the reviewer's suggestion, we plan to streamline the discussion about the paper's motivation and proposed approach in the provided extra space after acceptance.\n\n=======================\n\n**[Clarification 2] What are the bottleneck dimensions used in this setting?**\n\nThe bottleneck dimension $r$ we used is 48. Hence an adapter block looks like the following:\n$768 \\rightarrow 48 \\rightarrow 768$ with ReLU activation after the down-projection. We chose $r$ as per the wide range of experiments carried out by [1]. Since the reviewer pointed out that the bottleneck dimension can be a huge contributing factor to the pruning ratio, we performed new experiments on a text classification task to investigate it. Table 1 shows full model performance with a change in $r$ and Table 2 shows the impact of $r$ on the pruning ratio.\n\n|Bottleneck dimension ($r$) |Accuracy|\n|-------------------------|--------------|\n|2|88.30|\n|8|87.36|\n|32|87.58|\n|48|87.99|\n|64|88.13|\n|128|87.89|\n|Avg|87.87 $\\pm_{0.32}$|\n_Table1: Impact of $r$ on classification accuracy. Experiments were carried out on the rotten tomatoes (RT) dataset._\n\n\n|Bottleneck dimension ($r$) |% Pruning |@ 1||% Pruning |@ 5||% Pruning|@ 10|\n|-------------------------|--------------|---------------|--------|-------------------------|---------------|---------------|---------------|---------------|\n||Standard | Tropical ||Standard |Tropical||Standard|Tropical|\n|2|68.49|68.49||78.91|78.91||83.09|**87.30**|\n|8|79.24|79.24||83.81|**92.98**||86.08|**97.65**|\n|32|75.48|**83.65**||80.94|**86.41**||90.20|**93.38**|\n|48|82.54|82.54||88.47|**94.28**||94.16|**96.96**|\n|64|75.81|**85.95**||85.05|**89.38**||88.10|**94.13**|\n|128|68.47|**71.25**||80.73|**84.17**||84.01|**87.29**|\n|Avg|73.33$\\pm_{5.47}$ |   **80.16**$\\pm_{6.36}$ ||    81.02$\\pm_{5.17}$ |    **87.70** $\\pm_{6.02}$ ||    86.10 $\\pm_{4.37}$ |    **92.91** $\\pm_{4.51}$|\n\n_Table 2: Percentage pruning with respect to drop in accuracy. Pruning @ x denotes the percentage of parameters pruned with 1% drop in accuracy._\n\n**Observation:** We observe that irrespective of the bottleneck dimension $r$, Tropical pruning consistently outperforms Standard pruning. However, in Table 1 with the unpruned model and Table 2 with different pruning ratios, we do not notice any consistent relation between $r$ and model performance on the task. This answers the question from the reviewer: **What is the bottleneck dimension value used in this paper? Do you find any relationship between dimension and pruning ratio?**. While such an analysis could be highly task-dependent, if the reviewer feels that this experiment should be included in the paper, we would be happy to do so.\n\n=======================\n\n**[Clarification 3] How does this method compare to the setting of finetuning the similar amount of parameters which are randomly selected?**\n\nSince our approach is applicable to off-the-shelf models and in the inference phase, we believe a more concrete baseline would be randomly pruning the same percentage of parameters. As per the reviewer's suggestion, we carried out experiments on randomly selecting the same amount of pruned parameters. In Table 3, we can observe that random pruning of parameters hurts the model performance badly, dropping it close to a random predictor i.e. ~50\\%.\n\n|Bottleneck dimension ($r$) |Standard | Tropical |Random|\n|-------------------------|--------------|---------------|--------|\n|2|85.1|85.1|50.0|\n|8|84.99|88.18|65.29|\n|32|86.11|88.27|50.0|\n|48|85.45|86.39|50.5|\n|64|84.89|87.89|61.16|\n|128|82.64|85.36|50.28|\n| Avg                 | 84.66 $\\pm_{1.16}$ | **87.01** $\\pm_{1.12}$ | 54.54 $\\pm_{6.23}$ |\n\n_Table 3: Table showing performance of a model trained on RT with pruning 80% of the adapter parameters._\n\n=======================\n\n**[Clarification-5]: Any comparison between this idea with other pruning ideas?**\n\nThere are several related works such as L0 sparsification [2], low-rank compression [3], and lottery-ticket hypothesis [4][5]. We believe these are not direct baselines as they involve a critical step of tuning model parameters after compression, whereas the proposed adapter pruning, and the considered baselines directly act on an off-the-shelf trained model without further (or iterative) training of the pruned model. We discuss some of these baselines in lines 177-181. Since the reviewer pointed it out, we will include a separate section in the paper comparing the approach of adapter pruning with the relevant baselines. We extended our experiments to compare with another baseline i.e. random pruning of weights. We discuss it in Table 3 above.\n\n=======================\n\nReferences:\n- [1] Pfeiffer, Jonas, et al. \"Adapterhub: A framework for adapting transformers.\" arXiv preprint arXiv:2007.07779 (2020).\nReferences:\n- [2] Louizos, Christos, Max Welling, and Diederik P. Kingma. \"Learning sparse neural networks through \n regularization.\" arXiv preprint arXiv:1712.01312 (2017).\n- [3] Idelbayev, Yerlan, and Miguel A. Carreira-Perpinán. \"Low-rank compression of neural nets: Learning the rank of each layer.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n- [4] Wu, Jiarun, and Qingliang Chen. \"Pruning adapters with lottery ticket.\" Algorithms 15.2 (2022): 63.\n- [5] Wu, Jiarun, et al. \"Pruning Adatperfusion with Lottery Ticket Hypothesis.\" Findings of the Association for Computational Linguistics: NAACL 2022. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper studies an important problem in parameter-efficient tuning methods.\n2. The proposed method outperforms the magnitude-based baseline.", "reasons_to_reject": "1. The motivation behind the design is not introduced clearly. Why is it important to find adapter parameters to preserve the hypersurface geometry? Any comparison between this idea with other pruning ideas? \n2. The configuration of adapter is not clearly investigated. The bottleneck dimension is an important factor to determine the performance as well as influencing the pruning ratio. What are the bottleneck dimensions used in this setting?", "questions_to_authors": "Why is it important to find adapter parameters to preserve the hypersurface geometry? Any comparison between this idea with other pruning ideas?  Is it this method to limited to adapter pruning? If not, how does this method compared to other pruning methods on other tasks? \n\nWhat is the bottleneck dimension value used in this paper? Do you find any relationship between dimension and pruning ratio? \n\nHow does this method compared to the setting of finetuning the similar amount of parameters which are randomly selected?", "paper_topic_and_main_contributions": "This paper focuses on adapter pruning by studying the tropical characteristics of trainable modules. The authors propose a novel method based on tropical geometry to identify and remove irrelevant adapter parameters without altering the orientation of underlying tropical hypersurfaces. The main contribution lies in introducing a systematic optimization problem formulation for adapter pruning. Experimental results on five NLP datasets demonstrate the effectiveness of the proposed approach, which outperforms magnitude-based baselines and shows the combined approach's superiority across different tasks.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Qx0N0egB9T", "reviewer": "Reviewer_pb9m", "summary": "", "questions": "1. What is the essential motivation to adopt the tropical hypersurface for pruning?\n2. In most cases, the adapter is small in size, is it necessary to prune the adapter?\n3. Why the contribution of loss1 in Alg. 1 is set to 0.5?\n4. What is the time complexity of the optimization process?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Clarification-1]: Motivation.**\n\n- (Problem formulation): Find parameters that are least important to the network, such that removing them does not lead to significant performance degradation on the task.\n\n- (Why classical pruning is insufficient): Classical magnitude-based pruning is prone to errors as low magnitude weights can still have a huge impact on the adapter's output. Thus pruning them from the network can potentially lead to a significant drop in the model's performance. We posit that the magnitude of parameters is not sufficient to convey their importance to the network.\n\n- (Hypersurfaces to rescue): The hypersurfaces tell us about the characteristics of the function defined by the network, here the network is adapter modules. Thus, preserving the adapter's hypersurface is equivalent to preserving parameters that are important to the function, irrespective of their magnitudes. This analysis would provide extra insights into parameter importance to the network.\n\n- (How to find hypersurfaces): Because of the nonlinearity of the function, it is challenging to find exact hypersurfaces formed by the ReLU-based adapter. Thus, we adopt the zonotopes, a dual analysis that helps us identify important weights with more ease. During magnitude-based pruning, we find that preserving weights that are important for zonotope geometry retained more performance from the model than alone magnitude-based pruning of the same fraction of parameters disregarding their importance to hypersurfaces.\n\nAs per the reviewer's suggestion, we plan to streamline the discussion about the paper's motivation and proposed approach in the paper.\n\n===\n\n**[Clarification-2] Is it necessary to prune the adapter?**\n\nA task-specific adapter can take up to 10% of the overall space. When the backbone model size becomes large, consider a large language model for instance, even 10% of the extra space can be significant, thus even the parameter-efficient adapters can significantly limit the number of tasks LLMs can be specialized on. We show that the adapters can be pruned up to 80% with a little trade-off in accuracy, thus able to specialize over 5 times more number of tasks as compared to without pruning. While due to resource limitations, we could not experiment on LLMs, we provide analysis over various smaller LMs such as RoBERTa, DeBERTa, and ALBERT (please refer to Table 1 of reviewer 2vRX). Adapter pruning has seen recent interest in the research community such as structured pruning [1], adapter pruning with lottery tickets [2], and adapter-fusion pruning [3]. These works perform adapter pruning with (iterative) training, tropical pruning is directly applied to the on-the-shelf model. Thus, making [1][2][3] less interesting baselines. If the reviewer feels we should include an elaborate discussion, we will do so in the paper.\n\n===\n\n**[Clarification3]: Why the contribution of loss1 in Alg. 1 is set to 0.5?**\n\nIt's part of the expression to optimize, it denotes the factor $\\frac{1}{2}$ in equation-2. The derivation can be found in the related literature [4] as well as the appendix.\n\n===\n\n**[Clarification4]: What is the time complexity of the optimization process?**\n\nSince it is a convex optimization problem, we observe with step size $\\eta$ (=0.01), the process converges to global minima in $T$ steps. Each step constitutes $r$-times gradient updates. Thus, the time complexity is $O(rT)$. Empirically, we find for $r=48$, the process converges in $T=200$ steps. We observe the optimization process is very fast, converging in about 300 seconds of time on a standard CPU machine. Thanks for pointing it out, we will include the time-complexity analysis in the paper.\n\n==\n\nReferences:\n\n- [1] Hedegaard, Lukas, et al. \"Structured Pruning Adapters.\" arXiv preprint arXiv:2211.10155 (2022).\n- [2] Wu, Jiarun, and Qingliang Chen. \"Pruning adapters with lottery ticket.\" Algorithms 15.2 (2022): 63. \n- [3] Wu, Jiarun, et al. \"Pruning Adatperfusion with Lottery Ticket Hypothesis.\" Findings of the Association for Computational Linguistics: NAACL 2022. 2022.\n- [4] Alfarra, Motasem, et al. \"On the decision boundaries of neural networks: A tropical geometry perspective.\" IEEE Transactions on Pattern Analysis and Machine Intelligence 45.4 (2022): 5027-5037.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper tries to prune the adapter from the perspective of hypersurfaces.\n2. The paper is evaluated on five NLP datasets.\n3. This paper is clear and easy to follow.", "reasons_to_reject": "1. The major concern about this paper is the motivation to conduct pruning on the tropical hypersurface.\n2. Pruning methods have been investigated long ago; this paper does not include state-of-the-art baselines to show the comparison results. \n3. Several typos and grammar issues.", "questions_to_authors": "1. What is the essential motivation to adopt the tropical hypersurface for pruning?\n2. In most cases, the adapter is small in size, is it necessary to prune the adapter?\n3. Why the contribution of loss1 in Alg. 1 is set to 0.5?\n4. What is the time complexity of the optimization process?", "paper_topic_and_main_contributions": "This paper proposes to prune the adapter with the tropical characteristics, which is formulated as an optimization problem to identify the row-sparse projection matrices. The aim is to minimize the distance of tropical hypersurfaces before and after pruning. The method is tested on five NLP classification datasets.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. [Line 118] In the equation of Line 118, it is not clear variables A and B are contained in which functions.\n2. In the caption of Table 2, “S, T” should be “S and T ..., respectively.”\n3. [Line 260] In Conclusion, “studiying” should be “studying”.\n4. Figure 1 is a bit small."}}]

Author rebuttal:

Meta-review: This paper uses concepts from Tropical geometry to identify parameters for pruning from adapters. The paper is overall well-written considering the mathematical concepts that most of the NLP community is less likely to be familiar with. Reviewers overall appreciated the elegancy of the approach, and the results. One major question that immediately comes to mind is why prune the adapters which are already supposed to be relatively small, and not prune the full model? Based on the discussion (with reviewer pb9m), it seems like the answer is that the proposed approach relies on certain geometry assumptions that are specific to adapters. This should be better explained and discussed in the paper in order to properly define the scope of the introduced solution. Overall, while this limited scope slightly reduces the excitement, it could still be of interest to other researchers in the community. The soundness should be improved by the comments from the reviewers, specifically, explaining the assumptions/ constraints and why they are suitable for adapters, and comparing to other pruning baselines.
Decision: Accept-Findings