Prompt-prompted Adaptive Structured Pruning for Efficient LLM Generation
Authors: Harry Dong
Beidi Chen
Yuejie Chi
Venue: COLM (2024)
Keywords: adaptive pruning, structured pruning, mixture of experts, sparsity, efficiency, generation, structured sparsity, pruning, LLM, feedforward, activation function
TL;DR: We exploit naturally existing structured sparsity in LLM feedforward blocks to adaptively prune feedforward neurons for efficient generation.

Abstract: With the development of transformer-based large language models (LLMs), they have been applied to many fields due to their remarkable utility, but this comes at a considerable computational cost at deployment. Fortunately, some methods such as pruning or constructing a mixture of experts (MoE) aim at exploiting sparsity in transformer feedforward (FF) blocks to gain boosts in speed and reduction in memory requirements. However, these techniques can be very costly and inflexible in practice, as they often require training or are restricted to specific types of architectures. To address this, we introduce GRIFFIN, a novel training-free and calibration-free method that selects unique FF experts at the sequence level for efficient generation across a plethora of LLMs with different non-ReLU activation functions. This is possible due to a critical observation that many trained LLMs naturally produce highly structured FF activation patterns within a sequence, which we call flocking. Despite our method's simplicity, we show with 50% of the FF parameters, GRIFFIN maintains the original model's performance with little to no degradation on a variety of classification and generation tasks, all while improving latency (e.g. 1.29$\times$ and 1.25$\times$ speed-ups in Gemma 7B and Llama 2 13B, respectively, on an NVIDIA L40). Code is available at https://github.com/hdong920/GRIFFIN.

Published as a conference paper at COLM 2024
Prompt-prompted Adaptive Structured Pruning for Efficient LLM Generation
Harry Dong, Beidi Chen, Yuejie Chi
Department of Electrical and Computer Engineering
Carnegie Mellon University
Pittsburgh, PA 15213, USA
Meirryd,biddic,, Heandrew. edu
Abstract
With the development of transformer-based large language models (LLMs), they have been applied many fields due to their remarkable utility, but this comes at considerable computational cost at deployment. Fortunately, some methods such as pruning or constructing a mixture of experts (MoE) aim at exploiting sparsity in transformer feedforward (FF) blocks to gain boosts speed and reduction in memory requirements. However, these techniques can be very costly and inflexible in practice, as they often require training or are restricted to specific types of architectures. To address this, we introduce GRIFFIN, novel training-fre and calitration-free method that selects unique FF experts at the sequence level for efficient generation across plethora of LLMs with different non- Ren-ReLU activation functions. This is possible due to a critical observation that many trained LLMs naturally produce highly structured FF activation patterns within a sequence, which we call flocking. Despite our method's 's simplicity, we show with 50% of the FF parameters, GRIFFIN maintains the original model's s performance with little to no degradation on a variety of classification and generation tasks, all while improving latency (e.g. 1.29x and 1.25x speed-ups in Gemma 7B and Llama 2 13B, respectively, on an NVIDIA L40). Code is available at fitpps///ght thwb.//ggtttuu...c hotpu//o.ooo/oooogg/////FFFF
Introduction
Transformers Vaswani et al. 2017) have demonstrated incredible capabilities across
plethora of domains (Lin et al. 2022; Khan et al. 2022 Nerella et al. 2023). Their large
language model (LLM) successors [Tourron et al. 2023 leam et al. 2023 Jiang et al. 2023
2024 Team et al. 2024 Anthropic 2024) have pushed the bar higher, but these behemoths
have become pertormative at the price of enormous amounts of computation and memory demands. One significant contributor is the model size itself. Not only is storage an issue, model layers tend to be wide and plenty, slowing down inference. Moreover, given the existence of sparse structures in LLMs, especially in feedforward (FF) blocks (Geva al., 2020: Dettmers et al. 2022 Li et al. 2022 Liu et al., 2023), these models waste computation on intermediate features with little no impact on the final result. For instance, has been observed that in OPT-175B (Zhang et al. 2022), fewer than 5% of neurons in FF blocks have nonzero values per token (Liu et al. 2023), meaning 95% of the compute in each FF block is wasted. Usually consisting of around two-thirds of the parameters in an LLM, FF blocks can be serious memory and compute bottlenecks. These inefficiencies are highly problematic in latency-sensitive scenarios like in chatbots and autonomous vehicles.
There have been many methods to exploit sparsity in LLMs for efficiency gains, such as pruning model weights and constructing mixtures of experts (MoEs). Pruning removes low-impact pre-trained weights to reduce storage, yet this often does not translate to real speed-ups in practice, unless the pruning is done in structured and hardware-friendly manner (Xia et al. 2022; Santacroce et al.) 2023 Ma et al., 2023 Li et al.) 2023 Xia et al.) 2023) which typically causes greater performance deterioration. MoEs better preserve the

---

Published as a conference paper at COLM 2024
original performance by adaptively selecting subsets of the model to use per input but also come with drawbacks. Unless the model has been trained in this fashion (Fedus et al. 2022 Jiang et al., 2024), it will need to learn cheap yet effective gating function (expert selection mechanism) and sometimes even require full fine tuning. Perhaps an even bigger weakness of many of these methods is the limited effectiveness and special considerations to strictly enforce ReLU-like activation functions (Zhang et al., 2021: Li et al., 2022 Csordás et al.
2023). In summary, pruning and MoËs provide enormous benefits but also come with steep challenges:
1. Structured sparsity is difficult and costly to enforce without serious performance degradation.
2. Adaptive selection of model subsets need to be cheap and accurate.
3. The method should be effective with various activation functions.
Daunting at first, these challenges become surmountable because of a simple observation of phenomenon we call flocking, highly consistent sparse activations that persist throughout sequence observed in many LLMs. Flocking emerges in FF activations (inputs into the FF down projection) when we focus ои a sequence's relative activation magnitudes instead of the absolute values. Examples from Llama 2 7B and Gemma 7B are shown in Figure The key takeaway is that neurons which produce high relative magnitudes are naturally shared across tokens within sequence, as seen with the distinct vertical streaks. Increasingly bizarre, the two models have different architectures and different Ron- activation functions.
Llama 2 7B
Gemma 7B
0
0.10
0
0.10
100
0.08
100
0.08
Token Index Delaliig 280
Token Index
200
0.06
200
0.06
Magnitude
300
1004
0.04
300
0.04
400
400
100
200
300
400
500
100
200
300
400
500
Neuron Index
Neuron Index
Figure 1: Relative FF activation magnitudes of the first 512 features and tokens across sequence from PG-19 (Rae al. 2019: Gao al., 2020) in layer 10 of Llama 2 7B (left) and Gemma 7B (right). These heatmaps show flocking, where relative activation magnitudes are shared within sequence, illustrated with the distinct dark vertical streaks. More examples in Appendix
Unlike existing pruning or MoE methods, we exploit flocking our design of GRIFFIN (Gating by Repetition In Feedforward Intermediate Neurons), highly performative and efficient fraining-fres method to adaptively activate FF neurons. GRIFFIN does this by using sequence's is prompt to ietermine the experts activate during generation, allowing it to overcome all of the aforementioned challenges:
1. No Preparation: Our no-cost method is completely training-fres and requires calibration. Moreover, the simple implementation of GRIFFIN means it can be dropped into any FF block and instantly deployed.
2. Simple Expert Selection: Flocking in the prompt reveals the most relevant FF neurons for generation with little to no performance loss. The selection process is parameter-fri and adds negligible overhead.
3. Model & Activation Function Diversity: Thorough experimentation demonstrates the efficacy of GRIFFIN on numerous models, including Ilama 2 (Touvron et al.)
2

---

Published as a conference paper at COLM 2024
2023), , Gemma (Team et al., 2024), Mistral (Jiang et al. 2023), OPT (Zhang et al.
2022 and ReluLlama [leam 2023). Together, the tested activation functions include ReLU, SwiGLU, GEGLU, and ReGLU (Shazeer 2020).
In this paper, we show GRIFFIN is a simple and strong adaptive structured pruning method because of flocking. In the next section (Section)) we discuss in more detail some strengths and weaknesses of current methods that seek to improve FF efficiency. Then, we formalize the problem we are trying to solve, along with its motivation in Section 3 We present our novel approach in Section 4.2 which requires a thorough examination of the surprising phenomenon of flocking shared by many LLMs in Section 4.1 Our rigorous experiments demonstrate GRIFFIN preserves performance on classification and generation even after removing 50% of FF neurons (Section 5.1), all while having lower latency (Section 5.2). For instance, GRIFFIN reduces the number of active parameters in Llama 13B from T3B to 8.8B during generation to improve latency by 1.25x with almost no loss in performance. Finally, we show our method's 's incredible scalability and robustness in several ablation studies (Section 5.3
2 Background
Our novel method and FF activation observations are inspired and motivated by ample amounts of previous research that sought to characterize FF sparsity and accelerate LLMs.
Feedforward Activation Sparsity. The observation that transformer FF blocks produce sparse activations is not new (Geva et al. 2020 Dettmers et al. 2022 Li al. 2022 Dong et al.
2023 Liu et al. 2023). In ReLU-based LLMs like OPT (Zhang et al. 2022). the activatios can be exceptionally sparse and become more apparent for larger models (Lu et al. 2023). As more models use non-sparse activation functions like GLU variants (Shazeer) 2020), it
is difficult for neurons to have no contribution to the output since these functions do not have an interval that maps to zero. Without exact sparsity, the efficacy of these methods becomes limited. As such, this has ushered wave of models that are either adapted from available models (Zhang et al.) 2021: Liu et al., 2021 Mirzadeh et al.) 2023; Zheng et al., 2024 Jiang et al. 2024) trained from scratch (Fedus et al. 2022) which can produce activations that are exactly zero with little to no performance loss. Even so, these methods require considerable amounts of computational resources.
Pruning. Pruning [eCunn et al. 1989) is one sparsity-guided way to tackle compute and memory bottlenecks of models. Previously, the common method would be some variation of iteratiieely rounding weights down to zero based on some score and retraining to recover any lost performance (Frankle & Carbin 2018 Blalock et al. 2020 Liang et al. 2021 Liu et al. 20244) While this can result in most parameters being pruned, this method comes with few issues. First, with the increasing scale of LLMs, retraining becomes impractical for most. Fortunately, cheap methods to effectively prune LLMs have been developed (Frantar |Aiittrrh] 2023 Sun al. 2023 Jaiswal et al. 2023 Dery al.) 2024). The second issue is that unless pruning is done in a structured manner Xia et al. 2022) Santacroce et al. 2023
Ma et al. 2023 et al. 2023 Xia et al. 2023), it is difficult to see real computational savings, yet structured pruning often leads to much more severe performance degradation. Third, pruning usually enforces sparsity to be static at inference which can be strong assumption since FF blocks are widely believed to contain the model's memory (Geva et al., 2020).
Mixture of Experts. Making sparsity more dynamic has motivated the design of mixture- of-experts (MoEs) (Jacobs et al. 1991 to avoid computing low-impact features in trained
models at varying granularities (Zhang et al. 2021 Liu et al. 2023; Piórczyńíkii et al. 2023 Csordás et al., 2023 Yerram et al., 2024 Zheng et al. 2024). The main idea of MoËs is to use a gating function to identity a small subset of neurons that will be used to compute the adaptive form of pruning. In the ideal case, all active neurons are selected and inactive neurons are ignored for each input. However, current methods either require training or rely ReLU or ReL.U-like activation functions.
layer's 's output,
3

---

Published as a conference paper at COLM 2024
3 Problem Formulation
This section contains an overview of different components of the FF block followed by formulation of the FF compression problem which our method aims to tackle. Since FF blocks operate identically and independently for each token unlike attention, we begin with defining the FF block with single column vector input x E IRD
FF(x) = (FF (F1 (x)
(1)
where (z) = W2z + b2 is a linear transformation and FF, is nonlinear. This describes wide range of architectures and arbitrary activation functions o. For instance, in OPT,
FFI (x) cr (WW + b1)-
(2)
For FF blocks with GLU variants such as in Llama 2 and Gemma,
such
(x) = (W, + bg) @ (Wi++b)
(3)
where @ signifies element-wise multiplication. For all examples, W1, W3 R and W2 E DDD where typically, DFF >> D. We refer to = Fx) as the FF activations.
A popular method to compress FF blocks (which is also the goal of GRIFFIN, adaptive neuron pruning, and many MoB methods) to find W1 E R b E R², and W2 Rxc (additionally Wg E IR and by E R needed) where k < DFF such that when the FF block is reparameterized with these matrices, the output value is preserved. Namely, for
= FF, (x) = or (W. bg) @ (Wx++b)),
(4)
2F W₂ + b2,
FF(x) Re [FF (x)), and similarly for FF blocks with non-GLU functions. For instance, in the MoE setting, these smaller matrices can vary from token to token and are actually selections of chunks of the original structures. Crucially, solution to this problem leads to multiplication with smaller matrices which are naturally more efficient on GPUs and TPUs (Fatahalian et al. 2004 Wang et al. (209)). Note that for all equations defined in this section, they operate independently on each row of a length sequence input X RSSD (e.g. the
activations for sequence are Z (X) E R5xDDF).
4 From Flocking to GRIFFIN
Here, we take deeper dive into the phenomenon of flocking and describe the intuitive algorithm of GRIFFIN which is directly inspired by it.
Liama 2 7B
Gemma 7B
LAye
3,2
0.3
0.5
Jaccard Similarity
Similarity
0.7
0.7
23
0.1
0.4
0.2
10
14
0.-
0.4
3.6
0.4
3.6
Top Fraction Neurons
Top Fraction Neurons
Figure 2: Average Jaccard similarity between WikiText samples' top FF neuron activations in Llama 7B (left) and Gemma 7B (right). Higher values indicate greater similarity.

---

Published as a conference paper at COLM 2024
Generation FF
Prompt FF
W1
W
X
X
Z
Top-k
Expert Selection
Tokens
2
0.4 -0.4
0.2
Scale Raws Uni Veclors
Column Nom
-10
14
-0.6
-0.3
-0.4
0.7
0.4
0.7
0.2
1.8
0.0
0.0
-0.2
0.0
0.7
0.0
-0.7
Figure 3: GRIFFIN overview. Relative activations from the prompt determine expert neurons to use for generation.
4.1 Observing Flocking
Flocking arises when we look at the relative impact of each neuron per token within sequence. To this, normalize rows of Z to be unit vectors to construct Z E SxP (i.e. Z7; [Z] the relative activations. We show example relative activation magnitudes for a sequence in Llama 2 7B and Gemma 7B in Figure [ Since there are distinct vertical streaks, this intriguingly implies that activations that have relatively greater weight are common across all tokens in a sequence. Notably, Llama 2 7B and Gemma 7B use SwiGLU and GEGLU activations, respectively, along with other major architecture differences. We call this phenomenon flocking, like highly organized groups of birds, and we observe this in virtually all FF layers (see Appendix and randomized inputs (see Appendix C.
While relative activations magnitudes are shared within a sequence, they are not generally shared between sequences. We show this by taking the f2-norm of Z along the token axis to obtain length DFF vector for each sample sequence, roughly capturing the contribution of each FF neuron throughout a sequence. Taking the top-k of this for each sample at each layer, we find the Jaccard similarity between two sequences based the indices selected for different k. In other words, we compute the intersection over union of every unique pair of top-k sets. Higher values indicate more similar top-k sets. From Figure where we aggregate Jaccard similarities across Wikilext (Merity et al., 2016) samples, we observe lack of inter-sample activation similarities for the vast majority of layers in Llama 2 7B and Gemma 7B, unless the sets of selected neurons are large. This lack of consistency implies statically pruning entire FF neurons without retraining would be less effective than a more adaptive method.
4.2 GRIFFIN Algorithm
Using our insight on flocking, we introduce GRIFFIN as simple general purpose and training-fre free adaptive structured pruning method for efficient generation, captured in Figure 3 In nutshell, we select neurons during the prompt phase of each sample which are then used for the entire duration of the generation phase. This effective approach is based on key observation on flocking: since lokens within a sequence share activation patterns, the prompt and generated tokens will also share activation patterns.
Prompt Phase Expert Neuron Selection. Our expert neurons are chosen at the sequence level, so we need to consider the dynamics of the entire input sequence rather than just single token when choosing our neurons. To select expert neurons, need statistic

---

Published as a conference paper at COLM 2024
MODEL
HELLASWAG
PIQA
CОРA
ARC-E
ARC-C
BOOLQ
LLAMA 7B MAGNITUDE
57.16
78.07
87.00
76.35
43.34
77.71
57.12
77.31
84.00
70.33
40.27
66.54
GRIFFIN
57.11
77.69
86.00
74.54
42.75
73.15
LLAMA 13B
60.06
79.05
90.00
79.46
48.46
80.61
MAGNITUDE
60.00
79.00
88.00
74.07
46.25
70.52
GRIFFIN
60.10
79.11
89.00
77.19
46.84
78.50
GEMMA 7B
60.61
80.30
88.00
82.74
50.09
83.49
MAGNITUDE
46.24
73.12
57.00
45.20
32.76
62.84
GRIFFIN
60.62
79.98
88.00
81.65
50.09
81.90
MISTRAL. 7B MAGNITUDE
61.21
80.58
92.00
80.89
50.43
83.61
61.15
80.36
86.00
74.20
48,89
60.40
GRIFFIN
61.18
80.52
91.00
79.25
50.00
80.06
OPT 6.7B
50.48
76.28
81.00
65.53
30.55
66.12
MAGNITUDE
49.21
72.63
79.00
47.60
27.13
40.15
GRIFFIN
50.44
75.63
80.00
63.93
30.55
65.44
OPT 13B
52.46
75.90
86.00
67.05
32.94
65.81
MAGNITUDE
51.31
74.21
81.00
49.41
28.07
38.75
GRIFFIN
52.42
76.17
86.00
66.92
33.19
67.65
Table 1: Classification tasks (0-shot unnormalized accuracy) at 50% FF sparsity.
E RD to inform us of the importance of each neuron. At the prompt phase, do this by taking the f2-norm of Z along the token axis:
11,1
(6)
Taking the indices of the top-k across gives us the neurons will use for this sample's generation phase which make up the set E. Using the expert neurons in e, we can find
W, Wg bg, and W2 by selecting corresponding rows and columns in W1, b1, Wg. bg. and W2, respectively. This is done for every FF block during the prompt phase. Mustrated in Appendix A highlights neurons consistently activated at relatively high intensities.
Generation with Expert Neurons. When generating tokens, we directly use the pruned layers which contain the expert neurons, FF, and FF2, to estimate FF(X) z (FF(FF,(X)) for all future tokens. Llama 2 13B and Gemma 7B, this reduces the active number of parameters from 13B to 8.8B and from 8.5B 5.4B, respectively, during generation.
Experiments
We showcase the superb performance of GRIFFIN on numerous tasks and models (Sec- tion 5.1) while achieving lower latency (Section 5.2) along with study on several of its properties like sequence length scaling and batching (Section 5.3 All experiments are run on NVIDIA L40 GPUs.
5.1 Performance
Using various models, we evaluate on several generation and classification tasks. For generation, we evaluate on XSum (Narayan et 2018), Mail (Nallapati al., 2016), COQA (Reddy et al.) 2019), and SCROLLS QASPER (Dasigi et al. [L212 SSaham For classification, we we evaluate on HellaSwag (Zellers et al. 2019), PIQA (Bisk
et al.
2022
et al.
COPA (Roemmele et al., 2011), ARC-Easy Challenge Clark et al., 2018 and BoolQ (Clrk et al. 2019). With the exception of XSum and CNN Maily,,,all, we use LM Evaluation Harness for our experiments (Gao et al.) 2023). Aside from comparing with the original LLM, we also compare GRIFFIN with a static neuron pruning method based
2020
neuron magnitudes. Similar to neuron magnitude pruning, this baseline selects expert

---

Published as a conference paper at COLM 2024
MODEL
XSUM
CNN/DAILMMAL (ROUGE-1/2/L)
COQA (F1/EM)
QASPER
(ROUGE-1/2/L)
(F1)
LLAMA 7B MAGNITUDE
27.15/9.06/22.22
10.08/0.13/9.55
77.35/63.88
26.31
9.71/1.31/8.9
9.66/0.63/9.32
56.59/39.93
12.93
ADAPTIVE WANDA GRIFFIN
25.59/8.18/21.3
9.90/0.26/9.39
77.16/63.53
27.61
24.75/7.4120.555
10.97/0.66/10. 10.37
/3.58
25.76
LLAMA 2 13B MAGNITUDE
26.90/9.45/22.09
2.51/0.22/2.3
79.18/66.3
28.32
5.72/0.78/
0.02/0.00/0.02
65.69 /4.9/47.88
15.55
ADAPTIVE WANDA
14.65/8.08/2000
1.13/0.11/1007
19.43/67.43
27.98
GRIFFIN
25.69/7.85/ 20.89
3.31/0.78/3.07
79.22/66.62
27.91
GEMMA 7B MAGNITUDE
26.86/9.15/
17.45/4.15/15.94
79.04/65.25
30.78
1.49/0.01 01..0/11.477 /.447
0.00/0.00/0.00
2.92/1.50
7.02
ADAPTIVE WANDA
24.76/7.99/0..25
12.10/2.21/11.20
75.79/61.87
30.62
GRIFFIN
25.86/7.81/20.93
18.26/4.75/16.58
78.52/64.62
27.37
MISTRAL 7B MAGNITUDE
28.67/10.21/33.66
/0..0
80.70/67.30
24.56
3.58/0.27 /3331
0.26/0.03/0.26
61.99/45.93
17.18
ADAPTIVE WANDA
25.42/8.40/22 /21.11
0.16/0.00/0.11
80.68/67.60
24.73
GRIFFIN
26.59/8.70/22.1
1.26/0.21/1.17
50.15/66.50
23.92
OPT 6.7B
23.60/7.04/19.
13.85/1.54/13.04
68.70/54.98
18.53
MAGNITUDE
1.63/0.00/1.54
1.20/0.00/1.17
31.53/16.52
7.28
ADAPTIVE WANDA
22.40/6.04/18.57
12.94/1.14/12.23
69.01/55.22
18.47
GRIFFIN
21.17/542/17.58
13.01/1.06/12.26
68.99/55.00
17.40
OPT 13B
25.14/7.93/20.68
13.22/1.18/12.46
69.51/55.67
20.58
MAGNITUDE
1.23/0.00/1.21
1.29/0.00 /1.29
39.38/27.07
8.87
ADAPTIVE WANDA
23.68/6.97 19.77
1..88/..8//2.888
69.60/56.00
21.30
GRIFFIN
22.11/6.28/18.29
12.92/1.13/12.22
/9.07/54.83
20.16
RELULLAMA
25.10/7.81/20.76
20.95/6.79/19.
78.49/66.73
23.31
MAGNITUDE
9.09/0.22/8.20
8.50/0.14/8.17
19.43/6.46
7.21
ADAPTIVE WANDA
22.62/6.47 18.82
17.46/5.56/165 116.13
78. 78.96/66.99 66.97
21.38
GRIFFIN
21.83/5.88/ 18.09
16.85/4.96/14.0 14.69
78.35/ 67.10
22.29
Table 2: Generation tasks XSum (1-shot), CNN/DailyMail (1-shot), CoQA (0-shot), SCROLLS QASPER (0-shot) at 50% FF sparsity. Magnitude neuron pruning fails in almost every case while GRIFFIN effectively preserves performance.
neurons based on neuron magnitudes in Wi for the generation phase but uses the entire FF blocks for prompting like GRIFFIN. In the case of GLU variants, the neuron-wise norms of W1 and Wg are elementwise multiplied to produce the pruning metric. As we will see, this straightforward baseline achieves great classification results but falters for generation. Another baseline we consider also uses the full model for the prompt and Wanda (Sun et al. 2023) in FF blocks for generation, using the prompt activations to prune FF weights. We reter to this adaptive unstructured pruning baseline as Adaptive Wanda. Note that Adaptive Wanda is completely unstructured and does not reduce the FF activation dimension all.
As our method is designed specifically for generation, we alter classification evaluations to simulate generation. In typical classification tasks, LLMs do not enter the generative phase since the final token output of the prompt phase indicates the class. Consequently, directly applying GRIFFIN for classification tasks trivially yields the exact performance of the original model. Therefore, we treat all tokens but the final input token as the prompt. Then, the model forced go into the generation phase for one step.
We start with a look into the relationship between the sparsity levels and performance degradation. This translates to varying when we select the top-k our statistic s. To compare the performance degradation across multiple tasks, we plot the ratio of the final performance metrics between GRIFFIN and the full model in Figure 4 We see most of the performance is preserved at 50% FF sparsity in Llama 2 7B, Gemma 7B, and Mistral 7B. Different tasks have different tipping points where performance sharply drops, which may be related to the difficulty of the task (Yin et al. 2024).
7

---

Published as a conference paper at COLM 2024
Unma 2 7B
Gemma 28
Matrall 7B
De-atte Rerforraccee
Be-atten Rerformaree
0.2
02
Esam
25 DASPER
DOGPER
DAARER
Bosl)
Book)
00
0.2
044
0.5
1.0
3.0
02
0.4
0.
0.5
3.3
3.0
02
0.4
2.0
0.0
100
Figure 4: Relative performance of GRIFFIN for Llama 2 7B (left), Gemma 7B (center), and Mistral 7B (right) as we enforce varying degrees of sparsity per FF block. For all tasks, the original model's performance for each task is normalized to 1.
MODEL
SETUP
PROMPT
FULL
MAGNITUDE
GRIFFIN
LLAMA 13B
2048+128
0.5
6.8
5.4 5.0
5.4 54/5.1
2048+2048
0.5
119.1
95.0 / 83.4
94.9 82.8
GEMMA 7B
2048+128
0.3
4.5
4.1 / 4.2
4.2 / 4.1
2048+2048
0.3
87.1
67.7 / 65.0
67.4 / 65.0
Table 3: Generation phase latency (s). We denote "P + G" as the task of generating G tokens from a length P prompt. When relevant, times are in the format 50% / 75% FF sparsity.
Fixing FF sparsity to be 50% we evaluate on more tasks and models. Table I and Table show the performance of GRIFFIN classification and generation, respectively. We see that magnitude neuron pruning achieves reasonable results for classification but completely annihilates the original performance in most generation settings. For generation, Adaptive Wanda and GRIFFIN perform similarly, preserving most of the performance for all models. However, GRIFFIN is able to accomplish this in structured manner which allows it to be more efficient in practice. For example outputs, see Appendix D) Base models Mistral 7B and Llama 2 13B perform poorly on 1-shot CNN/ MDN//iilyMai,, as they often repeated "\n" or items from the example shot. With 3 shots, both obtain Rouge-1 scores above 25.
5.2 Efficiency
We now present efficiency metrics of GRIFFIN. We collect synthetic datasets with samples having identical lengths and average results across samples. Like many MoE methods, GRIFFIN is ideal for single sample inputs since it is adaptive, such as in the case of personal devices, so we use batch size 1 for these experiments. Using Hugging Face implementations of Llama 2 13B and Gemma 7B at FP16 precision, we measure the latency in different scenarios on an NVIDIA L40 GPU.
Recalling that our magnitude selection baseline is essentially neuron pruning at generation, this has the best possible speed-up since there no expert neuron selection overhead per sample. From Table 3 GRIFFIN matches the best case, producing up to a 1.29x and 1.25x improvement in latency for long generation at 50% FF sparsity in Gemma 7B and Llama 2 13B, respectively. This illustrates that our method is as fast a static neuron pruned LLM during generation while being adaptive to preserve the accuracy of the full model. In offloading settings with large models, our method has the potential to further accelerate inference. For a prompt, GRIFFIN essentially performs structured pruning on the massive network, and if this pruned model can fit on single device, it will avoid offloading for the entirety of generation.
5.3 Ablations and Analysis
Prompt vs. Generation Length. We find that GRIFFIN can potentially be made more robust for long generation by lengthening the prompt. To see this, use language modeling
8

---

Published as a conference paper at COLM 2024
the concatenated version of WikiText to simulate generation. For length S input into the FF block, we designate the first P tokens as the prompt and the last G tokens as the generated portion such that P + G S. The prompt partition is used to calculate our statistic s and determine the expert neurons. The prompt partition uses the full FF block while the generation partition only uses the selected neurons. When comparing the original model with GRIFFIN, we only compute the perplexity of the outputs from the generation partition since the other outputs will be identical. Based on Figure GRIFFIN gets closer to the full model outputs when the prompt length increases and generation length decreases, meaning the difficulty with long generation can be suppressed with longer prompts.
Liama
Gemma
712
250
256
200
1al 0004224 %
Generation Langsh
200
1.0
150
100
100
50
16
64
170
256
517
1874
2048
PromptLonge
Prompt Length
Figure 5: lengtt length generation length for Llama 2 7B (left) and Gemma 7B (right) as measured by increase in perplexity (PPL) from the full model on concatenated WikiText at 50% FF sparsity.
Sharing Selected FF Neurons. We further verify that GRIFFIN is simultaneously robust to batching and more performative than fixed neuron pruning methods using similar pruning metrics in 66. As such, GRIFFIN maintains its reliability as we extend beyond single inputs.
For static pruning, we test two methods. One is use the example shot in the prompt to compute s. Then, for all samples with the same shot, we select the same FF neurons for generation. With the other method, we can extend (6) such that can be a pruning metric for multiple inputs (i.e. the same FF neurons will be pruned). Suppose that s; is the metric for sample with prompt length S,. Then, we use
$
Σ
VS
to select FF neurons. Finding global across all prompts in dataset is the second method. Therefore, for a particular dataset, both methods have fixed expert neurons across samples.
To extend GRIFFIN to larger batch sizes, use Š from where the samples in batch are aggregated. Thus, adaptability is maintained. From Table GRIFFIN (even with batch size >1) achieves better performance than the static methods, reinforcing the performance benefit of adaptability. Interestingly, even though performance decays as we increase the batch size, the decay slow.
MODEL
FULL
SHOT
GLOBAL
GRIFFIN (1) GRIFFIN (4) GRIFFIN (16)
LLAMA 2 7B
27.15
21.11
23.28
24.75
24.26
23.75
GEMMA 7B
26.86
19.76
22.48
25.86
25.01
24.37
Table 4: Rouge-1 scores of 1-shot XSum with varying ways to use the same selected neurons across multiple samples. From left to right, the scores are the result of the full model, FF neurons selected based the prompt, FF neurons selected based on the entire dataset, and GRIFFIN in batch sizes of 1, 4, and 16. All pruning methods use the full model for the prompt and 50% of the neurons during generation.

---

Published as a conference paper at COLM 2024
6 Conclusion
In this work, we have shown a special form of sparsity in FF layers and a simple method to exploit Flocking a curious phenomenon present in many LLMs where tokens within sequence activate neurons at similar intensities. This structure motivated the design of GRIFFIN, a learning-free adaptive structured pruning mechanism to remove FF neurons during inference at the sequence level which preserves the full model's performance on large collection of classification and generative tasks at 50% FF sparsity while achieving lower latency. Furthermore, its applicability extends beyond just ReL.Ubassed LLMs. With its straightforward algorithm and no-cost deployment, GRIFFIN expands the accessibility of numerous LLMs for generative inference.
Acknowledgments
We thank Zixin Wen for insightful discussions. The work of H. Dong is supported in part by the Wei Shen and Xuehong Zhang Presidential Fellowship at Carnegie Mellon University. The work of Y. Chi is supported in part by the grants NSF DMS-2134080 and ONR 400014-19-1-2404.
References
Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024.
BBC. Three us hospitals hit by ransomware. The British Broadcasting Corporation, 2016. URL
https://ww. http:///////eeeeeeeoooooo---//-- 058886610
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? Proceedings of machine learning and systems, 2:129-146, 2020.
Christopher Clark, Kenton Lee, Ming-Wes Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes no questions. Xiv preprint ar Aixv::1905. 10044, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning 2018.
challenge. vrxii::803.05444
Róbert Csordás, Kazuki Irie, and Jürgen Schmidhuber. Approximating two-layer feedfor- ward networks for efficient transformers. arviv preprint ar Xio:2310.10837, 2023.
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah Smith, and Matt Gardner. A dataset of information-seeeing questions and answers anchored in research papers. Xrv preprint ar Xir:2205503001 2021.
Lucio Dery, Steven Kolawole, Jean-Francois Kagey, Virginia Smith, Graham Neubig, and Ameet Talwalkar. Everybody prune now: Structured pruning of Ilms with only forward passes. ar Xiv preprint XrXiv:2402.05406 2024.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. Xiv preprint "Xix:2208.07333 2022.
Harry Dong, Beidi Chen, and Yuejie Chi. Towards structured sparsity in transformers for
efficient inference. In Workshop on Efficient Systems for Foundation Models@ ICML2023, 2023.
Kayvon Fatahalian, Jeremy Sugerman, and Pat Hanrahan. Understanding the efficiency of gpu algorithms for matrix-matrix multiplication. In Proceedings of the ACM SIG- GRAPHIEURRCRRPPICC conference Graphics hardware,, pp. 133-137, 2004.
10

---

Published as a conference paper at COLM 2024
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1)5232--2270 2022.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, train- able neural networks. arXiv preprint arXit:1803.0065: 2018.
Elias Frantar and Dan Alistarh. Massive language models can be accurately pruned in one-shot. Xiv preprint ar XXXt:2301.00774, 2023.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of 2020.
diverse text for language modeling. ar arv preprint ar arXiv:2101.0000
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noachh, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL
10256836
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.14913, 2020.. 2020.
Robert Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Ajay Kumar Jaiswal, Shiwei Liu, Tianlong Chen, Ying Ding, and Zhangyang Wang. Instant soup: Cheap pruning ensembles in single pass can draw lottery tickets from large models. In International Conference on Machine Learning, pp. 14691-14701. PMLR, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arv preprint ar (rXin:2310.06855, 2023.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, LÂOlio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, ThAGophile Gervet, Thibaut Lavril, Thomas Wang, TimothA@e Lacroix, and William El Sayed. Mixtral of experts, 2024.
Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM computing surveys (CSUR), 54(10s):1-41, 2022.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Losparse: Structured compression of large language models based on low-rank and sparse approximation. Xiv preprint ar Xrx:x:2306.1 0eti2223...12,,,,,. 2023.
Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J Reddi, Ke Ye, Felix Chern, Felix Yu, Ruiqi Guo, al. The lazy neuron phenomenon: On emergence of activation sparsity in transformers. In The Eleventh International Conference on Learning Representations, 2022.
Tailin Liang, John Glossner, Lei Wang, Shaobo Shi, and Xiaotong Zhang. Pruning and quantization for deep neural network acceleration: A survey. Neurocomputing, 461:370- 403, 2021.
11

---

Published as a conference paper at COLM 2024
Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. A survey of transformers. Al Open, 2022.
Bohan Liu, Zijie Zhang, Peixiong He, Zhensen Wang, Yang Xiao, Ruimeng Ye, Yang Zhou, Wei-Shinn Ku, and Bo Hui. A survey of lottery ticket hypothesis. arXiv preprint arXft:2403.08661, 2024.
Zejian Liu, Fanrong Li, Gang Li, and Jian Cheng. Ebert: Efficient bert inference with dynamic structured pruning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 4814 48233, 2021.
Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja vu: Contextual sparsity for efficient ilms at inference time. In International Conference on Machine Learning, pp. 22137-22176.. PMLR, 2023.
Xinyin Ma, Gongfan F Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702-21720, 2023.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Iman Mirzadeh, Keivan Alizadeh, Sachin Mehta, Carlo C Del Mundo, Oncel Tuzel, Golnoosh Samei, Mohammad Rastegari, and Mehrdad Farajtabar. Relu strikes back: Exploiting activation sparsity in large language models. ar Xiv preprint ar arXiv:2310.04:: 2023.
Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. Abstractive text sum- marization using ggquence-to-eqqenne rnns and beyond. arXiv preprint 1ixx:1602.0600) 2016.
Shashi Narayan, Shay В Cohen, and Mirella Lapata. Don't give the details, just the summary! topic-aware convolutional neural networks for extreme summarization. Xiv preprint ar Xiix:1808.08745 2018.
Subhash Nerella, Sabyasachi Bandyopadhyay, Jiaqing Zhang, Miguel Contreras, Scott Siegel, Aysegul Bumin, Brandon Silva, Jessica Sena, Benjamin Shickel, Azra Bihorac, et al. Transformers in healthcare: A survey. Xiv preprint ar arXiv:2307.000 700067, 2023.
Mikolaj Piórczyński, Filip Szatkowski, Klaudia Balazy, and Bartosz Wójeik. Exploiting transformer activation sparsity with dynamic inference. arv preprint ar ar11:22310. 2023.
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P' Lillicrap. Compressive transformers for range sequence modelling. arXiv preprint, 2019. URL httpss//aaxiv.org//b///1111 05507
Siva Reddy, Danqi Chen, and Christopher Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266, 2019.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series, 2011.
Michael Santacroce, Zixin Wen, Yelong Shen, and Yuanzhi Li. What matters in the structured pruning of generative language models? arXiv preprint arXit:2302.03773, 2023.
Uri Shaham, Elad Segal, Maor lvgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, et al. Scrolls: Standardized comparison over long language sequences. Xiv preprint arXit:2201.03533 003533,,, 2022.
Noam Shazeer. Glu variants improve transformer. Xiv preprint arXim:2002.05202,222 2020.
12

---

Published as a conference paper at COLM 2024
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. arxio preprint (xXi::300..11695, 2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-aatttss Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. ar preprint ar axXio:2312.11805, 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXio preprint arXiv:2403.08295, 2024.
SpaseLLM Team. Sparse large language models with relu activation, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama Open foundation and fine-tuned chat models. arXiv preprint ArXi::2307 (tt::2307009288,, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. Adnances newral informa- tion processing systems, 30, 2017.
Yu Emma Wang, Gu-Yeon Wei, and David Brooks. Benchmarking tpu, gpu, and cpu platforms for deep learning. ar Xrv preprint 1rXt::1907.10701, 2019.
Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruning learns compact and accurate models. arXiv preprint arXiv:2204.000 00408, 2022.
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning, Ar preprint ar #rXie23300066994 2023,
Varun Yerram, Chong You, Srinadh Bhojanapalli, Sanjiv Kumar, Prateek Jain, Praneeth Netrapalli, et al. Hire: High recall approximate top-k estimation for efficient 11m inference. arXivv preprint ar Xxx::2402.09360, 2024.
Lu Yin, Ajay Jaiswal, Shiwei Liu, Souvik Kundu, and Zhangyang Wang. Pruning small pre-trained weights irreversibly and monotonically impairs "difficult" downstream tasks in Ilms, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv:1905.07880, 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Moefication: Transformer feed-forward layers are mixtures of experts. arXiv preprint arXi1:2110.01186, 2021.
Haizhong Zheng, Xiaoyan Bai, Beidi Chen, Fan Lai, and Atul Prakash. Learn to be efficient: Build structured sparsity in large language models. arv preprint dxXiv:2402.06126, 2024.
13

---

Published as a conference paper at COLM 2024
A
Pruning Metric
Here we present visualizations of our gating statistic from 6 For single sample, we find and sort the entries normalized between and 1 in Figure In both models, values in are heavily concentrated in a handful of features. Since aggregates the relative activation magnitudes across tokens, this implies s can capture heavily and frequently activated neurons.
Llama 7B
Gemma 7B
Normalized Gating Metric
0.8
0.8
Normalized Gating
0.6
0.6
0.0
4000
6000
B000
10000
5000
10000
15000
20000
25000
Neuron
Neuron
Figure 6: Sorted entries of for each layer of Llama 2 7B (left) and Gemma 7B (right) with sequence from PG-19 as the input. Each line is layer.
B Sampling-based Selection
Here, we verify that given the statistic s, top-k expert selection produces better results than sampling-based methods. The methods compare against include sampling based the weights in s and combining top-k selection for half of the experts followed by weighted sampling. Based on Table 5] we can that sampling generally degrades performance much more.
Table 5: Comparison between different expert selection methods at 50% FF sparsity.
SELECTION METHOD
XSUM (ROUGE-1/2/L)
CNN/DAILYMAIL (ROUGE-1/2/L)
CoQA (F1/EM)
QASPER (F1)
Liama 27B
FULL
27.15/9.06/22.66
10.08/0.13/9.55
77.35/63.88
26.31
TOP-k
24.75/7.41/2055
10.97/0.66/10.37
77.18/63.58
25.76
SAMPLING
21.04/5.22/17.12
8.78/0.49/8.0 /8.28
76.15/62.53
24.46
TOP-k SAMPLING
24.35/7.08/20.mm
10.45/0.48/9.88
77.17
25.22
Gemma 7B FULL
26.86/9.15/22.03
17.45/4.15/15.0
79.04/65.25
30.78
TOP-k
25.86/7.81/20.93
18.26/4.75/16.58
78.52/64.62
27.37
SAMPLING
00.25/5.16/16.77
8.34/1.71/7.72
75.02/59.93
24.97
TOP-k + SAMPLING
24.47/7.43/19998
10.93/2.60/9.98
76.76/62.12
27.09
C Sparsity in Random Sequences
As further exploration into flocking, we investigate this phenomenon with random inputs. As input sequences,
sample from concatenated WikiText, permuted version of that sample, and completely random sequence where tokens are uniformly sampled from the vocabulary. Seen in Figure] 7 this structure exists in permuted and random inputs,
14

---

Published as a conference paper at COLM 2024
perhaps even more consistently than in unperturbed sequences. This suggests something within language actually diversifies the activations, the cause of which would be of interest for future work.
Original
Permuted
Random
0.08
0.07
100
100
100
200
Toxer Indeex
200
Toieen Indox
200
Toxen
0.04
200
300
200
0.03
400
400
400
0.02
500
500
100
200
300
400
500
100
200
300
400
500
100
200
300
400
500
Neuron Index
Nouron Index
Neuron Index
Figure 7: First 512 tokens and their relative FF activation magnitudes in layer 18 of Gemma 7B when inputting the original WikiText sequence (left), permuted sequence (center), and random tokens (right). Best viewed zoomed in.
D Generation Examples
We show example generated text in Figure g similar summaries compared to the those produced by the full models and the target.
Adaptive Wanda and GRIFFIN produce
E More Flocking Examples
We provide more example of flocking across different layers of the LLM. Figure and Figure show flocking in Gemma 7B. Figure I1 and Figure show flocking in Llama 2 7B. Flocking in Gemma 7B more visually distinct while activations Llama 7B are more distributed.
15

---

Published as a conference paper at COLM 2024
Prompt
<shot
#hit
Article: The systems, at Kentucky Methodist Hospimal Chino Valley Medical Center and Desert Valley Hospital, California, are now running normllly again.
None of the hospitals is believed to have paid the ransom. And
now being investigated by the FBI.
The Kentucky Methodist Hospital had to shut down all of its desktop computers and activatc back-up system. message
homepage said: "Methodist Hospital currently working in internal state emergency computer that limiieed our use of clectronic web-based services.
"We curreatly working resolve this It later said no patient data or cure had heen affected. Fred Ortega, spokesman significant disruptions of our IT systems.
limited access bused
electronic communications:
Services, which Chino Valley Medical Center and Valley Hospiitll said: cause
"However, most systems
oritical infrastructure
brought online."
The altack comes woeks after it was revealed Hollywood Presbyterian Medical Contre Los Augeles had boca attacked ransonware.. that case, paid 7,000 get access Kentucky Mathodist Hospital information systems director Jamic Reid nauned the malware involved Locky, new bug that enerypts files, documents and images and renames them with the extension. Jocky. The most common way Locky gets itsctf on machines vin spam cmail with attached documeur that looks like nonsense and advises readers caable macros 'if' the data encoding is incorrect".
back.
Once the malware downlsaded, sends message desktops with instructions about how In Neverbeer, report from Intel's McAlee labs said the number of runsonware altacks was expected to grow in 2016. Security exper. Drian Krebs said: "It's fair that "I also warry that these more deliberate attnckers will take bit more time to discer how much the data they've enerypted really worth, and preciscly how much the victim might be willing pay get back." "
pay have files unlocked.
allacks and atlackers mature, these schemnes will slowly becomne more turgeted.
Stummarize the above article in 1 sentences.
Target
The IT' systems of three US bospitals have been infected with ransomware, which encrypts vital files and demands money to unlock them.
Llama 2 7B: Full
Hospitals in Cališamia and Kentucky have boen hit ransmmware attack.
Llama 2 7B: Adaptive Wanda (50%)
Three bospitals in the US have boen hit by runsonwwre.
Llama 2 7B: GRIFFIN (50%)
Three US bospitals by ransomware, which has affected their computer systems.
Gemma 7B: Full
computer
hospitals US, causing down their systems
back up servers.
Gemma 7B: Adaptive Wanda (50%)
Three hospitals the US have heen hit
attacks.
Gemma 7B: GRIFFIN (50%)
Hoqpitals in the US have been hit by ransomwaaee attack, which has encrypted their files and demanded a munsom
Figure 8: Generation examples by Llama 2 7B and Gemma 7B, given a prompt BBC 2016 from XSum (1-shot).
16

---

Published as a conference paper at COLM 2024
Thken Index
202
Tolan brddas
200
-00-
300
-000
300
400
402
400
400
503
Nburrn indic
Niurrn Incio
Nouron Indox
Ncuron Indox Layer
Layer 5
Layer 6
Layer
103
100
Thom Indee
Thean Indioc
202
200
400
503
500
300
100
200
300
400
500
100
200
300
400
500
100
200
3L0
500
100
200
3:0
400
503
Layer
Layer
Laver
102
100
300
Thom Inoe/neee Iode
203
Teaat Indlac
220
200
303
-000
200
407
400
502
500
200
500
300
400
500
100
200
300
400
Neuron Incees
Niurrcn Inoms
Neuron Indrx
Neturnn Indrt Layer
Layer 33
Layer
Laver 15
302
300
Tran Inter
203
Train Indde
403
400
500
500
500
500
200
100
300
400
500
200
350
330
400
37
Layer
140
Layyer
302
200
200
Toden mose
200
203
Touam Iocdae
200
200
0.00
-00-
303
300
300
0.0g
402
400
100
700
100
100
2000
300
400
500
100
200
300
100
202
320
450
500
0.00
Nearon Inckes
Neurion Inoux
Neuron Indiax
Nicuton Indix
Figure 9: First 512 tokens and their relative FF activation magnitudes in layers 1 to 20 of Gemma 7B when inputting a sequence from PG-19.
17

---

Published as a conference paper at COLM 2024
Layer 21
Layer
Layer 23
Layer
103
100
Tham Thamnrdee Ingeo
Takan Indio
202
200
503
500
100
220
200
400
500
100
200
200
400
500
300
200
300
400
500
200
205
320
400
303
Indiex
Indix
25
Layer
27
Layer
302
200
Thom mose
200
203
To-ott Indaue
1200
200
-200
203
-200
200
0.00
180
700
1.00
140
30000
409
520
100
200
350
100
330
400
5.0
0.00
Nouron Incies
Nourin Index
Neuron Indiax
Necurnn Indix
Figure 10: First 512 tokens and their relative FF activation magnitudes in layers 21 to 28 of Gemma 7B when inputting a sequence from PG-19.
Layer
Layer
Layer
100
302
Toien moee
300
200
Thieen Inddet
-.00
200
-00-
200
-200
300
400
400
200
300
100
400
500
100
200
300
300
400
500
Nourtnn Ircie
Nauron Inclax
Neuron Indixx
Nouron Indiox Layer
Layer
Layer
Layer
100
100
T04on In:04
200-
Toea Indine
200
400
400
500
500
508
505
100
200
300
400
500
100
200
300
400
500
100
200
300
400
500
100
200
300
400
500
Layer
Layer
Layer
11
Layer
100
200
721o1 mnde
300
200
Talee Inder
200
200
300
200
100
200
400
400
500
200
300
100
360
100
200
300
400
500
Neuron index
Netron Incix
Neuron Indrex
Neuron Indrex Layyer
Leyer 33
Layer
Layer 35
0.10
100
0.08
1000n Ind0e
Toben Indee
0.00
400
400
502
503
520-
100
200
300
400
100
200
300
400
560
100
200
300
400
1.0
200
300
400
500
Figure 11: First 512 tokens and their relative FF activation magnitudes in layers 1 to 16 of Llama 2 7B when inputting a sequence from PG-19.
18

---

Published as a conference paper at COLM 2024
Layer
Layer
Leyer
T30on Index
Thlom Inddex
302
320
402
400
400
430
500
300
340
500-
230
Neuron Index
Index
Leyer 21
Layer 22
Layer 23
Lever 24
1.00
100
100
100
Toton I0d0x
200
Thbem leddee
1000
200
300
300
100
200
300
400
500
100
250
300
400
500
100
200
300
400
500
100
200
300
400
500
Neurrn index
Neturnn index
Neuron Indexx
Neuror Index
10:0n Index
200
Tolen Inddrt
200
200
-000
300
400
400
400
405
500
500
500
Naturon
Natrron Indie
Nauron Indixx
Neuron Inditx
Layer 29
Laver 30
Laver 31.
Laver 32
0.19
100
100
100
100
0.08
100m moon hode
300
Toochh Incion
200
200
0.00
300
302
0.01
0.02
103
300 Neu Incex
220
400
520
100
250
300
400
560
300
200
300
400
500
100
200
350
400
500
0.00
Neuron Index
Neuron Index
Neuron Indeex
Figure 12: First 512 tokens and their relative FF activation magnitudes in layers 17 to 32 of Llama 2 7B when inputting a sequence from PG-19.
19


Reviews:
[{"review_id": "PMBzQwUE82", "reviewer": "Reviewer_cGRh", "summary": "This paper introduces GRIFFIN, an interesting MOE method, which leverages special activation patterns in Feedforward layers for pruning. The key feature of GRIFFIN lies in its training-free design for selecting unique FF experts at the sequence levels. Besides,  this method is adaptable to non-ReLU methods, which makes it practical in applications, given that many popular LLMs use other activations like SwiGLU. Experimental results demonstrate that, even with only 50% of FF parameters, an MOE LLM employing GRIFFIN can still maintain performance comparable to dense models.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and curiosity in our baselines! We address your comments below. Please let us know if you have any follow-up questions.\n\nAs a new baseline, we use Wanda [2] during generation. This baseline is stronger than the magnitude selection method and is well-known in the pruning community. In more detail, the full model is still used during the prompt phase while FF weights selected by Wanda using the current inputs’ features are used during generation (Adaptive Wanda). Comparisons between GRIFFIN and Adaptive Wanda can be found in Table R1.\n\nTable R1: Adaptive Wanda and GRIFFIN on XSum (1-shot), CNN/DailyMail (1-shot), CoQA (0-shot), SCROLLS QASPER (0-shot) at 50% FF sparsity. \n\n|Model|Method|XSum (Rouge-1/2/L)|CNN/DM (Rouge-1/2/L)|CoQA (F1/EM)|QASPER (F1)|\n|-|-|-|-|-|-|\n|Llama 2 7B|Full|27.15/9.06/22.62|10.08/0.13/9.55|77.35/63.88|26.31|\n||Adaptive Wanda|**25.59/8.18/21.34**|9.90/0.26/9.39|77.16/63.53|**27.61**|\n||GRIFFIN|24.75/7.41/20.55|**10.97/0.66/10.37**|**77.18/63.58**|25.76|\nGemma 7B|Full|26.86/9.15/22.03|17.45/4.15/15.94|79.04/65.25|30.78|\n||Adaptive Wanda|24.76/7.79/20.39|12.10/2.21/11.20|75.79/61.87|**30.62**|\n||GRIFFIN|**25.86/7.81/20.93**|**18.26/4.75/16.58**|**78.52/64.62**|27.37|\n\nWe observe that GRIFFIN and Adaptive Wanda are both great at preserving the original model’s performance. However, we note that Wanda’s pruning algorithm is unstructured, meaning it cannot be easily translated to real latency improvements. Meanwhile, GRIFFIN is able to be performative despite the structural constraint we imposed. We will add this baseline (including its performance with other models) in our next paper version.\n\nFollowing up on your comments on Deja Vu [1], we recognize its ability to dynamically select neurons during inference, but we do not think it is comparable. A major reason is that Deja Vu requires data curation, training, and extra parameters, so compared to GRIFFIN (which requires no preparation), there is a significant gap in resource requirements. Unlike Deja Vu, our new baseline, Adaptive Wanda, selects FF neurons depending on the input without training or data curation and exhibits competitive performance.\n\n[1] Liu, Z. et al, Deja vu: Contextual sparsity for efficient llms at inference time, 2023.\n\n[2] Sun, M., Liu, Z., Bair, A., & Kolter, J. Z, A simple and effective pruning approach for large language models, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method stands out for its training-free design, a practical advantage particularly beneficial for large-scale LLMs. The discovery of fixed patterns within Feedforward Neural Networks (FFNs) is an intriguing observation, evident across various LLM architectures such as LLaMA and Gemma, with activations like SwiGLU and GEGLU. This consistency underscores the potential compressibility of FFN layers. \n\n2. GRINFFIN's simplicity is notable, as it relies solely on the L2-norm of activations during runtime. This streamlined approach not only enhances computational efficiency but also simplifies the implementation. Moreover, the selected experts, i.e., the pruned layers identified by GRINFFIN, can be seamlessly integrated into the decoding process for generating future tokens.", "reasons_to_reject": "The main limitation of this paper is the absence of baseline methods in the experimental section. For instance, Dejavu[1] introduces a contextual structured sparsity approach to dynamically prune layers during runtime. This parallels the strategy adopted in this paper, which also focuses on adaptively pruning Feedforward Neural Networks (FFNs) for different inputs. However, the experiments presented in this submission solely involve magnitude pruning, which may not offer a robust baseline compared to more sophisticated methods like Dejavu. Integrating diverse baseline approaches could provide a more comprehensive assessment of the proposed method's performance and elucidate its relative strengths and weaknesses in comparison to existing techniques.", "questions_to_authors": "Please refer to the \"Reasons to Reject\"", "extra_scores": {}}, {"review_id": "7iT9fhIHT7", "reviewer": "Reviewer_yRHP", "summary": "This paper discusses the phenomena of “flocking”, where FF activations in dense LLMs tend to concentrate by column within the same sequence, and proposes GRIFFIN, a novel technique that selects subsets of FFN parameters per sample based on flocking pattern on the prompt to achieve inference speed up without much loss of quality. The proposed method is training-free, empirically applicable to a list of LLMs, and simple to implement.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our submission and highlighting some key takeaways! We address your concerns/questions below. Please let us know of any lingering questions and feedback.\n\nConcerns:\n1) We can include similarity plots between tokens’ relative activations. Since we are unable to share images for the rebuttal, we roughly describe them. Using the same sample in Figure 1, many Gemma 7B layers had high cosine similarity (0.6 to 0.8) between tokens’ activations. Most of Llama 2 7B’s hovered around 0.35 to 0.4.\n2) We can add example generation outputs. For 1-shot CNN/DM, Mistral 7B and Llama 2 13B often produced a sequence of “\\n” or repeated items from the example shot. With 3-shot, this is not an issue (>25 Rouge-1 for both).\n3) We will also include some example generation outputs from GRIFFIN and the full model in the next version.\n4) Thanks for bringing this to our attention. We will reframe our work as dynamic structured pruning in our next paper version.\n\nQuestions:\n1) We can elaborate on this in the next paper version. For example, the MoEfication algorithm [3] fine tunes non-ReLU activations to ReLUs before constructing MoEs, but it is not the only one to require exact sparsity in FF activations [4, 5]. In turn, these have motivated fine tuning LLMs to have ReLU activations [6, 7].\n2) Row vectors in Figure 3 are approximately l2 unit vectors. For illustration purposes, we rounded to one decimal place, inducing slight rounding errors.\n3) See #2 in “Concerns”.\n4) In our evaluations with XSum and CNN/DailyMail, the example shot and test samples are very different, yet GRIFFIN still appears to do well. The cause of flocking is of great interest to us. Flocking occurs in many different tasks–Figure 6 shows that it even occurs with nonsense inputs. Greater analysis in flocking in terms of cause and interpretability is something we are excited to work on in the future!\n5) Thank you for catching this typo! We will correct it in the next version.\n\n[3] Zhang, Z., et al., Moefication: Transformer feed-forward layers are mixtures of experts, 2021.\n\n[4] Csordás, R., Irie, K., & Schmidhuber, J., Approximating two-layer feedforward networks for efficient transformers, 2023.\n\n[5] Li, Z. et al., The lazy neuron phenomenon: On emergence of activation sparsity in transformers, 2022.\n\n[6] Mirzadeh, I. et al., Relu strikes back: Exploiting activation sparsity in large language models, 2023.\n\n[7] SpaseLLM Team, Sparse large language models with relu activation, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The algorithm this paper proposes is well-motivated and empirically strong. GRIFFIN tackles the important problem of efficient inference, achieving a decent boost in latency without costly retraining, and is empirically demonstrated to be widely applicable.\n\nThis paper also highlights the flocking behavior and shows that it is possible to exploit such inherent structure for practical speed up. I can see this work spark future ideas along this topic.", "reasons_to_reject": "I don’t find major reasons to reject the paper. Some lesser issues I have with the paper are\n\n1. There is not much quantitative characterization of the flocking behavior. The paper shows many examples of flocking, but it’d be good to have some kind of quantitative measurement of this behavior at scale, to demonstrate how ubiquitous is the flocking behavior \"in the wild\".\n2. ROUGE scores for Llama-13B and Mistral-7B on CNN-DM seems too low without any explanation (also Llama-13B < Llama 7B). This brings a bit doubt to the experiment results. It’d be good to provide some qualitative generation examples.\n3. In tandem to #2 - it’d be great to provide generation samples with GRIFFIN, as ROUGE-based evals are n-gram-overlap based and may hide problems. It’d be good to rule out possibilities such as generating incoherent yet overlapping phrases, etc., by showing some qualitative samples.\n4. A pedantic point: to posit GRIFFIN as an approach to convert a dense model to MoE seems a bit of a stretch of concept. GRIFFIN seems to have much more affinity to adaptive pruning or adaptive (structured) sparsity rather than MoE.", "questions_to_authors": "1. By end of page 1 - “Perhaps an even bigger weakness of many of these methods is the inability to effectively carry over to pre-trained LLMs with non-ReLU activations.” - this is not evident to me, could you elaborate, or if this is from previous studies, cite the study?\n2. Figure 3 - Rows of Z_bar are not unit vectors. Is this a mistake in the figure?\n3. Table 2 - Llama 2 13B is significantly worse than 7B in CNN-DM, and 2.5 pts of ROUGE-1 is pretty much no overlap. Mistral 7B also sees this issue. What is happening for these experiments? What do the models generate?\n4. The observation that flocking patterns are shared within 1 sequence and differ across sequences begs one question - what happens when you take 2 sequences that have distinct patterns and concat them together? will it create problems for GRIFFIN? the meta-question here is what causes patterns to stay stable within sequence, and will there be naturally occurring data that breaks this assumption (therefore potentially make GRIFFIN fail on them)?\n5. Section 4.2 right before eqn 6 - “...we do this by aggregating taking…” → \"...we do this by aggregating...\"?", "extra_scores": {}}, {"review_id": "63QMyUw6o1", "reviewer": "Reviewer_WEew", "summary": "The article introduces a new, training-free Mixture of Experts (MoE) method called GRIFFIN. This method capitalizes on a phenomenon observed in large language models (LLMs) known as \"flocking,\" where specific feed-forward (FF) neurons exhibit consistently higher activation relative to others within sequences. GRIFFIN leverages these observations to selectively activate the most relevant FF neurons for efficient processing, thereby maintaining model performance while reducing computational demands.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review of our work, especially on real world considerations for deployment!\n\nBased on your suggestions, we performed experiments on batching with highly promising results. Within a batch, we sum the selection statistic $\\mathbf s_i$ of each sample, scaled inversely by each sample’s root  prompt length, $S_i$, i.e. $\\mathbf{s} = \\sum_{i} \\mathbf{s}_i \\sqrt{S_i}$. Then, the same experts for all samples within a batch would be selected based on $\\mathbf{s}$. Results in Table R2 convey that GRIFFIN is also robust to batching, as performance degrades very slowly as we increase the batch size. As such, while GRIFFIN was especially designed for single sample use cases (e.g. personal devices), our method scales impressively well to large batch sizes.\n\nTable R2: 1-shot XSum Rouge-1 scores for increasing batch sizes. For each batch, the same 50% of FF expert neurons are selected.\n\n|Batch Size|1|2|4|8|16|32|64|\n|-|-|-|-|-|-|-|-|\n|Llama 2 7B|24.75|24.05|24.26|24.04|23.75|23.61|23.49|\n|Gemma 7B|25.86|25.55|25.01|24.20|24.37|23.32|23.12|\n\nIn the next version of our paper, we will include the results in Table R2 in the form of a graph. Let us know if you have any further questions and comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- GRIFFIN improves computational efficiency by reducing the number of active neurons during the model's forward pass, which lowers memory usage and speeds up processing without extensive hardware requirements.\n\n- Despite reducing the number of active parameters in FFN by up to 50%, GRIFFIN maintains near-original performance levels across multiple tasks, which is a significant achievement compared to other sparsity-inducing methods.\n\n- Unlike traditional MoEs that require retraining or fine-tuning, GRIFFIN operates without any additional training, making it easy to implement with pre-trained models.", "reasons_to_reject": "Scalability Limitation Due to Batch-Size Constraint: GRIFFIN's current implementation is constrained to a batch size of one, which limits the model's ability to leverage parallel processing capabilities inherent in modern computational hardware. While a batch size of one achieves a modest speed-up of approximately 1.2x, this benefit does not scale linearly with larger batch sizes. This is due to the unique 'flocking' patterns observed in individual samples, which may vary significantly across different inputs within a batch. This variability makes it challenging to maintain consistent efficiency gains when processing multiple samples simultaneously. Such constraints on batch size could restrict the method's utility in real world model severing.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "agy3RAJjcD", "reviewer": "Reviewer_6mFc", "summary": "This paper introduces an approach for dynamically pruning the FF of Transformer LLMs that doesn't require any extra training. The structured pruning is conditioned on the prompt. When computing the prompt the regular full model is used. Then, based on the prompt activations, only the top-k rows of each FF across all layers are being kept. The generation is performed with this pruned network. So essentially the  pruning is dynamic at the sequence level. The authors present motivation for this approach by describing a behavior they call \"flocking\" where the activated neurons are mostly similar within a sequence but vary across different sequnces (studied on open source 7B and 13B models). They show that they can prune around 50% of FF for generation and mostly preserve classification and generation quality leading to about 1.25X speedup in LLama 13B.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed feedback! We address each concern/question below.\n\nConcerns:\n1) Thanks for pointing this out, we can reframe the paper as a dynamic structured pruning method in our next version. \n2) We designed a new baseline inspired by Wanda [2], a popular pruning method. The full model is used for the prompt while FF weights selected by Wanda based on the input’s features are used during generation (Adaptive Wanda). Comparisons between GRIFFIN and Adaptive Wanda are found in Table R1 under Reviewer cGRh. Both perform similar to the full model, but GRIFFIN has the added benefit of highly structured pruning whereas Wanda is unstructured.\n3) Sampling-based methods show greedy top-k is best and are additional baselines. Fixing neurons across a dataset is interesting! In Table R3, we test two ways to select fixed FF neurons for generation which we will add to our next version. GRIFFIN is more performative in most cases, implying a need for adaptation.\n\nTable R3: 1-shot XSum and CNN/DailyMail Rouge-1 with fixed expert neurons at 50% sparsity. Shot Top-k selects FF experts based on $\\mathbf{s}$ in the example shot. Global Top-k selects FF experts based on the average $\\mathbf{s}$ across a dataset’s prompts.\n|Model|Method|XSum|CNN/DM|\n|-|-|-|-|\n|Llama 2 7B|Full|27.15|10.08\n||Shot Top-k|21.11|10.30\n||Global Top-k|23.28|10.84\n||GRIFFIN|24.75|10.97\n|Gemma 7B|Full|26.86|17.45\n||Shot Top-k|19.76|15.59\n||Global Top-k|22.48|20.96\n||GRIFFIN|25.86|18.26\n\t\n4) A longer prompt is unlikely to harm efficiency much: Table 3 shows latency is dominated by generation, e.g. Llama 2 7B spends ~8.5% of the total latency on prompts. \n5) GRIFFIN applies to FF blocks because it uniquely has flocking. This also makes it versatile as other efficient techniques can be independently deployed in attention (e.g. KV compression) and embedding layers that could further accelerate inference.\n\nQuestions:\n1) Relative FF activations are FF activations scaled down to unit vectors. If z is the vector of FF activations of a token, relative FF activations of that token is z / ||z||_2. \n2) We noticed this too and initially played around with some simple ways to vary k across layers (e.g. skipping some layers and linear scheduling), but we did not notice much effect on performance. In future work, it would be interesting to explore more ways to choose k, for different layers and different samples.\n\n[2] Sun, M. et al, A simple and effective pruning approach for large language models, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Simple but effective approach that doesn't require any extra training\n2. Approach seem to work on several different LLMs and datasets\n3. doesn't require any training so should be very easy to try on any open model", "reasons_to_reject": "1. The writing and framing needs a bit improvement. The title says MoE but then the paper goes back and forth between MoE and pruning. The actual method is doing dynamic structured pruning. I don't think MoEs should be mentioned anywhere (maybe other than related work) and certainly not in the title since its confusing the reader.\n2. Given that the method is actually pruning, I think the main weakness of the paper is lack of comparison to more pruning methods other than the presented naive magnitude approach.\n3. The ablations part can also be improved. I am not sure what is the point of the sampling-based selection experiment. Instead, it will be interesting to see more ablations on the dynamic requirement for example if you take top-k when averaging activations over the full dataset and then keeping it static.\n4. The method seems to work well on short generations but on longer ones the performance seems to substantially degrade. The authors suggest to use a long prompt for alleviating this but this defeats the efficiency purpose.\n5. There are some more limitations to this method like having to select columns or rows for structural pruning rather than selecting from all neurons, and overall the speedup potential is limited with this pruning as even 50% parameter reduction of FF leads to small speedup as observed in experiments.", "questions_to_authors": "1. Can you define exactly what you mean by relative FF activation?\n2. It seems by the heatmaps that maybe the distribution is slightly different across layers. Have you tried using different k across layers?", "extra_scores": {}}]

Author rebuttal:

Meta-review: Ngl this is an interesting paper. but the framing of this paper is quite haphazard. 

Reviewers generally like this paper and recommend acceptance. I am leaning accept too. However, i really think the framing needs to be less confusing and more clear. This is a fundamentally weight sparsity and pruning paper and i think the connection with MoE can be a little distracting. I think this paper would benefit from a more direct framing about what it actually is. Conceptually, drawing in MoE in the narrative makes the paper hard to parse. 

I would in fact start the paper by explaining and coining the term flocking and explaining why griffin leverages it. There are too many random plot points happening in this paper.

In short, i recommend acceptance but authors could benefit from cleaning up the story a bit. this paper is trying too hard. a more down to earth narrative that is simple will likely make more impact.

[comments from the PCs] We strongly recommend following the AC advice to improve the paper.
Decision: Accept