Prompt-prompted Adaptive Structured Pruning for Efficient LLM Generation

Authors: harry  dong
beidi  chen
yuejie  chi
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-23 03:32:00

Abstract: With the development of transformer-based large language models (LLMs), they have been applied to many fields due to their remarkable utility, but this comes at a considerable computational cost at deployment. Fortunately, some methods such as pruning or constructing a mixture of experts (MoE) aim at exploiting sparsity in transformer feedforward (FF) blocks to gain boosts in speed and reduction in memory requirements. However, these techniques can be very costly and inflexible in practice, as they often require training or are restricted to specific types of architectures. To address this, we introduce GRIFFIN, a novel training-free and calibration-free method that selects unique FF experts at the sequence level for efficient generation across a plethora of LLMs with different non-ReLU activation functions. This is possible due to a critical observation that many trained LLMs naturally produce highly structured FF activation patterns within a sequence, which we call flocking. Despite our method's simplicity, we show with 50% of the FF parameters, GRIFFIN maintains the original model's performance with little to no degradation on a variety of classification and generation tasks, all while improving latency (e.g. 1.29$\times$ and 1.25$\times$ speed-ups in Gemma 7B and Llama 2 13B, respectively, on an NVIDIA L40). Code is available at https://github.com/hdong920/GRIFFIN.

Main paper: 
 introduction: 
 Transformers (Vaswani et al., 2017) have demonstrated incredible capabilities across a plethora of domains (Lin et al., 2022; Khan et al., 2022; Nerella et al., 2023) . Their large language model (LLM) successors (Touvron et al., 2023; Team et al., 2023; Jiang et al., 2023; 2024; Team et al., 2024; Anthropic, 2024) have pushed the bar higher, but these behemoths have become performative at the price of enormous amounts of computation and memory demands. One significant contributor is the model size itself. Not only is storage an issue, model layers tend to be wide and plenty, slowing down inference. Moreover, given the existence of sparse structures in LLMs, especially in feedforward (FF) blocks (Geva et al., 2020; Dettmers et al., 2022; Li et al., 2022; Liu et al., 2023) , these models waste computation on intermediate features with little to no impact on the final result. For instance, it has been observed that in OPT-175B (Zhang et al., 2022) , fewer than 5% of neurons in FF blocks have nonzero values per token (Liu et al., 2023) , meaning 95% of the compute in each FF block is wasted. Usually consisting of around two-thirds of the parameters in an LLM, FF blocks can be serious memory and compute bottlenecks. These inefficiencies are highly problematic in latency-sensitive scenarios like in chatbots and autonomous vehicles.There have been many methods to exploit sparsity in LLMs for efficiency gains, such as pruning model weights and constructing mixtures of experts (MoEs). Pruning removes low-impact pre-trained weights to reduce storage, yet this often does not translate to real speed-ups in practice, unless the pruning is done in a structured and hardware-friendly manner (Xia et al., 2022; Santacroce et al., 2023; Ma et al., 2023; Li et al., 2023; Xia et al., 2023) which typically causes greater performance deterioration. MoEs better preserve the original performance by adaptively selecting subsets of the model to use per input but also come with drawbacks. Unless the model has been trained in this fashion (Fedus et al., 2022; Jiang et al., 2024) , it will need to learn a cheap yet effective gating function (expert selection mechanism) and sometimes even require full fine tuning. Perhaps an even bigger weakness of many of these methods is the limited effectiveness and special considerations to strictly enforce ReLU-like activation functions (Zhang et al., 2021; Li et al., 2022; Csordás et al., 2023) . In summary, pruning and MoEs provide enormous benefits but also come with steep challenges:1. Structured sparsity is difficult and costly to enforce without serious performance degradation. 2. Adaptive selection of model subsets need to be cheap and accurate. 3. The method should be effective with various activation functions.Daunting at first, these challenges become surmountable because of a simple observation of a phenomenon we call flocking, highly consistent sparse activations that persist throughout a sequence observed in many LLMs. Flocking emerges in FF activations (inputs into the FF down projection) when we focus on a sequence's relative activation magnitudes instead of the absolute values. Examples from Llama 2 7B and Gemma 7B are shown in Figure 1 . The key takeaway is that neurons which produce high relative magnitudes are naturally shared across tokens within a sequence, as seen with the distinct vertical streaks. Increasingly bizarre, the two models have different architectures and different non-ReLU activation functions.Figure 1 : Relative FF activation magnitudes of the first 512 features and tokens across a sequence from PG-19 (Rae et al., 2019; Gao et al., 2020) in layer 10 of Llama 2 7B (left) and Gemma 7B (right). These heatmaps show flocking, where relative activation magnitudes are shared within a sequence, illustrated with the distinct dark vertical streaks. More examples in Appendix E.Unlike existing pruning or MoE methods, we exploit flocking in our design of GRIFFIN (Gating by Repetition In Feedforward Intermediate Neurons), a highly performative and efficient training-free method to adaptively activate FF neurons. GRIFFIN does this by using a sequence's prompt to determine the experts to activate during generation, allowing it to overcome all of the aforementioned challenges:1. No Preparation: Our no-cost method is completely training-free and requires no calibration. Moreover, the simple implementation of GRIFFIN means it can be dropped into any FF block and instantly deployed. 2. Simple Expert Selection: Flocking in the prompt reveals the most relevant FF neurons for generation with little to no performance loss. The selection process is parameter-free and adds negligible overhead. 
background: 
 Our novel method and FF activation observations are inspired and motivated by ample amounts of previous research that sought to characterize FF sparsity and accelerate LLMs.Feedforward Activation Sparsity. The observation that transformer FF blocks produce sparse activations is not new (Geva et al., 2020; Dettmers et al., 2022; Li et al., 2022; Dong et al., 2023; Liu et al., 2023) . In ReLU-based LLMs like OPT (Zhang et al., 2022) , the activations can be exceptionally sparse and become more apparent for larger models (Liu et al., 2023) .As more models use non-sparse activation functions like GLU variants (Shazeer, 2020) , it is difficult for neurons to have no contribution to the output since these functions do not have an interval that maps to zero. Without exact sparsity, the efficacy of these methods becomes limited. As such, this has ushered a wave of models that are either adapted from available models (Zhang et al., 2021; Liu et al., 2021; Mirzadeh et al., 2023; Zheng et al., 2024; Jiang et al., 2024) or trained from scratch (Fedus et al., 2022) which can produce activations that are exactly zero with little to no performance loss. Even so, these methods require considerable amounts of computational resources.Pruning. Pruning (LeCun et al., 1989) is one sparsity-guided way to tackle compute and memory bottlenecks of models. Previously, the common method would be some variation of iteratively rounding weights down to zero based on some score and retraining to recover any lost performance (Frankle & Carbin, 2018; Blalock et al., 2020; Liang et al., 2021; Liu et al., 2024) . While this can result in most parameters being pruned, this method comes with a few issues. First, with the increasing scale of LLMs, retraining becomes impractical for most. Fortunately, cheap methods to effectively prune LLMs have been developed (Frantar & Alistarh, 2023; Sun et al., 2023; Jaiswal et al., 2023; Dery et al., 2024) . The second issue is that unless pruning is done in a structured manner (Xia et al., 2022; Santacroce et al., 2023; Ma et al., 2023; Li et al., 2023; Xia et al., 2023) , it is difficult to see real computational savings, yet structured pruning often leads to much more severe performance degradation. Third, pruning usually enforces sparsity to be static at inference which can be strong assumption since FF blocks are widely believed to contain the model's memory (Geva et al., 2020) .Making sparsity more dynamic has motivated the design of mixtureof-experts (MoEs) (Jacobs et al., 1991) to avoid computing low-impact features in trained models at varying granularities (Zhang et al., 2021; Liu et al., 2023; Piórczy ński et al., 2023; Csordás et al., 2023; Yerram et al., 2024; Zheng et al., 2024) . The main idea of MoEs is to use a gating function to identify a small subset of neurons that will be used to compute the layer's output, an adaptive form of pruning. In the ideal case, all active neurons are selected and inactive neurons are ignored for each input. However, current methods either require training or rely on ReLU or ReLU-like activation functions. 
model & activation function diversity: thorough experimentation demonstrates: 
 the efficacy of GRIFFIN on numerous models, including Llama 2 (Touvron et al., 2023) , Gemma (Team et al., 2024) , Mistral (Jiang et al., 2023) , OPT (Zhang et al., 2022) , and ReluLlama (Team, 2023) . Together, the tested activation functions include ReLU, SwiGLU, GEGLU, and ReGLU (Shazeer, 2020) .In this paper, we show GRIFFIN is a simple and strong adaptive structured pruning method because of flocking. In the next section (Section 2), we discuss in more detail some strengths and weaknesses of current methods that seek to improve FF efficiency. Then, we formalize the problem we are trying to solve, along with its motivation in Section 3. We present our novel approach in Section 4.2, which requires a thorough examination of the surprising phenomenon of flocking shared by many LLMs in Section 4.1. Our rigorous experiments demonstrate GRIFFIN preserves performance on classification and generation even after removing 50% of FF neurons (Section 5.1), all while having lower latency (Section 5.2). For instance, GRIFFIN reduces the number of active parameters in Llama 2 13B from 13B to 8.8B during generation to improve latency by 1.25× with almost no loss in performance. Finally, we show our method's incredible scalability and robustness in several ablation studies (Section 5.3).This section contains an overview of different components of the FF block followed by a formulation of the FF compression problem which our method aims to tackle. Since FF blocks operate identically and independently for each token unlike attention, we begin with defining the FF block with a single column vector input x ∈ R D :EQUATIONwhere FF 2 (z) = W 2 z + b 2 is a linear transformation and FF 1 is nonlinear. This describes a wide range of FF architectures and arbitrary activation functions σ. For instance, in OPT,EQUATION)For FF blocks with GLU variants such as in Llama 2 and Gemma,EQUATIONwhere ⊙ signifies element-wise multiplication. For all examples, W 1 , W g ∈ R D FF ×D and W 2 ∈ R D×D FF where typically, D FF ≫ D. We refer to z = FF 1 (x) as the FF activations.A popular method to compress FF blocks (which is also the goal of GRIFFIN, adaptive neuron pruning, and many MoE methods) is to findW 1 ∈ R k×D , b 1 ∈ R k , and W 2 ∈ R D×k (additionally W g ∈ R k×D and b g ∈ R k if needed)where k < D FF such that when the FF block is reparameterized with these matrices, the output value is preserved. Namely, forEQUATIONEQUATIONFF(x) ≈ FF 2 ( FF 1 (x)), and similarly for FF blocks with non-GLU functions. For instance, in the MoE setting, these smaller matrices can vary from token to token and are actually selections of chunks of the original structures. Crucially, a solution to this problem leads to multiplication with smaller matrices which are naturally more efficient on GPUs and TPUs (Fatahalian et al., 2004; Wang et al., 2019) . Note that for all equations defined in this section, they operate independently on each row of a length S sequence input X ∈ R S×D (e.g. the activations for a sequence are Z = FF 1 (X) ∈ R S×D FF ). 
from flocking to griffin: 
 Here, we take deeper dive into the phenomenon of flocking and describe the intuitive algorithm of GRIFFIN which is directly inspired by it. Flocking arises when we look at the relative impact of each neuron per token within a sequence. To see this, we normalize rows of Z to be unit vectors to constructZ ∈ R S×D FF (i.e. [Z] i = [Z] i /∥[Z] i ∥ 2 ), the relative activations. We show example relative activation magnitudes for a sequence in Llama 2 7B and Gemma 7B in Figure 1 . Since there are distinct vertical streaks, this intriguingly implies that activations that have relatively greater weight are common across all tokens in a sequence. Notably, Llama 2 7B and Gemma 7B use SwiGLU and GEGLU activations, respectively, along with other major architecture differences. We call this phenomenon flocking, like highly organized groups of birds, and we observe this in virtually all FF layers (see Appendix E) and randomized inputs (see Appendix C).While relative activations magnitudes are shared within a sequence, they are not generally shared between sequences. We show this by taking the ℓ 2 -norm of Z along the token axis to obtain a length D FF vector for each sample or sequence, roughly capturing the contribution of each FF neuron throughout a sequence. Taking the top-k of this for each sample at each layer, we find the Jaccard similarity between two sequences based on the indices selected for different k. In other words, we compute the intersection over union of every unique pair of top-k sets. Higher values indicate more similar top-k sets. From Figure 2 where we aggregate Jaccard similarities across WikiText (Merity et al., 2016) samples, we observe a lack of inter-sample activation similarities for the vast majority of layers in Llama 2 7B and Gemma 7B, unless the sets of selected neurons are large. This lack of consistency implies statically pruning entire FF neurons without retraining would be less effective than a more adaptive method.Using our insight on flocking, we introduce GRIFFIN as a simple general purpose and training-free adaptive structured pruning method for efficient generation, captured in Figure 3 . In a nutshell, we select neurons during the prompt phase of each sample which are then used for the entire duration of the generation phase. This effective approach is based on a key observation on flocking: since tokens within a sequence share activation patterns, the prompt and generated tokens will also share activation patterns.Prompt Phase Expert Neuron Selection. Our expert neurons are chosen at the sequence level, so we need to consider the dynamics of the entire input sequence rather than just a single token when choosing our neurons. To select expert neurons, we need a statistic s ∈ R D FF to inform us of the importance of each neuron. At the prompt phase, we do this by taking the ℓ 2 -norm of Z along the token axis:EQUATIONTaking the indices of the top-k across s gives us the neurons we will use for this sample's generation phase which make up the set E . Using the expert neurons in E , we can find W 1 , b 1 , W g , b g , and W 2 by selecting corresponding rows and columns in W 1 , b 1 , W g , b g , and W 2 , respectively. This is done for every FF block during the prompt phase. Illustrated in Appendix A, s highlights neurons consistently activated at relatively high intensities.Generation with Expert Neurons. When generating tokens, we directly use the pruned layers which contain the expert neurons, FF 1 and FF 2 , to estimate FF(X) ≈ FF 2 ( FF 1 (X)) for all future tokens. In Llama 2 13B and Gemma 7B, this reduces the active number of parameters from 13B to 8.8B and from 8.5B to 5.4B, respectively, during generation. 
experiments: 
 We showcase the superb performance of GRIFFIN on numerous tasks and models (Section 5.1) while achieving lower latency (Section 5.2), along with a study on several of its properties like sequence length scaling and batching (Section 5.3). All experiments are run on NVIDIA L40 GPUs.Using various models, we evaluate on several generation and classification tasks. For generation, we evaluate on XSum (Narayan et al., 2018) , CNN/DailyMail (Nallapati et al., 2016) , COQA (Reddy et al., 2019) , and SCROLLS QASPER (Dasigi et al., 2021; Shaham et al., 2022) . For classification, we evaluate on HellaSwag (Zellers et al., 2019) , PIQA (Bisk et al., 2020) , COPA (Roemmele et al., 2011) , ARC-Easy/Challenge (Clark et al., 2018) , and BoolQ (Clark et al., 2019) . With the exception of XSum and CNN/DailyMail, we use LM Evaluation Harness for our experiments (Gao et al., 2023) . Aside from comparing with the original LLM, we also compare GRIFFIN with a static neuron pruning method based on neuron magnitudes. Similar to neuron magnitude pruning, this baseline selects expert Table 2 : Generation tasks XSum (1-shot), CNN/DailyMail (1-shot), CoQA (0-shot), SCROLLS QASPER (0-shot) at 50% FF sparsity. Magnitude neuron pruning fails in almost every case while GRIFFIN effectively preserves performance.neurons based on neuron magnitudes in W 1 for the generation phase but uses the entire FF blocks for prompting like GRIFFIN. In the case of GLU variants, the neuron-wise norms of W 1 and W g are elementwise multiplied to produce the pruning metric. As we will see, this straightforward baseline achieves great classification results but falters for generation.Another baseline we consider also uses the full model for the prompt and Wanda (Sun et al., 2023) in FF blocks for generation, using the prompt activations to prune FF weights. We refer to this adaptive unstructured pruning baseline as Adaptive Wanda. Note that Adaptive Wanda is completely unstructured and does not reduce the FF activation dimension at all.As our method is designed specifically for generation, we alter classification evaluations to simulate generation. In typical classification tasks, LLMs do not enter the generative phase since the final token output of the prompt phase indicates the class. Consequently, directly applying GRIFFIN for classification tasks trivially yields the exact performance of the original model. Therefore, we treat all tokens but the final input token as the prompt.Then, the model is forced to go into the generation phase for one step.We start with a look into the relationship between the sparsity levels and performance degradation. This translates to varying k when we select the top-k of our statistic s. To compare the performance degradation across multiple tasks, we plot the ratio of the final performance metrics between GRIFFIN and the full model in Figure 4 . We see most of the performance is preserved at 50% FF sparsity in Llama 2 7B, Gemma 7B, and Mistral 7B. Different tasks have different tipping points where performance sharply drops, which may be related to the difficulty of the task (Yin et al., 2024) . Fixing FF sparsity to be 50%, we evaluate on more tasks and models. Table 1 and Table 2 show the performance of GRIFFIN on classification and generation, respectively. We see that magnitude neuron pruning achieves reasonable results for classification but completely annihilates the original performance in most generation settings. For generation, Adaptive Wanda and GRIFFIN perform similarly, preserving most of the performance for all models. However, GRIFFIN is able to accomplish this in a structured manner which allows it to be more efficient in practice. For example outputs, see Appendix D. Base models Mistral 7B and Llama 2 13B perform poorly on 1-shot CNN/DailyMail, as they often repeated "\n" or items from the example shot. With 3 shots, both obtain Rouge-1 scores above 25.We now present efficiency metrics of GRIFFIN. We collect synthetic datasets with samples having identical lengths and average results across samples. Like many MoE methods, GRIFFIN is ideal for single sample inputs since it is adaptive, such as in the case of personal devices, so we use batch size 1 for these experiments. Using Hugging Face implementations of Llama 2 13B and Gemma 7B at FP16 precision, we measure the latency in different scenarios on an NVIDIA L40 GPU.Recalling that our magnitude selection baseline is essentially neuron pruning at generation, this has the best possible speed-up since there is no expert neuron selection overhead per sample. From Table 3 , GRIFFIN matches the best case, producing up to a 1.29× and 1.25× improvement in latency for long generation at 50% FF sparsity in Gemma 7B and Llama 2 13B, respectively. This illustrates that our method is as fast as a static neuron pruned LLM during generation while being adaptive to preserve the accuracy of the full model. In offloading settings with large models, our method has the potential to further accelerate inference. For a prompt, GRIFFIN essentially performs structured pruning on the massive network, and if this pruned model can fit on a single device, it will avoid offloading for the entirety of generation.Prompt vs. Generation Length. We find that GRIFFIN can potentially be made more robust for long generation by lengthening the prompt. To see this, we use language modeling on the concatenated version of WikiText to simulate generation. For a length S input into the FF block, we designate the first P tokens as the prompt and the last G tokens as the generated portion such that P + G = S. The prompt partition is used to calculate our statistic s and determine the expert neurons. The prompt partition uses the full FF block while the generation partition only uses the selected neurons. When comparing the original model with GRIFFIN, we only compute the perplexity of the outputs from the generation partition since the other outputs will be identical. Based on Figure 5 , GRIFFIN gets closer to the full model outputs when the prompt length increases and generation length decreases, meaning the difficulty with long generation can be suppressed with longer prompts. Sharing Selected FF Neurons. We further verify that GRIFFIN is simultaneously robust to batching and more performative than fixed neuron pruning methods using similar pruning metrics to s in (6). As such, GRIFFIN maintains its reliability as we extend beyond single inputs.For static pruning, we test two methods. One is to use the example shot in the prompt to compute s. Then, for all samples with the same shot, we select the same FF neurons for generation. With the other method, we can extend (6) such that s can be a pruning metric for multiple inputs (i.e. the same FF neurons will be pruned). Suppose that s i is the metric for sample i with prompt length S i . Then, we useEQUATIONto select FF neurons. Finding a global s across all prompts in a dataset is the second method. Therefore, for a particular dataset, both methods have fixed expert neurons across samples.To extend GRIFFIN to larger batch sizes, we use s from (7) where the samples in a batch are aggregated. Thus, adaptability is maintained. From Table 4 , GRIFFIN (even with batch size >1) achieves better performance than the static methods, reinforcing the performance benefit of adaptability. Interestingly, even though performance decays as we increase the batch size, the decay is slow.  
conclusion: 
 In this work, we have shown a special form of sparsity in FF layers and a simple method to exploit it. Flocking is a curious phenomenon present in many LLMs where tokens within a sequence activate neurons at similar intensities. This structure motivated the design of GRIFFIN, a learning-free adaptive structured pruning mechanism to remove FF neurons during inference at the sequence level which preserves the full model's performance on a large collection of classification and generative tasks at 50% FF sparsity while achieving lower latency. Furthermore, its applicability extends beyond just ReLU-based LLMs. With its straightforward algorithm and no-cost deployment, GRIFFIN expands the accessibility of numerous LLMs for generative inference.perhaps even more consistently than in unperturbed sequences. This suggests something within language actually diversifies the activations, the cause of which would be of interest for future work.  
 
 Appendix: We show example generated text in Figure 8 . Adaptive Wanda and GRIFFIN produce similar summaries compared to the those produced by the full models and the target.We provide more example of flocking across different layers of the LLM. Figure 9 and Figure 10 show flocking in Gemma 7B. Figure 11 and Figure 12 show flocking in Llama 2 7B. Flocking in Gemma 7B is more visually distinct while activations in Llama 2 7B are more distributed.  


Decision: Ngl this is an interesting paper. but the framing of this paper is quite haphazard. 

Reviewers generally like this paper and recommend acceptance. I am leaning accept too. However, i really think the framing needs to be less confusing and more clear. This is a fundamentally weight sparsity and pruning paper and i think the connection with MoE can be a little distracting. I think this paper would benefit from a more direct framing about what it actually is. Conceptually, drawing in MoE in the narrative makes the paper hard to parse. 

I would in fact start the paper by explaining and coining the term flocking and explaining why griffin leverages it. There are too many random plot points happening in this paper.

In short, i recommend acceptance but authors could benefit from cleaning up the story a bit. this paper is trying too hard. a more down to earth narrative that is simple will likely make more impact.

[comments from the PCs] We strongly recommend following the AC advice to improve the paper.

Reviews: review: paper_summary: paper_summary: This paper introduces GRIFFIN, an interesting MOE method, which leverages special activation patterns in Feedforward layers for pruning. The key feature of GRIFFIN lies in its training-free design for selecting unique FF experts at the sequence levels. Besides,  this method is adaptable to non-ReLU methods, which makes it practical in applications, given that many popular LLMs use other activations like SwiGLU. Experimental results demonstrate that, even with only 50% of FF parameters, an MOE LLM employing GRIFFIN can still maintain performance comparable to dense models.
strength_weakness: strength_weakness: The main limitation of this paper is the absence of baseline methods in the experimental section. For instance, Dejavu[1] introduces a contextual structured sparsity approach to dynamically prune layers during runtime. This parallels the strategy adopted in this paper, which also focuses on adaptively pruning Feedforward Neural Networks (FFNs) for different inputs. However, the experiments presented in this submission solely involve magnitude pruning, which may not offer a robust baseline compared to more sophisticated methods like Dejavu. Integrating diverse baseline approaches could provide a more comprehensive assessment of the proposed method's performance and elucidate its relative strengths and weaknesses in comparison to existing techniques.
questions: questions: Please refer to the "Reasons to Reject"
paper_summary: paper_summary: This paper discusses the phenomena of “flocking”, where FF activations in dense LLMs tend to concentrate by column within the same sequence, and proposes GRIFFIN, a novel technique that selects subsets of FFN parameters per sample based on flocking pattern on the prompt to achieve inference speed up without much loss of quality. The proposed method is training-free, empirically applicable to a list of LLMs, and simple to implement.
strength_weakness: strength_weakness: I don’t find major reasons to reject the paper. Some lesser issues I have with the paper are

1. There is not much quantitative characterization of the flocking behavior. The paper shows many examples of flocking, but it’d be good to have some kind of quantitative measurement of this behavior at scale, to demonstrate how ubiquitous is the flocking behavior "in the wild".
2. ROUGE scores for Llama-13B and Mistral-7B on CNN-DM seems too low without any explanation (also Llama-13B < Llama 7B). This brings a bit doubt to the experiment results. It’d be good to provide some qualitative generation examples.
3. In tandem to #2 - it’d be great to provide generation samples with GRIFFIN, as ROUGE-based evals are n-gram-overlap based and may hide problems. It’d be good to rule out possibilities such as generating incoherent yet overlapping phrases, etc., by showing some qualitative samples.
4. A pedantic point: to posit GRIFFIN as an approach to convert a dense model to MoE seems a bit of a stretch of concept. GRIFFIN seems to have much more affinity to adaptive pruning or adaptive (structured) sparsity rather than MoE.
questions: questions: 1. By end of page 1 - “Perhaps an even bigger weakness of many of these methods is the inability to effectively carry over to pre-trained LLMs with non-ReLU activations.” - this is not evident to me, could you elaborate, or if this is from previous studies, cite the study?
2. Figure 3 - Rows of Z_bar are not unit vectors. Is this a mistake in the figure?
3. Table 2 - Llama 2 13B is significantly worse than 7B in CNN-DM, and 2.5 pts of ROUGE-1 is pretty much no overlap. Mistral 7B also sees this issue. What is happening for these experiments? What do the models generate?
4. The observation that flocking patterns are shared within 1 sequence and differ across sequences begs one question - what happens when you take 2 sequences that have distinct patterns and concat them together? will it create problems for GRIFFIN? the meta-question here is what causes patterns to stay stable within sequence, and will there be naturally occurring data that breaks this assumption (therefore potentially make GRIFFIN fail on them)?
5. Section 4.2 right before eqn 6 - “...we do this by aggregating taking…” → "...we do this by aggregating..."?
paper_summary: paper_summary: The article introduces a new, training-free Mixture of Experts (MoE) method called GRIFFIN. This method capitalizes on a phenomenon observed in large language models (LLMs) known as "flocking," where specific feed-forward (FF) neurons exhibit consistently higher activation relative to others within sequences. GRIFFIN leverages these observations to selectively activate the most relevant FF neurons for efficient processing, thereby maintaining model performance while reducing computational demands.
strength_weakness: strength_weakness: Scalability Limitation Due to Batch-Size Constraint: GRIFFIN's current implementation is constrained to a batch size of one, which limits the model's ability to leverage parallel processing capabilities inherent in modern computational hardware. While a batch size of one achieves a modest speed-up of approximately 1.2x, this benefit does not scale linearly with larger batch sizes. This is due to the unique 'flocking' patterns observed in individual samples, which may vary significantly across different inputs within a batch. This variability makes it challenging to maintain consistent efficiency gains when processing multiple samples simultaneously. Such constraints on batch size could restrict the method's utility in real world model severing.
paper_summary: paper_summary: This paper introduces an approach for dynamically pruning the FF of Transformer LLMs that doesn't require any extra training. The structured pruning is conditioned on the prompt. When computing the prompt the regular full model is used. Then, based on the prompt activations, only the top-k rows of each FF across all layers are being kept. The generation is performed with this pruned network. So essentially the  pruning is dynamic at the sequence level. The authors present motivation for this approach by describing a behavior they call "flocking" where the activated neurons are mostly similar within a sequence but vary across different sequnces (studied on open source 7B and 13B models). They show that they can prune around 50% of FF for generation and mostly preserve classification and generation quality leading to about 1.25X speedup in LLama 13B.
strength_weakness: strength_weakness: 1. The writing and framing needs a bit improvement. The title says MoE but then the paper goes back and forth between MoE and pruning. The actual method is doing dynamic structured pruning. I don't think MoEs should be mentioned anywhere (maybe other than related work) and certainly not in the title since its confusing the reader.
2. Given that the method is actually pruning, I think the main weakness of the paper is lack of comparison to more pruning methods other than the presented naive magnitude approach.
3. The ablations part can also be improved. I am not sure what is the point of the sampling-based selection experiment. Instead, it will be interesting to see more ablations on the dynamic requirement for example if you take top-k when averaging activations over the full dataset and then keeping it static.
4. The method seems to work well on short generations but on longer ones the performance seems to substantially degrade. The authors suggest to use a long prompt for alleviating this but this defeats the efficiency purpose.
5. There are some more limitations to this method like having to select columns or rows for structural pruning rather than selecting from all neurons, and overall the speedup potential is limited with this pruning as even 50% parameter reduction of FF leads to small speedup as observed in experiments.
questions: questions: 1. Can you define exactly what you mean by relative FF activation?
2. It seems by the heatmaps that maybe the distribution is slightly different across layers. Have you tried using different k across layers?

Comments: title: Thank you to the reviewers!
Thank you for the rebuttal
comment: We thank all the reviewers again for making thorough comments and suggestions! Since the discussion period will come to a close soon, if  not done so already, we would greatly appreciate if the reviewers could let us know if we have addressed their remarks, so we can further improve the quality of our paper. Thank you!
Thank for very much for the new result, and it looks quite promising. I will raise the score.