House of Cards: Massive Weights in LLMs

## Abstract

Abstract Massive activations, which manifest in specific feature dimensions of hidden states, introduce a significant bias in large language models (LLMs), leading to an overemphasis on the corresponding token. In this paper, we identify that massive activations originate not from the hidden state but from the intermediate state of a feed-forward network module in an early layer. Expanding on the previous observation that massive activations occur only in specific feature dimensions, we dive deep into the weights that cause massive activations. Specifically, we define top- k 𝑘 k italic_k massive weights as the weights that contribute to the dimensions with the top- k 𝑘 k italic_k magnitudes in the intermediate state. When these massive weights are set to zero, the functionality of LLMs is entirely disrupted. However, when all weights except for massive weights are set to zero, it results in a relatively minor performance drop, even though a much larger number of weights are set to zero. This implies that during the pre-training process, learning is dominantly focused on massive weights. Building on this observation, we propose a simple plug-and-play method called MacDrop ( ma ssive weights c urriculum drop out), to rely less on massive weights during parameter-efficient fine-tuning. This method applies dropout to the pre-trained massive weights, starting with a high dropout probability and gradually decreasing it as fine-tuning progresses. Through experiments, we demonstrate that MacDrop generally improves performance across zero-shot downstream tasks and generation tasks.

## 1 Introduction

Large language models (LLMs), such as GPT (Achiam et al., ) and Llama (Touvron et al., ; Dubey et al., ), have achieved remarkable success across diverse natural language tasks (Roziere et al., ; Mitra et al., ; Labrak et al., ; Wu et al., ). Their success is largely attributed to the pre-training phase, during which they are trained on extensive high-quality corpora datasets to predict the next token (Longpre et al., ; Zhao et al., ; Shen et al., ). However, despite the impressive achievements of LLMs, a crucial gap remains in our understanding of the underlying mechanisms that drive their remarkable performance.

Recently, Xiao et al. () uncovered an intriguing phenomenon in LLMs, referred to as *attention sinks*: an unexpectedly large portion of attention is directed toward the initial tokens, regardless of their semantic context, after a small number of early layers. They demonstrated that under a restricted budget, focusing attention solely on recent window leads to poor performance, and that performance is recovered when initial tokens are included. Based on this observation, they proposed StreamingLLM, which retains the key-value caches of the initial sink tokens and the recent tokens for streaming use of LLMs.
Yu et al. () further investigated the attention sinks phenomenon, finding that attention sinks can appear both in the initial tokens and in later tokens with less semantic importance (e.g., ‘.’ and ‘\n’). They showed that when sink tokens appear later in a sequence, sink tokens can potentially result in performance degradation. Inspired by this observation, they proposed a head-wise attention calibration technique without requiring additional training.
Concurrently, Sun et al. () discovered the existence of *massive activations* in the hidden states of LLMs, with magnitudes substantially larger than the others. Massive activations are jointly identified based on their sequence and feature dimensions within the hidden states. Specifically, massive activations occur at the initial tokens and weak semantic tokens according to the model, and are consistently present in only a few fixed feature dimensions. Moreover, they connected massive activations with attention sinks, suggesting that massive activations inject implicit bias into the self-attention mechanism throughout the pre-training phase.

In this paper, we first delve deeper into massive activations, providing two key observations. (1) The bos token always has massive activations in the same feature dimensions and makes attention sinks, regardless of its position in the sequence. This observation enables us to focus only on the feature dimensions, instead of both the sequence and feature dimensions, when addressing massive activations. Namely, a simplified and consistent analysis of massive activations can be achieved by using only the bos token. (2) Massive activations originate in the intermediate state $\hat{h}^{inter}_{l}$ within an early layer $l$, before appearing in the hidden state $h_{l}$, as illustrated in Figure 1(a). Namely, massive activations triggered in $\hat{h}^{inter}_{l}$ are subsequently and continuously propagated through skip connections. This observation implies that the feed-forward network in layer $l$ plays a crucial role in LLMs.

Next, we shift our focus from activations to the weights, relying on the fact that massive activations consistently appear in the same feature dimensions. In detail, we define the *top-$k$ massive weights* as the rows of ${\bm{W}}_{up}$ and ${\bm{W}}_{gate}$ in the feed-forward network at layer $l$ that produce the top-$k$ magnitudes of the intermediate state $\hat{h}^{inter}_{l}$, as illustrated in Figure 1(a). It is important to note that massive weights, defined within a single layer $l$, account for a substantially small fraction compared to the model’s total parameters. This holds true even when compared to the entire ${\bm{W}}_{up}$ and ${\bm{W}}_{gate}$. Nevertheless, massive weights are crucial factors that can completely influence the performance of LLMs. Figure 1(b) presents the generated responses of three models to the given user prompt: original model, top-5 massive weights attacked model, and other weights attacked model. Here, other weights represent all weights in ${\bm{W}}_{up}$ and ${\bm{W}}_{gate}$ at layer $l$ that do not belong to the top-5 massive weights, and an attack sets corresponding weights to zero. When the massive weights are attacked, the model becomes poor and repeats the user prompt. On the contrary, when other weights are attacked, the model does not entirely lose its generation capability, even though a much greater number of weights are set to zero in the same projection matrices. These observations imply that massive weights are dominantly learned during pre-training and highly related to the performance of LLMs.

Finally, we propose a straightforward plug-and-play method during parameter-efficient fine-tuning, named massive weights curriculum dropout (MacDrop). This method applies dropout to the pre-trained massive weights, rather than additional trainable weights, starting with a high dropout rate that is progressively reduced throughout the fine-tuning phase. The intuition behind MacDrop is that a high initial dropout rate encourages the model to lessen dependence on the massive weights predominantly learned during the pre-training phase. Then, reducing the dropout rate facilitates a more stable convergence, ensuring the pre-trained model is leveraged with neglectable damage by the end of fine-tuning. In zero-shot downstream tasks and generation tasks, we demonstrate that MacDrop generally enhances model performance.

Figure: (a) Massive weights.
Refer to caption: extracted/5895615/figures/intro/massive_weights.png

## 2 Massive Weights

In this section, we review the key observations on massive activations reported by Sun et al. () and extend the analysis by exploring various states using the bos token, which was not covered. Based on this expanded analysis, we formally define top-$k$ massive weights in a specific layer and investigate their importance through two opposite types of attacks.

### 2.1 Prerequisite: Massive Activations

Autoregressive Transformers (Vaswani et al., ) are structured with $L$ decoding layers. Each layer $l\in[1,L]$ includes an attention (ATTN) module and a feed-forward network (FFN) module. These modules are connected via residual connections (He et al., ), each following a layer normalization (LN) layer (Ba, ). The previous hidden state $h_{l-1}$ is fed into layer $l$ and processed to produce the subsequent hidden state $h_{l}$:

$$ $h_{l}=\hat{h}_{l}+\text{FFN}(\text{LN}(\hat{h}_{l})),\text{where}\,\hat{h}_{l} =h_{l-1}+\text{ATTN}(\text{LN}(h_{l-1}))\vspace*{-5pt}$ (1) $$

Sun et al. () primarily concentrated on the activations within hidden states, identifying that certain activations exhibit exceptionally large magnitudes, which they termed *massive activations*. Massive activations are observed at the starting position (i.e., input-agnostic) or at the delimiter tokens, depending on the model. Furthermore, these activations are confined to a small number of fixed feature dimensions, even within these tokens. These activations initially emerge after passing through several early layers and then decreases as they near the last layer.

Massive activations are strongly tied to the attention sinks phenomenon, as identified by Xiao et al. (), in which attention is abnormally concentrated on a small subset of tokens. In detail, a given query state tends to have positive cosine similarity with the key states of the tokens exhibiting massive activations, and negative cosine similarity with those of other tokens. Consequently, attention is heavily skewed toward the tokens associated with massive activations.

### 2.2 Further Analysis on Massive Acitvations

We primarily utilize the Llama-3-8B model (Dubey et al., ) and explicitly specify other models when necessary.

##### bos token always has massive activations, regardless of its poisition.

Figure: Figure 2: (Top) Magnitudes of the hidden state and (Bottom) attention scores after Softmax, according to the position of the bos token. The described hidden state is the output of layer 2 (i.e., $h_{2}$). The attention scores are calculated at layer 3 (i.e., after massive activations appear) and averaged across different heads. The bos token always has massive activations, regardless of its position.
Refer to caption: extracted/5895615/figures/bos_token/llama3_8b_hidden_bos9999.png

We begin by examining whether any specific token consistently triggers massive activations. As noted by Sun et al. (), massive activations are observed when any token is placed in the starting position; however, no single token has been found to trigger massive activations regardless of its position. The existence of such a token would greatly facilitate the analysis and algorithm development for handling massive activations, making the process more deterministic by focusing on that token.

Based on the previous observation that nonsemantic tokens can trigger massive activations in certain LLMs, we focus our analysis on the bos token. Figure 2 describes the magnitudes of activations of the hidden state and normalized attention scores according to the position of the bos token, after massive activations appear. We use the implementation(^1^11https://github.com/locuslab/massive-activations) of massive activations for this example and visualization. Llama-3-8B does not have massive activations at delimiter tokens such as ‘.’ (first column). When the bos token is placed in the starting position (second column), the ‘Summer’ token loses its massive activations, while when the bos token is placed in the middle or ending position (third and fourth columns), it triggers massive activations in the same feature dimensions (i.e., 788, 1384, and 4062). What is intriguing is that, despite the difference in magnitude according to the position, the bos token similarly exhibits attention sinks. Therefore, we use only the *bos* token (placed at the starting position) for the continuation of analysis and algorithm development.

##### Massive activations originate in the intermediate state of a FFN module.

Figure: (a) Various states in early layers.
Refer to caption: extracted/5895615/figures/tracking/Meta-Llama-3-8B_tracking.png

Next, we trace various states in early layers until the first massive activations appear, using the bos token. Specifically, we monitor $h_{l-1}$, $\text{LN}(h_{l-1})$, $\text{ATTN}(\text{LN}(h_{l-1}))$, $\hat{h}_{l}$, $\text{LN}(\hat{h}_{l})$, and $\text{FFN}(\text{LN}(\hat{h}_{l}))$ in Eq. (1) throughout early layers. Figure 3(a) illustrates the magnitudes of various states in layers $l\in[1,3]$. It is observed that $\text{FFN}(\text{LN}(\hat{h}_{2}))$ has massive activations before $h_{2}$. With Figure 3(b), which describes the top three and median magnitudes of the hidden state(^1^1footnotemark: 1), it is observed that the massive activations generated within a FFN module at layer 2 are transmitted directly to the next hidden state and then propagated solely through the residual connections.

Furthermore, we decompose a FFN module into ${\bm{W}}_{down}(\sigma({\bm{W}}_{gate}(\cdot))\odot{\bm{W}}_{up}(\cdot))$, to analyze the intermediate states (i.e., the output of $\sigma({\bm{W}}_{gate}(\cdot))\odot{\bm{W}}_{up}(\cdot)$). Figure 3(c) describes the top three and median magnitudes of the intermediate state across layers. It is demonstrated that *massive activations originate in the intermediate state of a FFN module in an early layer $l$*. This result implies that ${\bm{W}}_{up}$ and ${\bm{W}}_{gate}$ in layer $l$ are closely tied to massive activations. Additional results for other LLMs are provided in the Appendix C.

### 2.3 Massive Weights

Massive weights are defined based on massive activations in the intermediate state at layer $l$, denoted as $\hat{h}_{l}^{inter}$, when the bos token is fed into LLMs. To elaborate, we define the rows in the projection matrix ${\bm{W}}_{up}$ (and ${\bm{W}}_{gate}$, if it exists) that correspond to the indices of the top-$k$ magnitudes in $\hat{h}_{l}^{inter}$ as *top-$k$ massive weights*, depicted in Figure 1(a). It is noted that massive weights are defined within one specific layer, which means the number of massive weights is significantly smaller compared to the total number of parameters in LLMs. For example, in Llama-3-8B, the number of top-$k$ massive weights is calculated as $2\times k\times 4096$, where $4096$ represents the dimensions of hidden state. If $k$ is set to 5, massive weights account for approximately 0.0005% of the total parameters in Llama-3-8B, approximately 0.0001% in Llama-3-70B, and approximately 0.00004% in Llama-3.1-405B.

##### Massive weights are extremely small in quantity, their impact is tremendous.

**Table 1: Perplexity and zero-shot downstream tasks performance according to the attack.**
| Models | WikiText | C4 | PG-19 | Avg. $(\downarrow)$ | ARC-E | ARC-C | BoolQ | PIQA | WG | Avg. $(\uparrow)$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Llama-3-8B | 5.75 | 9.94 | 8.98 | 8.22 | 77.4 | 52.7 | 81.4 | 80.8 | 72.9 | 73.0 |
| top-$5$ zeroing | 104.57 | 132.83 | 130.83 | 122.74 | 29.1 | 22.0 | 41.8 | 53.8 | 50.3 | 39.4 |
| top-$5$ retaining | 10.15 | 23.98 | 28.72 | 20.95 | 75.0 | 48.0 | 80.2 | 77.7 | 74.1 | 71.0 |
| Llama-3-70B | 2.68 | 7.59 | 6.02 | 5.43 | 85.8 | 64.2 | 85.3 | 84.6 | 80.5 | 80.1 |
| top-$5$ zeroing | 11135.81 | 7288.86 | 4696.49 | 7707.05 | 28.6 | 22.8 | 38.0 | 55.5 | 49.5 | 38.9 |
| top-$5$ retaining | 3.47 | 9.93 | 7.26 | 6.89 | 45.9 | 25.6 | 83.9 | 64.6 | 77.0 | 59.4 |
| Llama-3.1-405B (8bit) | 1.41 | 6.20 | 3.23 | 3.61 | 86.3 | 66.0 | 88.2 | 85.0 | 81.1 | 81.3 |
| top-$5$ zeroing | 1785.56 | 985.36 | 633.40 | 1134.77 | 26.6 | 25.9 | 37.8 | 49.7 | 50.3 | 38.1 |
| top-$5$ retaining | 2.47 | 9.36 | 5.61 | 5.81 | 83.0 | 62.5 | 83.1 | 83.2 | 70.3 | 76.4 |

To assess the significance of massive weights, we conduct two types of attacks: top-$k$ zeroing and top-$k$ retaining. Note that these attacks only affect the ${\bm{W}}_{up}$ and ${\bm{W}}_{gate}$ projection matrices in layer $l$, where massive weights are present. The first attack is to set the top-$k$ massive weights to zero (i.e., darker orange weights in Figure 1(a)). In essence, this attack is very similar to the one proposed in Sun et al. (), where massive activations in the hidden state are zeroed out in a single layer. The difference is that their attack targets the hidden state, while our attack targets the intermediate state. The second attack is to set all weights to zero except for top-$k$ massive weights (i.e., lighter orange weights in Figure 1(a)). That is, the number of rows being damaged in each attack is $k$ and the dimensions of intermediate state $-$ $k$, respectively.

Following Sun et al. (), we assess perplexity(^1^1footnotemark: 1) on three datasets: WikiText (Merity et al., ), C4 (Raffel et al., ), and PG-19 (Rae et al., ). Additionally, we evaluate zero-shot accuracy(^2^22https://github.com/EleutherAI/lm-evaluation-harness) on five tasks: Arc-Easy, Arc-Challenge (Clark et al., ), BoolQ (Clark et al., ), PIQA (Bisk et al., ), and WinoGrande (WG) (Sakaguchi et al., ). Table 1 presents the results of two attacks on Llama-3-8B, Llama-3-70B, and Llama-3.1-405B (8bit)(^3^33We use 8xA100-80G GPUs for our work; therefore, we employ an 8-bit model.), when $k$ is set to $5$. Llama-3-8B has its massive weights in layer 2 out of 32 layers, Llama-3-70B in layer 4 out of 80 layers, and Llama-3.1-405B in layer 6 out of 126 layers (Appendix B). The larger the difference from the original performance, the stronger the attack.

Top-5 zeroing is a much stronger attack than top-5 retaining, even though top-5 retaining sets several thousands times more weights to zero compared to top-5 zeroing does for the same projection matrices. This means that in the projections ${\bm{W}}_{up}$ and ${\bm{W}}_{gate}$ at layer $l$, having only massive weights is significantly better than having all weights except for massive weights. In detail, similar to the findings of Sun et al. (), the top-$k$ zeroing attack proves to be highly effective in disrupting the Llama-3 family, even for extremely large-scale models (e.g., 70B and 405B). On the other hand, the top-$k$ retaining attack does not cause complete damage. In conclusion, these findings reveal that massive weights are predominantly learned during pre-training, highlighting their essential contribution to the overall performance of LLMs.

Moreover, massive weights also exist in instruction-tuned LLMs such as Llama-3-8B-Instruct. These attacks are effective, as depicted in Figure 1(b). When massive weights are set to zero (i.e., darker orange box), the model repeats the same text as the user prompt. On the other hand, when all weights are set to zero except for massive weights (i.e., lighter orange box), the model retains its ability to generate text, although the generated text differs from the original.

##### k 𝑘 k italic_k , which affects performance, depends on the model architecture.

Figure: (a) Top-$k$ zeroing.
Refer to caption: extracted/5895615/figures/k_robust/k_zeroing.png

Figure: Figure 5: Probability of experts of Mixtral-8x7b for the bos token. In the layer with massive weights (i.e., layer 2), the router probability between experts becomes completely skewed to one expert.
Refer to caption: extracted/5895615/figures/tracking/expert_probs.png

We examine the robustness of various LLMs against the top-$k$ zeroing and retaining attacks, with a focus on the impact of the parameter $k$. Llama-2 (Touvron et al., ), Llama-3 (Dubey et al., ), Mistral (Jiang et al., ) and Mixtral (Jiang et al., ), Phi-3 (Abdin et al., ), and Gemma-2 (Team et al., ) families are used. Figure 4 illustrates the mean zero-shot accuracy of LLMs under the two attacks, according to the $k$. In top-$k$ zeroing, more weights are set to zero as $k$ increases, whereas in top-$k$ retaining, more weights are set to zero as $k$ decreases. When $k$ is 0 in top-$k$ zeroing, it corresponds to the original performance without any attack, whereas, when $k$ is 0 in top-$k$ retaining, it sets the entire weights of ${\bm{W}}_{up}$ and ${\bm{W}}_{gate}$ in layer $l$ to zero.

The Llama families are highly sensitive to massive weights. In the top-$k$ zeroing, a noticeable performance drop occurs even when $k$ is as small as 3, irrespective of the model’s scale. In top-$k$ retaining, when $k$ is set to 1 (i.e., with only one row active in ${\bm{W}}_{up}$ and ${\bm{W}}_{gate}$ in layer $l$), the performance nearly reaches the original level in smaller-scaled models ($\leq$ 13B). While, for larger-scaled models ($\geq$ 70B), the top-30 massive weights are required to maintain performance.

Similarly, Mistral is also disrupted, when $k$ is set to 5. Mixtral is a sparse Mixture of Experts (MoE) architecture that uses a top-2 routing mechanism, where two experts are activated among eight FFN modules in each layer. To attack the Mixtral model, we identify the active experts in the layer with massive weights using the bos token. Figure 5 describes the probability distribution of experts in the Mixtral model across all layers. Notably, it is observed that when massive activations occur, a single expert (i.e., expert 4) is assigned a significantly higher probability than the others. Therefore, we target only the ${\bm{W}}_{up}$ and ${\bm{W}}_{gate}$ of this expert, rather than all experts. Although Mixtral does not completely break down, there is a considerable decline in performance when the top-50 massive weights are zeroed out. These results indicate that, despite the immense resources required to build high-performance LLMs, they can collapse like a house of cards even under minimal attacks.

The Phi-3 family exhibits different robustness against attacks depending on the model size. As noted by Abdin et al. (), the phi-3-mini (3.8B) is trained on 3.3T tokens, while the phi-3-medium (14B) is trained on 4.8T tokens. A key architectural difference from the Llama family is the use of dropout to the outputs of both the ATTN and FFN modules, formed by Eq. (2). While a specific recipe for dropout is not provided in the technical report (Abdin et al., ), in the case of phi-3-medium, applying dropout with longer pre-training might ensure that the residual connections contribute meaningfully, mitigating the risk of excessive dependence on massive weights.

$$ $h_{l}=\hat{h}_{l}+\text{dropout}(\text{FFN}(\text{LN}(\hat{h}_{l}))),\text{ where}\,\hat{h}_{l}=h_{l-1}+\text{dropout}(\text{ATTN}(\text{LN}(h_{l-1})))$ (2) $$

The Gemma-2 family is exceptionally resilient to the top-$k$ zeroing attack, maintaining almost no loss in performance even when $k$ is large. Additionally, even if ${\bm{W}}_{up}$ and ${\bm{W}}_{gate}$ are entirely eliminated (i.e., $k=0$ in top-$k$ retaining), there is no noticeable performance degradation. This family incorporates two additional LN layers after both the ATTN and FFN modules, formed by Eq. (3). These added normalization layers result in completely different hidden and intermediate states compared to other models, as described in Appendix C. Furthermore, attention sinks at the initial token are not observed in the Gemma-2 family (Appendix D).

$$ $h_{l}=\hat{h}_{l}+\text{LN}(\text{FFN}(\text{LN}(\hat{h}_{l}))),\text{where}\, \hat{h}_{l}=h_{l-1}+\text{LN}(\text{ATTN}(\text{LN}(h_{l-1})))$ (3) $$

Nevertheless, because most existing open-source models, except for a few, are still heavily dependent on massive weights, it is critical to take them into account.

## 3 Massive Weights Curriculum Dropout

In this section, we propose a straightforward plug-and-play method, termed massive weights curriculum dropout (MacDrop), during parameter-efficient fine-tuning such as low rank adaptation (Hu et al., ). This method applies dropout to the pre-trained massive weights with a curriculum that gradually reduces the dropout probability. The reason for applying dropout to weights (Wan et al., ) instead of activations (Srivastava et al., ) is that the number of massive activations is only $k$, but that of massive weights is $k\times d$, where $d$ is the dimension of the hidden states. Note that our method is not applied to the trainable parameters of adapters. Therefore, MacDrop can be applied orthogonally to the process of training the adapter. Algorithm 1 describes MacDrop in a pseudo PyTorch (Paszke et al., ) style, and is implemented within the trainer code of transformers(^4^44https://github.com/huggingface/transformers/tree/main/src/transformers). Initially, massive weights are identified using the bos token before fine-tuning (Lines 1-3). Subsequently, an adapter is trained while the pre-trained massive weights are dropped. Meanwhile, a curriculum strategy is applied to progressively enable the use of the original pre-trained weights without masking. Note that in Line 8, ‘model’ includes both the masked pre-trained network and a trainable adapter.

MacDrop is motivated by the observation that massive weights are predominantly learned during the pre-training phase, and that zeroing them can severely undermine LLMs. Therefore, at the early stages of fine-tuning, the objective is to reduce the reliance on massive weights, as their excessive dominance may lead the model to over-rely on specific patterns. Moreover, considering that the undamaged pre-trained model is used after fine-tuning is finished, we develop a strategy to adjust the dropout rate using a curriculum.

Figure: Algorithm 1 Top-$k$ Massive Weights Curriculum Dropout in pseudo PyTorch style

## 4 Experiments

### 4.1 Zero-shot Downstream Task

**Table 2: Zero-shot downstream tasks performance.**
| Model | Method | ARC-Easy | ARC-Challenge | BoolQ | PIQA | WinoGrande | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Llama-3-8B | LoRA | 79.6 | 58.2 | 83.9 | 82.4 | 75.9 | 76.0 |
| + MacDrop | 82.9 | 58.3 | 83.9 | 82.6 | 75.0 | 76.5 |  |
| DoRA | 80.8 | 57.7 | 83.9 | 82.5 | 75.8 | 76.1 |  |
| + MacDrop | 81.9 | 58.2 | 83.9 | 82.2 | 75.6 | 76.4 |  |
| Mistral-7B | LoRA | 78.5 | 54.9 | 84.9 | 82.9 | 75.3 | 75.3 |
| + MacDrop | 80.9 | 56.7 | 85.0 | 83.0 | 75.3 | 76.2 |  |
| DoRA | 78.4 | 55.1 | 85.0 | 82.9 | 75.1 | 75.3 |  |
| + MacDrop | 80.6 | 56.7 | 85.3 | 82.9 | 75.1 | 76.1 |  |

**Table 3: Generation tasks performance measured by Prometheus-2-7B.**
| Model | Method | MT-1 | MT-2 | translation | summarization | QA | math reasoning | RAG | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Llama-3-8B | LoRA | 3.71 | 3.54 | 4.50 | 3.24 | 4.35 | 3.64 | 3.74 | 3.82 |
| + MacDrop | 3.76 | 3.49 | 4.51 | 3.39 | 4.29 | 3.50 | 3.71 | 3.81 |  |
| DoRA | 3.85 | 3.71 | 4.56 | 3.34 | 4.35 | 3.59 | 3.73 | 3.80 |  |
| + MacDrop | 3.79 | 3.41 | 4.55 | 3.26 | 4.29 | 3.79 | 3.85 | 3.85 |  |
| Mistral-7B | LoRA | 3.55 | 3.30 | 4.61 | 3.33 | 4.41 | 3.45 | 4.05 | 3.81 |
| + MacDrop | 3.75 | 3.42 | 4.58 | 3.26 | 4.35 | 3.30 | 3.98 | 3.81 |  |
| DoRA | 3.55 | 3.29 | 4.64 | 3.34 | 4.49 | 3.35 | 3.88 | 3.79 |  |
| + MacDrop | 3.49 | 3.29 | 4.59 | 3.52 | 4.41 | 3.33 | 3.95 | 3.80 |  |

We fine-tune the Llama-3-8B and Mistral-7B using the alpaca_gpt4_en dataset (Peng et al., ) for 3 epochs (579 steps), and evaluate on five zero-shot tasks. We use two parameter-efficient fine-tuning methods, LoRA (Hu et al., ) and DoRA (yang Liu et al., ). DoRA decomposes the pre-trained weights into two components, magnitude and direction, and applies LoRA to the direction component. Our method is based on the implementation of Llama-Factory (Zheng et al., )(^5^55https://github.com/hiyouga/LLaMA-Factory). For MacDrop, $k$ and $p_{0}$ are set to 5 and 0.8, respectively. Details for implementations are explained in Appendix A. Table 2 presents the results on zero-shot downstream tasks. For both the models and methods, MacDrop consistently leads to performance gains, especially in ARC-Easy and ARC-Challenge tasks.

### 4.2 Generation Task

We evaluate on the generated texts of the same models in Section 4.1 using the Spec-Bench dataset (Xia et al., ). This benchmark includes six subtasks, each containing 80 instances: multi-turn (MT) conversation from MT-bench (Zheng et al., ), translation from WMT14 DE-EN (Bojar et al., ), summarization from CNN/Daily Mail (Nallapati et al., ), question answering (QA) from Natural Questions (Kwiatkowski et al., ), mathematical reasoning from GSM8K (Cobbe et al., ), and retrieval-augmented generation (RAG) from Natural Questions (Kwiatkowski et al., ). We utilize the direct assessment of Prometheus-2-7B (Kim et al., ) to evaluate the generated texts using a 5-point Likert scale. Prometheus-2-7B is an open-source language model specifically designed for evaluation purposes(^6^66https://github.com/prometheus-eval/prometheus-eval). Table 3 presents the results on generation tasks. MT-1 and MT-2 indicate the first turn and second turn, respectively. Unfortunately, MacDrop shows limited performance improvements in generation tasks. Examples of the generated texts and judgements are provided in Appendix E.

### 4.3 Ablation Study

We further provide ablation studies related to MacDrop using Llama-3-8B. Unless otherwise stated, for MacDrop, $k$ and $p_{0}$ are set to 5 and 0.8.

#### 4.3.1 Dropout Scope and Probability

Figure: Figure 6: Mean zero-shot accuracy according to the dropout scope and probability $p$.
Refer to caption: extracted/5895615/figures/ablation/dropout_scope.png

We investigate the effect of dropout scope and probability compared to the original performance achieved through LoRA without dropout. This ablation study is also conducted on the ${\bm{W}}_{up}$ and ${\bm{W}}_{gate}$ projection matrices in layer $l$. The dropout scope is divided into three categories: all weights, massive weights, all weights except for massive weights. Additionally, to assess the impact of dropout probability, it is kept constant throughout the fine-tuning process, without using a curriculum.

Figure 3(b) illustrates the mean zero-shot accuracy according to the dropout scope and dropout probability $p$. It is observed that among three scopes, the original performance (i.e., without dropout), represented by the dotted line at 76.0, can be surpassed only when dropout is applied solely to massive weights. Nevertheless, if strong dropout (e.g., $p\geq 0.85$) is maintained on the pre-trained massive weights during fine-tuning, performance deteriorates. This highlights the need to safeguard the pre-trained massive weights during the final stages of fine-tuning, because we are ultimately using them without causing any damage.

#### 4.3.2 Curriculum Methods and Initial Dropout Probability

Figure: Figure 7: Curriculum methods.
Refer to caption: extracted/5895615/figures/ablation/curriculum.png

We investigate the effect of curriculum methods and initial dropout probability $p_{0}$ in MacDrop, when LoRA is applied. We compare four curriculum methods: step-wise linear (Step), before epoch-wise linear (Epoch(before)), after epoch-wise linear (Epoch(after)), and exponential (Exp.). In formula, Step is defined as $p=p_{0}\times(1-\frac{t_{step}}{T_{step}})$. Epoch(before) and Epoch(after) are defined as $p=p_{0}\times(1-\frac{t_{epoch}-1}{T_{epoch}})$ and $p=p_{0}\times(1-\frac{t_{epoch}}{T_{epoch}})$, respectively. Exp. is defined as $p=p_{0}\times exp(-\alpha t_{step})$. Figure 7 describes dropout probability $p$ according to curriculum methods, when $p_{0}$ is 1.0. The distinct difference between Epoch(before) and Epoch(after) is that at the final epoch, the former continues to apply dropout to the pre-trained massive weights with a probability of $p_{0}\times\frac{1}{T_{epoch}}$, while the latter fully utilizes the pre-trained massive weights.

**Table 4: Mean zero-shot accuracy according to curriculum methods and initial dropout probability.**
| Curriculum | $p_{0}$ |  |  |  |
| --- | --- | --- | --- | --- |
| Method | $0.2$ | $0.5$ | $0.8$ | $1.0$ |
| Step | 76.0 | 76.3 | 76.5 | 75.5 |
| Epoch(before) | 76.1 | 76.1 | 76.1 | 75.5 |
| Epoch(after) | 75.9 | 76.0 | 76.2 | 76.3 |
| Exp. ($\alpha=0.01$) | 76.0 | 76.2 | 76.5 | 76.4 |
| Exp. ($\alpha=0.05$) | 76.0 | 76.1 | 76.2 | 76.3 |
| Exp. ($\alpha=0.10$) | 76.0 | 76.1 | 76.1 | 76.2 |
| Mean | 76.0 | 76.1 | 76.3 | 76.0 |

Table 4 presents mean zero-shot accuracy according to curriculum methods and initial dropout probability $p_{0}$. It is observed that step-based curriculum methods (e.g., Step and Exp.) generally achieve greater performance improvements compared to epoch-based curriculum methods (e.g., Epoch(before) and Epoch(after)). Nevertheless, when the initial dropout probability is relatively low (e.g., $p_{0}\leq 0.2$), even step-based curriculum methods fail to bring any performance gain compared to the original performance of 76.0. Additionally, it is shown that using a smaller $\alpha$ in the Exp. method leads to greater performance improvements, suggesting that a rapid decline in dropout probability to zero can diminish the effectiveness of MacDrop. On the other hand, for the Step and Epoch(before) methods, a significant performance drop is observed at a $p_{0}$ value of 1.0, highlighting the necessity of a near-zero dropout probability for the end of fine-tuning. In conclusion, for MacDrop, we recommend using the Step or Exp. with a smaller $\alpha$, initiated from a moderately high $p_{0}$.

## 5 Related work

The attention sinks phenomenon and their importance, uncovered by Xiao et al. (), have been widely used to compress key-value caches. For quantization, KVQuant (Hooper et al., ) applies attention sink-aware quantization, which retains only the first token in fp16. CushionCache (Son et al., ) inserts sink tokens into the prefix to mitigate massive activations in the middle of the sequence, enhancing the performance of quantized models. For token eviction and token merging, sink tokens are never evicted or merged; they remain unchanged (Xiao et al., ; Ge et al., ; Li et al., ; Zhang et al., ; Wang et al., ; Zhang et al., ).

Nevertheless, there has been limited in-depth research on the phenomenon itself. In fact, the idea of global attentions, such as [CLS] and [SEP] tokens–similar to attention sinks–was introduced and emphasized even before the LLM era (Zaheer et al., ; Beltagy et al., ). In the LLM era, Yu et al. () showed that sink tokens can appear not only at the beginning of a sentence but also in the middle, and they are often shown to be nonsemantic (e.g., ‘.’). Sun et al. () discovered the presence of massive activations in the hidden state space of sink tokens, demonstrating that massive activations trigger the attention sinks phenomenon. Meanwhile, in vision transformers, similar phenomenon is observed (Darcet et al., ). They showed that training with register tokens, which is additional meaningless tokens similar to sink tokens, resulted in improved dense prediction and interpretability. Different from previous work, we explore this phenomenon in the weight space.

## 6 Conclusion

In this paper, we explore the weight space of LLMs and identify the presence of massive weights within a FFN module in an early layer, which are predominantly pre-trained and have a significant impact on the performance of LLMs. Based on our observation, we propose a plug-and-play fine-tuning method called MacDrop, which applies dropout to the pre-trained massive weights, rather than to the parameters of adapters, during parameter-efficient fine-tuning. We hope that our findings will inspire future research in weight space of LLMs, including model merging (Li et al., ) and model editing (Yao et al., ).

## Reproducibility

For our analysis in Section 2, we conduct a consistent and reproducible analysis using only the *bos* token (Section 2.2) and provide the specific position of massive weights across various models in Appendix B. For our algorithm, MacDrop, we provide PyTorch-style pseudo code in Section 3 and training details in Appendix A. Furthermore, we present github links for all our implementations with footnotes, when necessary.

## Appendix A Implementation Details

We use 8xA100-80GB, for our all implementations. As discussed in Section 2.2, we use only the bos token to analyze massive activations and massive weights, and design MacDrop. In Section 2.2, we use the example and visualization of massive activations. For parameter-efficient fine-tuning, Llama-Factory is used and configurations are summarized in Table 5. For evaluation, we use the code of massive activations for perplexity, use lm-eval-harness for zero-shot accuracy, and use FastChat and Prometheus for generation tasks. Related papers and codes are cited in the main.

**Table 5: Configuration for low rank adaptation (LoRA and DoRA).**
| Argument | Setting |
| --- | --- |
| dataset | alpaca_gpt4_en |
| validation size | 0.05 |
| per device train batch size | 8 |
| gradient accumulation steps | 4 |
| learning rate | 1e-4 |
| num train epochs | 3 |
| warmup ratio | 0.05 |
| adam $\beta_{1}$ | 0.9 |
| adam $\beta_{2}$ | 0.999 |
| lora target | all linear layers except for embedding layer and lm head |
| lora rank | 16 |
| lora alpha | 16 |

## Appendix B Position of Massive Weights

Table 6 summarizes the position of massive weights across various models.

**Table 6: Layer and indices of top-$5$ massive weights.**
| Model | Layer | Top-$5$ indices |
| --- | --- | --- |
| Llama-2-7b-hf | 2 | [7890, 10411, 1192, 8731, 5843] |
| Llama-2-7b-chat-hf | 2 | [7890, 10411, 1192, 8731, 5843] |
| Llama-2-13b-hf | 4 | [7678, 8811, 11371, 6619, 12281] |
| Llama-2-13b-chat-hf | 4 | [7678, 8811, 11371, 6619, 12281] |
| Meta-Llama-3-8B | 2 | [2427, 198, 6412, 12657, 591] |
| Meta-Llama-3-8B-Instruct | 2 | [2427, 198, 6412, 591, 12657] |
| Meta-Llama-3-70B | 4 | [16581, 3590, 16039, 19670, 13266] |
| Meta-Llama-3-70B-Instruct | 4 | [16581, 3590, 16039, 19670, 13266] |
| Meta-Llama-3.1-405B (8bit) | 6 | [11891, 30740, 2392, 36238, 12328] |
| Meta-Llama-3.1-405B-Instruct (8bit) | 6 | [11891, 30740, 36238, 2392, 1073] |
| Mistral-7B-v0.1 | 2 | [7310, 8572, 2514, 1878, 8693] |
| Mistral-7B-Instruct-v0.1 | 2 | [7310, 8572, 2514, 2484, 1878] |
| Mixtral-8x7B-v0.1 | 2 (expert 4) | [7310, 7530, 11981, 7492, 3178] |
| Mixtral-8x7B-Instruct-v0.1 | 2 (expert 4) | [7310, 11981, 2514, 7530, 3178] |
| Phi-3-mini-4k-instruct | 3 | [808, 340, 3644, 2473, 2987] |
| Phi-3-medium-4k-instruct | 6 | [181, 7540, 19, 15874, 5137] |
| gemma-2-2b | - | - |
| gemma-2-2b-it | - | - |
| gemma-2-9b | - | - |
| gemma-2-9b-it | - | - |
| gemma-2-27b | - | - |
| gemma-2-27b-it | - | - |

## Appendix C Futher Analysis for Various LLMs

We investigate various LLM families: Llama-2 (Touvron et al., ), Llama-3 (Dubey et al., ), Mistral (Jiang et al., ) and Mixtral (Jiang et al., ), Phi-3 (Abdin et al., ), and Gemma-2 (Team et al., ). Similar to Figure 3 in the main, we provide the top-3 and median magnitudes in the hidden states and the intermediate states throughout the layers. In subcaptions, H and I represent the hidden state and the intermediate state, respectively.

When comparing pre-trained LLMs (e.g., Llama-2-7b-hf) and instruction-tuned LLMs (e.g., Llama-2-7b-chat-hf) of the same model, the shape of their graphs is almost identical. This suggests that massive weights are formed during the pre-training process. Llama-2, Llama-3, Mistral, Mixtral, and Phi-3 exhibit similar patterns in their hidden states: following a single explosive amplification in an early layer, massive activations are sustained through residual connections almost until the final layer, although Phi-3 experiences a few additional amplifications. In fact, as we discuss in the main, such an explosion initially occurs in the intermediate state, and this phenomenon is observed across different models. However, the behavior of the Gemma-2 family significantly deviates from that of other models. Firstly, instead of the values being maintained in the hidden state, Gemma-2 shows a continuous increase followed by a decrease. Secondly, the magnitude of the explosion observed in the intermediate state is considerably lower compared to other models. These unique characteristics suggest that Gemma-2 operates under different internal dynamics, which may influence its overall performance and stability.

### C.1 Llama-2 family

Figure: (a) Llama-2-7b-hf (H).
Refer to caption: extracted/5895615/figures/appx/Llama-2-7b-hf_position0_hidden_state.png

### C.2 Llama-3 family

Figure: (a) Meta-Llama-3-8B (H).
Refer to caption: extracted/5895615/figures/appx/Meta-Llama-3-8B_position0_hidden_state.png

### C.3 Mistral and Mixtral family

Figure: (a) Mistral-7B-v0.1 (H).
Refer to caption: extracted/5895615/figures/appx/Mistral-7B-v0.1_position0_hidden_state.png

### C.4 Phi-3 family

Figure: (a) Phi-3-mini-4k-instruct (H).
Refer to caption: extracted/5895615/figures/appx/Phi-3-mini-4k-instruct_position0_hidden_state.png

### C.5 Gemma-2 family

Figure: (a) gemma-2-2b (H).
Refer to caption: extracted/5895615/figures/appx/gemma-2-2b_position0_hidden_state.png

## Appendix D Attention Sinks

Figure 13 describes attention after Softmax in the early layers (from layer 1 to layer 8) across various models. Attention sinks are observed in the layers after the massive weights layer. In Llama-2-7B (Figure 13(a)), Mistral-7B (Figure 13(c)), and Mixtral-8x7B (Figure 13(d)), sink tokens are the initial token (‘Summer’) and the first delimiter token (‘.’), discovered by Sun et al. (). In Llama-3-8B (Figure 13(b)) and Phi-3-mini (Figure 13(e)), sink token is the only the initial token (‘Summer’). Interestingly, in these five models, it is commonly observed that significant attention is concentrated on non-semantic tokens (‘.’) before attention sinks occur. However, in Gemma-2 (Figure 13(f)), attention sinks do not happen and attention is primarily assigned to local tokens. Note that what we provide is the average of the heads, and there might be heads that do not fully sink when viewed individually.

Figure: (a) Llama-2-7B. Attention sinks happen from layer 3.
Refer to caption: extracted/5895615/figures/appx_attention/Llama-2-7b-hf_attention_map_layer1-8.png

## Appendix E Examples in Generation Task

Assistant A and B indicate Llama-3-8B with and without MacDrop, respectively.

**Table 7: Example of generation task (translation).**
| User | Translate German to English: St. Hubertus war der Sage nach ein recht rücksichtsloser Jäger, der durch eine Vision bekehrt und später zum Bischof von Lüttich ernannt wurde. |
| --- | --- |
| Reference answer | St. Hubert was, according to legend, a truly ruthless hunter, who converted following a vision and was later appointed Bishop of Lüttich. |
| Assistant A | According to legend, St. Hubertus was a rather ruthless hunter who was converted and later appointed as the Bishop of Liège after a vision. |
| Judgment for A | The response provided a translation that was generally accurate and relevant to the user’s request. However, there were minor inaccuracies in the phrasing, such as ”rather ruthless” instead of ”truly ruthless,” which slightly deviated from the original text. Additionally, the term ”Liège” was used instead of ”Lüttich,” which is the correct name of the city in the original German text. Despite these minor issues, the response still conveyed the main points of the legend about St. Hubertus. Therefore, while the response was helpful and mostly aligned with the user’s inquiry, it did not perfectly cater to the user’s needs due to the slight inaccuracies in the translation. So the overall score is 4. |
| Assistant B | According to legend, St. Hubertus was a rather reckless hunter who was converted and later appointed as the Bishop of Liège after a vision. |
| Judgment for B | The response provided a translation that was generally accurate and relevant to the user’s request. However, there were slight deviations from the original text. For instance, the term ”rather reckless” is less precise than ”truly ruthless,” which could lead to some ambiguity. Additionally, the term ”Liège” is used instead of ”Lüttich,” which is a minor inaccuracy. Despite these minor issues, the response still managed to convey the main points of the original text, making it useful for the user’s needs. Therefore, the response aligns well with the user’s inquiry, with only rare inaccuracies, and thus meets the criteria for a score of 4. |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to those in the Weakness section.
soundness: 2
strengths: 1. The paper identifies a new and critical phenomenon in LLMs where specific feature dimensions in hidden states exhibit massive activations, leading to an overemphasis on certain tokens. This discovery contributes to a deeper understanding of potential biases within LLMs.
2. The authors provide detailed experiments and analysis on massive activations and massive weights, in various LLMs.
3. MacDrop is introduced as a plug-and-play method that applies curriculum dropout to pre-trained massive weights during fine-tuning, which is an innovative approach to reducing reliance on these dominant weights.
summary: This paper reveals that large language models (LLMs) have a bias towards specific tokens due to massive activations in their hidden states, which stem from the intermediate state of early layer feed-forward network modules. The authors define top-k massive weights as those with the greatest impact on these activations and propose MacDrop, a method that employs dropout on these weights during fine-tuning to reduce their influence. By starting with a high dropout rate and gradually decreasing it, MacDrop improves LLM performance on zero-shot tasks and generations, offering a parameter-efficient fine-tuning strategy that enhances model robustness.
weaknesses: 1. Novelty.

1.1 In this paper, the massive weights are rows in weight matrics related to the massive activations. It is similar to Wanda-sp proposed in [1], which is modified from [2]. From this perspective, the novelty of this paper is limited.

1.2 Moreover, the proposed fine-tuning method MacDrop is similar to LoRA. The only difference is that MacDrop uses a curriculum strategy to gradually reduce the dropout rate of the massive weights. However, the improvement of MacDrop over LoRA is less significant.

2. As the authors discovered, different models have different robustness to the massive weights and the phenomenon of massive weights is not as universal as massive activation [3]. It might also imply that MacDrop only works on models that are sensitive to massive weights, e.g. Llama-3-8B and Mistral-7B.

3. The experiments are not sufficient, for example, 

3.1. Does the choice of the dataset have any influence on the massive activations and massive weights? 

3.2. This paper illustrates the Llama-3-8B model shows the phenomenon that the bos token always has massive activations, regardless of its position. Do other models share the same phenomenon?


[1] Fluctuation-based Adaptive Structured Pruning for Large Language Models. AAAI 2024

[2] A simple and effective pruning approach for large language models. ICML 2023

[3] Massive activations in large language models. ICML 2024

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: - What is the additional computational cost of performing MacDrop?
soundness: 3
strengths: - This paper includes an in-depth analysis of massive weights in LLMs and their relationship to massive activations.
- The investigation builds upon previous observations around massive activations and attention sink tokens. Rather than previous work which focused on token location, this work explores how the bos token, no matter its location in the sentence, emerges as an attention sink.
- The authors present a set of ablation studies demonstrating the effects of preserving the top k massive weights and removing the rest, or dropping the top k massive weights and keeping the remaining weights. They compare these results to baselines on several relevant tasks.
- The MacDrop method performs dropout to weights on a curriculum. The aim of this method is to reduce the dependence on massive weights and therefore achieve higher performance on downstream tasks. These ideas are validated through experimental results.
summary: This paper analyzes the phenomenon of massive weights in LLMs. These weights are related to previously-observed massive activations. The authors find that massive weights are critical for model performance. They introduce a novel dropout method, called MacDrop, which reduces reliance on massive weights and improves performance.
weaknesses: - MacDrop’s inability to improve performance on generation tasks.
- There does not seem to be any analysis validating the hypothesis that there are fewer massive weights when training with MacDrop. Some demonstration of this would be nice.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see the Weaknesses.
soundness: 2
strengths: The experiments in Section 2 are very insightful. The paper's motivation is based on the empirical analysis massive activations, and then shifts to the relation between massive activations to massive weights, making the motivation very good.
summary: This paper examines the massive weights in LLMs and investigates their significance. Through targeted interventions, the authors demonstrate that these weights, while constituting only a small fraction of the total, are crucial for model performance. Building on these findings, they introduce a parameter-efficient algorithm for fine-tuning LLMs.
weaknesses: 1. The analysis of the influence of massive weights is very empirical, limiting the paper's contribution and novelty. To enhance rigor and generalizability, the authors should include a more solid mathematical analysis to better elucidate the roles of massive weights.

2. In Tables 2 and 3, the authors compare their approach with only a single vanilla LoRA/DoRA baseline, which appears insufficiently comprehensive.

3. In Table 3, the performance improvements with "w/ MacDrop" are marginal compared to the LoRA/DoRA baseline.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Discrepancy between Perplexity and Auto-evaluation Scores: There is a significant loss in perplexity with MacDrop, while auto-evaluation metrics show only minimal losses or slight improvements for some tasks. Could the authors provide further analysis on why these two metrics diverge so drastically? Additional clarification on this point could help to interpret the overall impact of MacDrop on model performance.
2. Generalizability of BOS Token Findings: As shown in Figure 2, the magnitudes associated with the BOS token do not appear to generalize to other tokens. This observation seems to contradict the underlying principle of MacDrop, which relies on massive weights associated with the BOS token to generalize across the model. Can the authors explain or justify how MacDrop remains effective given this discrepancy? Further explanation could strengthen the validity of the method across diverse contexts.
3. Using in Long Context: Specifically, when adapted to long context area, the tokens near EOS tokens usually has differential feature distrubutions compared with BOS tokens, is this case, will MacDrop still work?
4. Reproduction: Will the authors make the code for MacDrop publicly available? I would appreciate it if the code could be made accessible before the next review round.
soundness: 1
strengths: 1. Practical and Simple Method: MacDrop is presented as a straightforward, plug-and-play approach that can be applied with minimal modifications to fine-tuning pipelines.
2. High-Quality Writing and Visuals: The paper is well-written and supported by clear figures, making the complex concepts and experimental findings accessible and visually engaging.
3. Clear Experimental Results: The paper provides comprehensive experimental results across various LLM architectures and tasks.
summary: The paper investigates the phenomenon of “massive activations” in large language models (LLMs), identifying specific “massive weights” in the intermediate layers that significantly impact model performance. The authors show that these massive weights, although few in number, are highly influential during the pre-training process. To leverage this insight, the paper introduces a method called “Massive Weights Curriculum Dropout” (MacDrop), which applies dropout specifically to these massive weights during fine-tuning. By starting with a high dropout probability and gradually reducing it, MacDrop reduces the model’s reliance on massive weights, leading to improved performance on zero-shot downstream tasks and modest improvements in generation tasks. Through extensive experiments across different models and tasks, the paper demonstrates MacDrop’s effectiveness as a simple, parameter-efficient fine-tuning technique that can be easily integrated into existing workflows.
weaknesses: 1.  Insufficient Comparative Analysis with Other Techniques
The paper does not provide sufficient comparisons between MacDrop and other dropout or regularization techniques, such as Deja Vu[1] and N:M Sparsity[2]. Without such comparisons, it is difficult to assess whether MacDrop offers any distinct advantages over existing approaches. Including these baselines would make the findings more robust and allow for a clearer understanding of MacDrop’s relative performance.
2. Lack of Theoretical Justification for Massive Weights' Impact. 
The paper identifies the phenomenon of massive weights and their importance, but it lacks a strong theoretical explanation as to why these weights have such a significant influence on the model’s performance. Establishing a deeper theoretical foundation could enhance the credibility and understanding of why massive weights dominate model behavior.
3. Unclear Robustness Improvement Claims
Although MacDrop aims to reduce dependence on massive weights, the paper lacks an analysis of whether this actually leads to improved robustness. For example, evaluating the model’s performance under adversarial attacks or noisy inputs would provide evidence of whether MacDrop indeed enhances the robustness of the model. This is crucial to establish the value of MacDrop beyond improving performance in specific tasks.
4. Lack of Practical Implications for Deployment
The paper does not adequately discuss the practical implications of the identified massive weights and the MacDrop method for real-world deployment. For instance, how does the presence of massive weights affect the robustness of models in production settings? Are there potential risks for using such models? Addressing these concerns would help bridge the gap between theory and practical application.
5. Discrepancy between Perplexity and Auto-eval Scores: In Table 1, the paper reports a significant loss in perplexity with MacDrop, while auto-evaluation scores show minimal loss or slight improvement on certain metrics, which is unusual and warrants further investigation to understand the divergence in these evaluation metrics. For example, based on Llama-3-8B, the ppl of "top-5 retaining" is as large as 20.95, with which the model can't even generate a logical diaglogue  in my experimental experimence, while its average score is 71.0 (only drop 2.0), is it possible?
6. Limited Generalizability of BOS Token Magnitudes: Figure 2 shows that the magnitudes associated with the BOS token do not generalize to other tokens, which contradicts the underlying assumption of MacDrop, potentially limiting its effectiveness across different tokens and contexts.

reference:
[1] Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time
[2] Progressive Gradient Flow for Robust N:M Sparsity Training in Transformers

Meta-review: (a) Summary

The paper discovers massive weights as causes for massive activations, and proposes MacDrop, a dropout-based method targeting massive weights in LLMs, to improve fine-tuning efficiency and performance.

(b) Strengths

Identifies massive weights as causes for recent observations in LLMs; novel dropout-based fine-tuning for addressing reliance on massive activations; improvement in zero-shot downstream tasks

(c) Weaknesses

Limited experiments with larger models and diverse settings; marginal improvements in generation tasks; increased complexity from dropout curriculum (hence potential limited usage)

(d) Reasons for Decision

The limited experiments on datasets and models, and marginal gains, coupled with added complexity, do not outweigh the contributions. Not enough support from reviewers in general.

Decision: Reject