Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors are interested in structured pruning of generative language models. In particular, this work builds on top of [Movement Pruning](https://arxiv.org/abs/2005.07683) and [Block movement pruning](https://arxiv.org/abs/2109.04838) to prune entire structures (and not individual weights) in models similar to GPT2 and the fine-tuning stage.

The authors first notice that current structure pruning applied to decoder only language models on NLG tasks perform relatively similarly at pruning rates between 10% and 50% (percentage of remaining weights), and more surprisingly, similarly to random pruning.

The authors then introduce two fundamental measures of redundancy called “sensitivity” and “uniqueness” which respectively measure how much a group of parameters impact the training objective and how unique a group of parameters is compared to other groups of parameters. This motivates the introduction of Globally Unique Movement, a method that essentially encourages remaining neurons to be dissimilar (as measured by cosine similarity), by modifying the regularization in movement pruning.

strength_and_weaknesses: Strengths:
- The problem is relatively well-motivated and the paper makes a noticeable effort to be didactic.
- The contributions are somewhat novel, and experiments are well conducted on reasonably large setups.
- The result about gradual pruning performing similarly to previous state-of-the-art structured pruning is surprising and insightful.

Weaknesses:
- The numbers showing the superiority of the method are weak or show only weak improved performance or trends.
- The connection between sensitivity/uniqueness and fine-pruning performance is not well articulated (see questions).
- I have doubts about whether the choice of benchmarks is the most appropriate. For instance, the authors note that on E2E, pruned models perform better than the non-pruned baseline.

clarity,_quality,_novelty_and_reproducibility: - Presentation suggestions:
  - Does Top_v throughout the presentation refer to Hard movement pruning in Table 2? If so, it would help the reader to uniformize these short names.
  - I have found not intuitive to present results with GUM (Figure 1 and 2) before introducing GUM
- Why is the sensitivity decreasing as the model is pruned? I would have expect that the remaining weights have a greater impact on the final output and thus as you prune, the average sensitivity increases.
- Section 3.2: you mention a few times that poor sensitivity and uniqueness explains poor benchmark performance. Is it a causality link? A correlation? Could you articulate the intuition behind?
Examples:
  - *“Magnitude pruning universally scores worst on both metrics, explaining its poorer performance in all experiments”*
  - *“which partially explains why distillation improves it significantly”*
  - *“explaining its superiority across various tasks”*
- What is the maximum number of epochs you tried for block soft movement pruning? In my experience, this method requires a very slow pruning (5x to 10x more steps than hard weight movement pruning).
- What are the trends for extreme pruning (i.e. less than 10% leftover) for Table 2, 3 and 4, 5? Does it lead to more unequivocal results?
- *“Random pruning obtains similar distillation sensitivity and uniqueness, though slightly lower, to hard movement, lending credence to its overall high performance. However, sensitivity is markedly lower without distillation as is reflected in all tables. We point to this as proof that hard movement does not target uniqueness.”* Could you expand on this insight? I have found it to be a very generous conclusion from Figure 1 and 2. Handholding the reader for that conclusion might bring some clarity.

summary_of_the_review: This paper extends movement pruning and block movement pruning by introducing a "uniqueness" metric in the training objective which encourages the model to prune similar neurons.
While the method and insights are relatively novel, the numbers and comparisons of previous pruning methods are somewhat weak.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper systematically studied the task of structured pruning of generative language models. It tests the performance of several different pruning methods on GPT-2 and GPT-Neo. The author also proposes two redundancy measures for each neuron in the MLP layer and shows that there is a strong correlation between these measures and the performance of pruning methods. Based on the observation, the author proposes a new pruning method, named GUM. The method combines a uniqueness regularization and a global top_v strategy to achieve strong performance on several pruning tasks.
strength_and_weaknesses: Strength:
1. The paper conducted a systematic evaluation of several structured pruning methods on the generative language model.
2. The proposed redundancy measures are intuitive and are shown to have a strong correlation with the pruning results.
3. The newly proposed GUM achieves strong performance when compared to the baseline methods.

Weaknesses
1. Most of the experiments focus on the WikiSQL and Wikitext tasks. More experiments and diverse tasks are needed to prove that the GUM can generalize to different tasks.
2. The result of E2E NLG is confusing. Although the author provided some explanation, it's still unclear what point the experiment wants to make. If the E2E NLG is not a good task to test the method, another text-to-text task should be used.
clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow. The proposed measures and GUM are intuitive. However, the paper didn't provide an explanation of why the soft movement method scores poorly on uniqueness and saliency. It's also unclear why the GUM only has regularization on uniqueness, but not saliency. The lack of sensitivity seems to be a more severe problem (much lower than the uniqueness) according to figures 1 and 2. 
summary_of_the_review: Overall the paper proposes two interesting redundancy measures for pruning and a promising method GUM. But the current experiment section can be further improved and more analysis on the movement methods can be done.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors conduct a study of structured pruning methods on natural language generation tasks. They demonstrate that the prior art do not improve significantly over the naive random pruning baseline. Then the authors attempt to understand the results through two measures that they introduce: sensitivity and uniqueness. The insights from these two measures allow the authors to construct a new pruning method, GUM, that explicitly enforces uniqueness through a cosine similarity proxy of the measure. The methods are implemented on large GPT-like models on a collection of downstream tasks.
strength_and_weaknesses: Strengths:

1. The topic of compressing LLM for generation tasks is an important one and I am happy to see that the authors focused on it.

2. The introduced sensitivity and uniqueness measures will be useful to future work in the area. 

3. The experiments are thorough and demonstrate meaningful trends.

Weaknesses:

1. There are marginal improvements and not all of them are consistent. That is OK, but it probably signals that better methods than GUM can be constructed. I am curious to learn what the opinion of the authors is.

2. What happens if we use these pruning methods on non-generative tasks? I think it would be good to see that experiment in the paper, because the GUM method, by construction, is agnostic to the type of downstream task.

Minor:

* The results for the E2E NLG Challenge are not linked in the paragraph in the main text. Please link to Table 7.
clarity,_quality,_novelty_and_reproducibility: The paper very clearly explains the main idea, methods and experiments. The experiments are well-chosen and executed, i.e. of good quality. There are original aspects of the work, e.g. the focus on sensitivity and uniqueness.
summary_of_the_review: There are useful contributions in this work, which I think will benefit the community. The work is very well-presented too. The topic of pruning for natural language generation is an important one, in my opinion, so I recommend weak acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper investigated pruning for generative language models. They compared several existing pruning methods on decoder-only language models which had not been empirically evaluated before. The authors found that popular pruning methods such as movement pruning do not perform robustly on causal language models. In contrast, random pruning was shown to be a strong baseline. The authors propose two metrics to analyze the effectiveness of pruning. Based on these insights, they propose a new pruning method (GUM) which considers uniqueness of neurons. They evaluated this method on three benchmarks and showed some improvement over the prior methods. 
strength_and_weaknesses: Strengths:
- This paper studies an important problem, i.e. pruning LLMs, for increasing the accessibility of LLMs as they keep growing larger and larger. This problem is also under-explored so far where both empirical results and theoretical understandings are lacking. 
- This authors carefully analyzed existing methods through neuron redundancy metrics which shed lights on their performance in pruning. 
- The proposed method is novel and interesting.   
- The writing is clear and easy to follow. 

Weaknesses:
- The evaluation tasks and datasets do not have full coverage of how generative LLM are often evaluated. The authors evaluated on three tasks: language modeling (wikitext-103), text-to-code generation (wikiSQL), text-to-text generation (E2E NLG challenge). There are a couple of problems. First, E2E NLG challenge is not very diagnostic as the authors found that "E2E NLG challenge is not very diagnostic " and "speculate these discrepancies are due to the open-endedness of the problem domain and underfitting the data." I would expect the authors to evaluate on a more robust text-to-text task (e.g. summarization) to derive more informative evaluation results. Second, besides language modeling and text-to-code generation, LLMs are often used for language understanding and reasoning tasks, e.g. those evaluated in the GPT-3 paper. Experiments on pruning should be conducted on some of those tasks. 
- The proposed approach computes cosine similarity for any two neurons' outputs. Although it's a sound solution, it is computationally expensive. Also, I wonder whether the running cosine similarity has high variance and how it affects the effectiveness of pruning. 
- Improvements from GUM are minor especially on language modeling. 
- An ablation on components and design choices of the proposed method is missing, although it's mentioned in the abstract. For example, how does GUM compare to LUM (Locally Unique Movement)?
clarity,_quality,_novelty_and_reproducibility: Overall, the paper sets out to tackle an important problem and provides comprehensive empirical results. The proposed method is novel and interesting although it's not well justified whether the minor improvements outweigh the extra complexity introduced. For experiments completeness, the authors should evaluated on other more common tasks/datasets as is often experimented with causal language models. 

Some of the experimentation details could use more clarifications:
- The authors could provide more motivation around why pruning is only performed on MLP layers.
- Different from other methods, GUM does not benefit from combining with distillation on language modeling (Table 4 & 5). Any insights on why?
- Could you provide comparison on training speed (GUM vs. other methods)?
- The improvement from GUM are small in most cases. Are they statistically significant? Could you provide results from multiple runs with different seeds and show the variance?
summary_of_the_review: This paper studies an important problem and provides some novel insights. The analysis and empirical evaluation part is mostly well executed except that the evaluation tasks/datasets could be expanded to improve representativeness and coverage. The effectiveness of the proposed method is not very clear with current results due to 1) improvements are small and may not be statistically significant. 2) the proposed method has high complexity which may slow down training and consume more memory. More clarity on both and some justification on whether the improvement outweigh the complexity would make it a strong paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies structured pruning (wherein neurons are pruned out) in generative models. The strength of the paper is the application to generative modeling, and a (somewhat) novel finding that random pruning works well. The main weakness is the lack of comparison against baselines other than movement pruning. For example Wang et al. 2019 (https://arxiv.org/abs/1910.04732) also study pruning of generative models using relaxations of discrete variables.
justification_for_why_not_higher_score: The complete lack of comparison against Wang et al. 2019 who tackle the exact same problem (i.e., structured pruning of generative LMs) makes it difficult to justify accepting this paper.
Decision: Reject