Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: High Potential (HP): a submission which meets the reviewing criteria and has potential to make an impact on the field
review_content: summary_of_contributions: Summary: The authors experiment with augmenting a previously established one-shot pruning method for large language models, SparseGPT, with a fine-tuning step. They find that doing so improves the tuned models' accuracy as measured by perplexity scores on the WikiText2 corpus.
strengths_and_weaknesses: Strengths:
- The fine-tuning idea proposed by the authors is intuitive and worth exploring.

Weaknesses:
- Would need extensively more validation in terms of datasets and varying number of retrain steps to be publication ready.

suggested_changes: - Define perplexity.
- Be clearer about what fine-tuning is, and give a sense for when fine-tuning becomes a complete retrain.
review_title: null

Meta-review: metareview: This paper explores pruning techniques for language models, building on the one-shot pruning method SparseGPT. The authors focus on the effect of fine-tuning after applying SparseGPT, and in particular consider either

- pruning with SparseGPT once, then fine-tuning for a small number of iterations
- iteratively pruning with SparseGPT, fine-tuning, then pruning again and fine-tuning again, repeated over a number of iterations and increasing sparsity levels.

The authors evaluate these approaches by pruning Meta OPT models and fine-tuning them on the WikiText2 corpus, and demonstrate that iterative pruning and fine-tuning with SparseGPT performs very well relative to the other approaches, in addition to replicating findings from previous work. They also provide many details on their experimental results and promise to release code, which seems very useful for reproducing these findings.

Reviewer aZDA found the idea "intuitive and worth exploring" and noted the high potential for impact, although they also noted that a few terms could be clarified and that experimental validation could be more thorough.

Overall I believe this work definitely meets the criteria for Clarity, Correctness, and Reproducibility. The results also seem potentially impactful for enabling more accessible LLM research, and I was impressed by the detail given about the empirical results, so I recommend this work be invited to present as a notable submission.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Invite to present (notable)