Venue: NeurIPS 2022 Workshop HITY

Reviews:

final_score: rating: 1: accept
review_content: This paper demonstrates that averaging the weights of a vision or language model over the last $k$ checkpoints can speed up training in the middle phase. The introduced method is called LAtest Weight Averaging (LAWA) and the paper explores different degrees of freedom, e.g. number of averaged checkpoints and type of average operation. LAWA's benefits are measured in time savings of GPU hours. The paper also discusses directions for future work.

---

I think the paper would benefit from a short "Conclusion" section at the end that restates the most important findings.

---

Miscellaneous comments:
- Fig. 2 is referenced before Fig. 1. Maybe swap them?

review_title: Middle-phase training improvements via weight averaging

final_score: rating: 1: accept
review_content: The paper proposes an algorithm called LAWA (LAtest Weight Averaging) which keeps a separate model that is an average of the parameters of the latest $k$ model checkpoints. This LAWA model provides a significantly better predictive performance in the middle of the training process.

The experiments are presented clearly and show that LAWA provides better predictions in the middle of training compared to the baselines.

Feedback:
- Perhaps it would be better to not talk about training speed improvements or that LAWA "speeds up convergence" (caption of Figure 1). LAWA doesn't reduce the time it takes these models to converge. However, LAWA improves the any-time performance of a model (while still in training). This shouldn't diminish the contributions of the papers, but I think would be more precise.
review_title: Applying weight averaging to the last $k$ checkpoints during training provides better any-time performance.

Decision: Accept