Post-Hoc Reversal: Are We Selecting Models Prematurely?

Authors: rishabh  ranjan
saurabh  garg
mrigank  raman
carlos  guestrin
zachary  lipton
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 06:35:55

Abstract: Trained models are often composed with post-hoc transforms such as temperature scaling (TS), ensembling and stochastic weight averaging (SWA) to improve performance, robustness, uncertainty estimation, etc. However, such transforms are typically applied only after the base models have already been finalized by standard means. In this paper, we challenge this practice with an extensive empirical study. In particular, we demonstrate a phenomenon that we call post-hoc reversal, where performance trends are reversed after applying post-hoc transforms. This phenomenon is especially prominent in high-noise settings. For example, while base models overfit badly early in training, both ensembling and SWA favor base models trained for more epochs. Post-hoc reversal can also prevent the appearance of double descent and mitigate mismatches between test loss and test error seen in base models. Preliminary analyses suggest that these transforms induce reversal by suppressing the influence of mislabeled examples, exploiting differences in their learning dynamics from those of clean examples. Based on our findings, we propose post-hoc selection, a simple technique whereby post-hoc metrics inform model development decisions such as early stopping, checkpointing, and broader hyperparameter choices. Our experiments span real-world vision, language, tabular and graph datasets. On an LLM instruction tuning dataset, post-hoc selection results in >1.5x MMLU improvement compared to naive selection.

Main paper: 
 introduction: 
 Many widely used techniques in deep learning operate on trained models; we refer to these as post-hoc transforms. Examples include temperature scaling (TS) [19] , stochastic weight averaging (SWA) [28] and ensembling [39] . These techniques have shown promise for improving predictive performance, robustness, uncertainty estimation, out-of-distribution generalization, and few-shot performance [4, 6, 39, 56, 84] . Typically, the pre-training and post-hoc stages are isolated. The workflow is: (1) pick model architecture, training recipe, hyperparameters, etc. to optimize for individual model performance; (2) train one or more models; (3) pick best-performing checkpoints; (4) apply post-hoc transforms. We refer to this procedure as naive selection.In this paper, we demonstrate interesting drawbacks of naive selection. In a large-scale empirical study, we uncover post-hoc reversal-a phenomenon whereby post-hoc transforms reverse performance trends between models (Fig. 1 ). We demonstrate post-hoc reversal with respect to training epochs, model sizes, and other hyperparameters like learning rate schedules. We further establish that posthoc reversal is a robust phenomenon by experimenting on real-world datasets across domains and modalities, with diverse model classes and training setups. ), but post-hoc performance is better (p 2 < p 1 ). The current practice of naive selection considers base metrics to pick models at epoch t 1 .Our proposed technique of post-hoc selection instead uses post-hoc metrics to pick models at epoch t 2 , resulting in > 2× improvement over naive selection in both test loss and error. SWA+Ens+TS refers to the post-hoc transform obtained by composing SWA, ensemble (Ens) and temperature scaling (TS).Base curves show mean of 8 runs, models from which constitute the ensembles. Individual runs are shown in lighter colors. See Fig. 5 for more detailed curves on this dataset.Post-hoc reversal is most prominent on noisy datasets (Fig. 2 ). Other phenomena exacerbated by noise include catastrophic overfitting [50] ,double descent [55] , and loss-error mismatch [19] . While these phenomena pose challenges to model development, post-hoc reversal suggests a path to alleviate them. Noise can arise not only from labeling errors, but also from inherent uncertainty in the prediction task, such as in next token prediction [60] . Indeed, severe performance degradation has limited multi-epoch training of large language models (LLMs) [81] . Here too, post-hoc reversal reveals a promising path for sustained performance improvements over longer training.The core intuition for post-hoc reversal is that models continue to learn generalizable patterns from clean examples, even when spurious patterns learnt from mislabeled examples worsen the overall performance. Post-hoc transforms exploit differences in the learning dynamics of clean and mislabeled examples [42] to reinforce the influence of the former, while suppressing that of the latter. When strong enough, this effect leads to reversal. We show evidence for these intuitions in § 5.Based on our findings, we propose post-hoc selection-a simple technique whereby base models are selected based on post-transform performance. The technique is practical as the transforms of interest can be cheaply incorporated into the validation phase of the training loop. Post-hoc selection significantly improves the performance of the transformed models, with > 2× improvements over naive selection in some cases (Fig. 2 ). In terms of absolute performance, post-hoc selection leads to > 3-point reduction in test error over naive selection on a satellite imaging dataset (Fig. 1 ). The reduction is even higher (> 5 points) when using out-of-distribution (OOD) val/test splits for the same dataset. On an LLM instruction tuning dataset, under our procedure a composed transform of SWA, ensemble and TS gives > 1.5× MMLU improvement over a naive application of the same transform on prematurely selected models.Clean (0) Aggre ( 9) Rand1 ( 17) Worst ( 40) Clean (0) Noisy (40) Label Set (Approx. Noise %) C-10/100-N) for the SWA+TS transform. On noisy label sets, post-hoc selection is often > 2× better. 
related work: 
 A slew of empirical works [10, 17, 31, 55, 57, 58] have revealed both challenges and opportunities for improving the understanding and practice of deep learning. Our work expands this list with a novel phenomenon tying together noisy data learning and post-hoc transforms. Orthogonal to our work, a number of training-stage strategies for noisy data have been proposed (see [69] for a survey).TS belongs to a family of calibration techniques [2, 19] proposed with the goal of producing wellcalibrated probabilities. Ensembling is a foundational technique in machine learning, with simple variants routinely used in deep learning [3, 39] . SWA [28] is the culmination of a line of work [18, 25] seeking to cheaply approximate ensembling. Despite their prevalence, a thorough understanding of best practices for wielding these techniques is lacking, especially in the context of noisy data. Our work fills this gap. For a more detailed discussion on related work, see App. A. 
preliminaries and background: 
 We describe our learning setup in § 3.1, with emphasis on noisy data, a key focus of this work. In § 3.2, we introduce the post-hoc transforms we study.Setup. f : Θ × X → R C outputs the logit vector z = f (x; θ), given parameter vector θ ∈ Θ. Predicted probability of class k is P f [y = k | x] = σ(z) k, where σ is the softmax function.Noise. Data D is said to beclean if P D [y | x] is one-hot for all x, i.e., P D [y | x] = 1{y = y * (x)}for some labeling function y * : X → Y. Then, for any example input x (i) in the dataset, the observed label is The exponentiated loss, also called perplexity, is common in language modeling, where it is computed on a per-token basis. A standard result states that loss is minimized if and only if the ground truth conditional probability is recovered [20] . See App. B.1 for additional background.y (i) = y * (x (i) ). When P D [y | x] is not one-hotDefinition 1 (Post-Hoc Transform) A post-hoc transform T maps a classifierf : Θ × X → Y to another classifier T • f : Θ K × X → Y, for some K.Temperature Scaling (TS). TS [19] involves scaling the logits with a temperature τ ∈ R obtained by optimizing the cross-entropy loss over the validation set, with model parameters fixed (Eqn. 1). Temperature scaling preserves error as it does not affect the predicted class. We use the torchcal [63] implementation, which optimizes the temperature on GPU with Newton's method [15] .EQUATIONEnsembling. In this method, predictions from an ensemble of classifiers are combined. In deep learning, simply averaging the temperature-scaled logits is effective (Eqn. 2). θ 1 , . . . , θ K are obtained from multiple training runs with the same architecture and dataset, with stochasticity from mini-batch sampling and random initialization, if applicable.EQUATIONStochastic Weight Averaging (SWA). SWA [28] involves averaging weights θ 1 , . . . , θ K from the same training run (Eqn. 3). BatchNorm statistics are recomputed after averaging, if required. We pick checkpoints at epoch boundaries. Unlike Izmailov et al. [28] , we do not skip the initial epochs (warmup) or modify the learning rate schedule 3 .EQUATIONCompositions. TS, ensembling and SWA can be readily composed. In particular, we consider SWA+TS and SWA+Ens+TS, for single-and multi-model settings respectively. We denote them withT S+T = T TS • T SWA and T S+E+T = T TS • T Ens • T SWA (explicit forms in App. B.2). 
post-hoc reversal: formalization and empirical study: 
 To use post-hoc transforms, one must first select models to apply them to. Current practice is to select the best-performing model independent of post-hoc transforms, rationalized by an implicit monotonicity assumption -"better-performing models result in better performance after transformation". As we shall see, this assumption is often violated in practice. We call such violations post-hoc reversal. In § 4.1, we formalize post-hoc reversal and discuss ways to detect it. In § 4.2, we empirically study various kinds of post-hoc reversal with special practical relevance.First, we give a general definition of post-hoc reversal (Def. 2). If Def. 2 holds with φ k 's which are optimal for the base metric M f , then naive selection becomes suboptimal as it picks φ k 's, but θ k 's are better under the post-hoc metric M T •f . Since the entire space of parameter tuples Θ K can be large, we study post-hoc reversal restricted to indexed parameters (Def. f : Θ × X → Y to T • f : Θ K × X → Y. T applied to f exhibits post-hoc reversal for a metric M if there exist (θ 1 , . . . , θ K ), (φ 1 , . . . , φ K ) ∈ Θ K such that M f (θ k ) ≥ M f (φ k ) for all k = 1, . . . , K but M T •f (θ 1 , . . . , θ K ) < M T •f (φ 1 , . . . , φ K ).Definition 3 (Index-wise post-hoc reversal) Let I be a set of indices and P : I → Θ K map indices to parameter tuples. When Def. 2 holds with (θ 1 , . . . , θ K ) = P(s), (φ 1 , . . . , φ K ) = P(t) for some s, t ∈ I, we call it index-wise post-hoc reversal.Diagnosis. To enable a visual diagnosis of post-hoc reversal, we define base and post-hoc curves (Def. 4) and a relaxed notion of post-hoc reversal for them (Def. 5). Post-hoc reversal is characterized by non-monotonicity between the base and post-hoc curves, i.e., there exist regions where one improves while the other worsens. This happens, for instance, when one curve exhibits double descent but the other doesn't. Different optimal indices for the two curves is another indicator of post-hoc reversal.Definition 4 (Base and post-hoc curves) The base and post-hoc curves M base , M post :I → R are given by M base (t) = 1 K K k=1 M f (θ k ) and M post (t) = M T •f (θ 1 , . . . , θ K ), where (θ 1 , . . . , θ K ) = P(t).Definition 5 (Post-hoc reversal for curves) Base and post-hoc curves M base , M post : I → R exhibit post-hoc reversal when there exist s, t ∈ I such that M base (s) ≥ M base (t) but M post (s) < M post (t). Test Error (%) Experimental setup. We focus on the CIFAR-N dataset [74] . CIFAR-10-N uses the same images as CIFAR-10 but provides multiple human-annotated label sets, allowing the study of realistic noise patterns of varying levels in a controlled manner. Clean is the original label set; Rand1,2,3 are 3 sets of human labels; Aggre combines Rand1,2,3 by majority vote; and Worst combines them by picking an incorrect label, if possible. Similarly CIFAR-100-N has two label sets, Clean and Noisy, with the latter being human-labeled. We train ResNet18 [21] models for 100 epochs with a cosine annealed learning rate. Additional details on datasets and training setup are in App. C. Fig. 3 shows test curves on CIFAR-10-N Clean, Rand1 and Worst. Other label sets and CIFAR-100-N are in App. E. For clarity, we omit the SWA base curve M base SWA (t) = (M f (θ 1 ) + • • • + M f (θ t ))/t in the plots, and simply re-use the curve M base (t) = M f (θ t ) to compare with the post-hoc SWA curve. While deviating from Def. 4, this better reflects the current practice of early stopping on the latest epoch's base metric.Observations. First, we focus on the base curves: (1) Overfitting: As noise increases, test curves go from a single descent to a double descent to a U-shaped curve with increased overfitting. (2) Double descent: Noise amplifies double descent, and the second descent worsens with increasing noise (as compared to the first). (3) Loss-error mismatch: Loss overfits more drastically than error, leading to a mismatch with higher noise. Optimal models for loss and error can be different. Finally, we discuss some benefits from post-hoc reversal: (8) Overfitting: All transforms reduce overfitting, often reverting performance degradation. (9) Double descent: SWA, ensemble and compositions flatten the double descent peak. TS, on the other hand, leads to a double descent for some cases where there was none before. (10) Loss-error mismatch: TS aligns the loss and error curves, enabling simultaneously good loss and error.Here, indices represent model sizes. Models of all sizes are trained for T epochs, large enough for convergence. Following [55] , we avoid early stopping. Notation-wise, we add a subscript to θ to indicate the model size s. Parameters are indexed as follows: P TS (s) = (θ T,s ); P Ens (s) = (θ 1 T,s , . . . , θ N T,s ); P SWA (s) = (θ 1,s , . . . , θ T,s ). Experimental setup. We parameterize a family of ResNet18s by scaling the number of filters in the convolutional layers. Specifically, we use [k, 2k, 4k, 8k] filters for width k. The standard ResNet18 corresponds to k = 64. Otherwise the training setup is same as before. Fig. 4 shows the curves. Concretely, the index set I = {2, 4, . . . , 64} is the set of ResNet widths k described above.Observations. Post-hoc transforms improve performance (up to ≈ 10 points for error) and mitigate double descent. Further, we see yet another way in which higher-capacity models are better: they give better results under post-hoc transforms even when lower-capacity base models perform better. In general, the index set I can contain any hyperparameter configurations. Here, we consider two hyperparamters: learning rate schedule and training epochs. To avoid repeating CIFAR-N epoch-wise curves, we experiment on a fresh dataset, FMoW.Experimental setup. We experiment on learning rates (LRs) and training epochs, with index set I = {const, exp, cos} × {1, . . . , T }. Here, const, exp and cos refer to constant, exponentially decaying and cosine annealed LRs respectively, and T is the total number of epochs. We train DenseNet121 [26] models on the FMoW dataset [9] which constitutes a 62-way classification of land use from satellite images. For more details, see App. C. Fig. 5 shows the curves.LR-wise observations. We see some interesting instances of post-hoc reversal: (1) constant LR has the worst base performance but the best post-hoc performance; (2) under SWA and TS (composed), the curves continue to improve at the later epochs for constant LR, but not for the decaying LRs5 .Epoch-wise observations. Epoch-wise post-hoc reversal occurs for all LR schedules. SWA and ensembling convert the double descent into a strong single descent, with approx. 10-point improvement in error for the latter. For constant LR, this also changes the optimal epoch. SWA only recovers about half of the ensemble gain, and perhaps surprisingly, ensembling SWA models is not better than ensembling alone. Pre-TS loss curves show a strong mismatch with the error curves, but TS enables simultaneously good loss and error with the last epoch models. Overall, these observations reinforce the trends gleaned from the CIFAR-N experiments. 
intuitions for post-hoc reversal: 
 In this section, we give hypotheses for post-hoc reversal, backed by experimental evidence.Ensembling and SWA delay catastrophic overfitting. Models learn generalizable patterns from clean examples, and spurious patterns from mislabeled ones. The latter causes overfitting. When noise is low, the former dominates and overfitting is benign. Otherwise, overfitting is catastrophic.Ensembling and SWA improve fitting of clean examples, and reduce memorization of mislabeled ones. When this overturns the dominance of spurious patterns, we observe reversal. ing SWA to easily revert their memorization. In App. G, we extend this analysis to ensembling and solidify the intuition further by visualizing decision boundaries on a synthetic dataset. This explanation also applies to flattening of the double descent peak, which is a manifestation of catastrophic overfitting.TS mitigates loss-error mismatch. Once a neural net has fit a train example, the cross-entropy loss on it can be lowered by simply upscaling the weights of the linear output layer. This makes the model overconfident later in training, as shown in [19] . For a mislabeled example, this leads to worse loss on similar test instances. The test error is not affected as it is independent of the scale of the logits.In high-noise settings, test loss can worsen due to memorization of mislabeled examples, even as the test error improves from continued learning on clean examples, leading to loss-error mismatch. TS fixes this by downscaling the logits. Indeed, one finds that the temperature (as obtained with a held-out set) increases with epochs (Fig. 8 ). of-distribution (OOD) val/test sets, as seen for FMoW (we use ID and OOD splits from WILDS [34] ). For some datasets, like C-100-N Noisy, post-hoc selection is only marginally better on test error. Often, in such cases, the error floor is already quite high (e.g., C-100-N Noisy has ∼ 40% noise and ResNet-18 has ∼ 10% error on clean C-100, so a test error of ∼ 50% is already impressive), and test loss is a more appropriate metric.Early stopping. We advocate monitoring post-hoc metrics for early stopping. Only a running average needs to be updated for SWA, and TS involves a quick single-parameter optimization. Further, while the base curves can fluctuate wildly between consecutive runs, SWA+TS curves are considerably smoother (see Figs. 3, 11 and 10) , making them more reliable for automated early stopping. One can similarly monitor metrics for SWA+Ens+TS under parallel training runs. 
experiments across domains and modalities: 
 In § 4 and § 6, we introduced post-hoc reversal and selection with experiments on the CIFAR-N and FMoW datasets. In this section, we supplement our experimental analysis with additional experiments across diverse domains and modalities to demonstrate the generality of our findings.Language models are pre-trained or fine-tuned with a self-supervised objective of predicting the next token in a text corpus. There might be many acceptable tokens following a given prefix, albeit with different probabilities. Thus next token prediction is noisy and one might reasonably expect to see post-hoc reversal. In this section, we test this hypothesis for the task of fine-tuning LLMs to follow instructions (instruction tuning [72] ). Instruction tuning datasets are naturally small [85] and amenable to multi-epoch training where catastrophic overfitting becomes an important concern. Recent works [53, 81] have argued for data repetitions for LLM pre-training as well, but such experiments are beyond the scope of this paper.Experimental setup. We fine-tune LLaMA-2-7B [70] on the Guanaco dataset [12] of chat completions. We evaluate perplexity and causal language modeling (CLM) error on the test set, and also the MMLU accuracy [24] to better contextualize model improvements. Fig. 9 shows the curves. Tab. 7 in App. E gives exact numbers, and App. F explores sub-epoch checkpointing. For TS, we use a shared temperature parameter to scale the logits of all tokens and leave more involved strategies like long-horizon temperature scaling [66] to future work.Observations. We observe post-hoc reversal between epochs 1 and 2 for perplexity and error, and between epochs 2 and 3 for MMLU. Both SWA+TS and SWA+Ens+TS transforms show significant improvements, much of which is only realized under post-hoc selection.In this section, we further expand our experimental coverage to text, tabular and graph classification datasets from real-world applications. Observations. Post-hoc reversal is a recurring feature across datasets, transforms and metrics. The 3 datasets show different patterns between the base and post-hoc curves, showing that post-hoc reversal can take a variety of forms.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: Please see Tabs. 6 and 8 for detailed results with standard deviations. We don't report error bars for ensembles as it is computationally prohibitive to train many independent ensembles.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
conclusion: 
 We empirically studied temperature scaling (TS), ensembling, stochastic weight averaging (SWA) and their compositions, and found that these transforms can reverse model peformance trends (post-hoc reversal). Based on our findings, we presented the simple technique of post-hoc selection, and showed that it outperforms naive selection. We validated our findings and proposals over diverse settings.Our work has broad implications for the field of deep learning. It shows that current practices surrounding the use of post-hoc transforms leave much room for improvement. This is especially true for noisy data, which is pervasive in real-world applications. Future directions include better strategies for checkpoint selection, developing a theoretical understanding, investigating impacts on scaling laws, and characterizing other instances of post-hoc reversal.Summary of practical recommendations. We advocate for the use of TS, ensembling and SWA across deep learning applications. Further, such transforms should be tightly integrated into the model development pipeline, following the methodology outlined in the paper. In particular: (1) apply SWA+TS and SWA+Ens+TS transforms for better results in the single-and multi-model settings respectively; (2) track temperature-scaled loss to overcome loss-error mismatch; (3) monitor post-hoc metrics to avoid premature early stopping; (4) make hyperparameter decisions informed by post-transform performance; (5) use post-hoc selection to pick model checkpoints. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the broader impact of our work in Sec. 8. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All datasets and models used in this paper are under permissive licenses allowing for research use. We have credited their authors by exhaustively citing sources.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: [NA] Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: [NA] Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: [NA] Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Phenomena. Empirical works like double descent [55] , grokking [58] , scaling laws [31] , neuralcollapse [57] , edge-of-stability [10] , lottery-ticket-hypothesis [17] have revealed both challenges and oppotunities for improving the understanding and practices of deep neural network training.Post-hoc reversal expands this list as a novel phenomenon regarding learning dynamics under the lens of post-hoc transforms. It is most intimately connected with double descent, offering a way to mitigate it. Some works [7, 23, 54, 59, 65, 76] show other mitigations, such as regularization and data augmentation.Temperature Scaling (TS). TS belongs to a family of post-hoc calibration techniques [2, 19, 32, 66, 83] , with the unique property of preserving classification error. Recently, calibration has been applied to large vision and language models [11, 71, 84] . While loss-error mismatch has been reported before [11, 19] , to the best of our knowledge, we are the first to report post-hoc reversal with TS.Ensembling. Ensembling is a foundational technique in machine learning, encompassing bagging, boosting, etc. In deep learning, a uniform ensemble is most popular [3, 39] , although recent work on ensembling LLMs has explored more efficient routing-based ensembles [29, 46, 48, 49] . Various works have explored strategies to form optimal ensembles [36, 47, 51, 77] , generally based on model diversity [38] , but recently Abe et al. [1] have warned against this. In contrast, our recommendation for forming ensembles relies directly on the validation performance of the ensemble, introducing no proxies, and still being computationally cheap.Stochastic Weight Averaging (SWA). SWA [28] is the culmination of a line of work [18, 25] which seek to cheaply approximate ensembling. It has inspired numerous works which average weights in some form [4, 6, 27, 41, 61, 77] often in combination with ensembling. Recently, weight averaging has shown up in the LLM space [62, 64] . While these works generally apply SWA with a fixed training time determined independently, we present SWA in the role of early stopping and model selection. In practice, SWA has often been found to be unreliable 7 , and is often skipped from training recipes even when considered [35, 75] . Our work sheds some light on this, offering a rather counter-intuitive choice of models to include in the weight average for best results.Noise. Many training strategies have been introduced to deal with noisy data (see [69] for a survey). However, the efficacy of simple post-hoc transforms has been left unexplored. Further, most of these works are motivated by labeling errors, which leaves some of the core practical considerations for dealing with general noisy data unaddressed. For instance, access to a clean validation set is assumed and test loss is overlooked as an important metric [43, 44] . We also entirely avoid experiments on synthetic noise, informed by recent work which questions the transferability of findings to realistic noise patterns [30, 73] . Some recent datasets [30, 40, 68, 73, 79 ] make it possible to study realistic noise along with known noise estimates. Noise due to insufficient information in the input context (Fig. 11 ) has also been studied under different settings, such as for RLHF [67] .Multi-epoch training of LLMs. Multi-epoch training of LLMs runs into severe catastrophic overfitting. Xue et al. [81] examine the contributing factors and explore possible solutions. They find that regularization is not helpful, except for dropout. Muennighoff et al. [53] study scaling laws considering data repetitions. Complementarily, we put forward post-hoc transforms as an effective solution with our post-hoc selection methodology. This is especially important for fine-tuning LLMs, e.g. in instruction tuning [72] , where [85] and [8] advocate for fine-tuning with a smaller amount of higher quality samples for more epochs. The explicit forms of the composed transforms SWA+TS and SWA+Ens+TS (denoted as T S+T and T S+E+T ) are given by Equations 4 and 5 respectively. For T S+E+T , parameters θ l 1 , . . . , θ l K l are weightaveraged and the L resulting models are ensembled, followed by temperature scaling. τ l is the temperature for weight-averaged models, and τ Ens is the temperature for the ensemble. As before, they are obtained by optimizing the cross-entropy loss over the validation set, with model parameters fixed. EQUATION(T S+E+T • f ) x; θ 1 1 , . . . , θ 1 K1 , . . . , θ L 1 , . . . , θ L K L = 1 τ Ens 1 L L l=1 1 τ l f x; 1 K l K l k=1 θ l k (5)Tabs. 2 and 3 summarize the datasets and training details for our experiments. They are described in detail below. We trained our models under these hyperparameters on 48 GB A6000 GPUs in a single-GPU setup, except for LLaMA-2-7B fine-tuning on Guanaco, for which we used 80 GB A100 GPUs. Single model training completes in a few hours for all datasets except FMoW and Guanaco, on which training took upto 12 hours. We experiment most extensively on the CIFAR-N datasets, where our optimized script can train a single model in 3-5 minutes on an A6000 GPU.CIFAR-N [74] . CIFAR-10-N uses the same images as CIFAR-10 but provides multiple humanannotated label sets. Clean is the original label set; Rand1,2,3 are 3 sets of human labels; Aggre combines Rand1,2,3 by majority vote; and Worst combines them by picking an incorrect label, if possible. CIFAR-100 has 2 variants, a fine-grained one with 100 classes and a coarse-grained one with 20 classes, obtained by grouping the fine-grained classes. Correspondingly, there are CIFAR-100-N Coarse and CIFAR-100-N Fine datasets. They have two label sets each: Clean and Noisy, with the latter being human-labeled. In the main paper, CIFAR-100-N refers to the fine-grained version.By cross-referencing with the original labels, it is possible to estimate the noise levels. These are shown in Table 4 .CIFAR-N allows access to clean labels. In the literature, the validation and test sets for CIFAR-N typically use the clean labels [42, 45, 78] . However, access to clean labels is a luxury only available for label noise settings. Even there, obtaining clean labels is expensive, as it requires careful expert annotation. For other sources of noise it might not even be feasible to obtain clean labels. Hence, we restrict ourselves to using noisy (i.i.d. to train) validation and test sets. Since CIFAR-N only provides human labels for the original 50k CIFAR-10/100 train images, we split these into 40k/5k/5k images for train/val/test sets.FMoW [9, 34] . This is the version of the original FMoW dataset [9] as used in the WILDS benchmark [34] . For FMoW (ID) we use the in-distribution val and test sets, and for FMoW (OOD), we use the out-of-distribution val and test sets, where the val set is shifted with respect to the train set, and the test set is shifted with respect to both the train and val sets. All splits are as provided by WILDS. The input is an RGB satellite image (rescaled to 224 x 224 pixels) and the label is one of 62 building or land use categories. The labels were obtained by a combination of human annotation and cross-referenced geographical information. The original dataset provides additional metadata about location, time, sun angles, physical sizes, etc. which is ignored in the WILDS dataset (and hence in ours). While the labels have low noise compared to the ground-truth, this dataset is noisy because of insufficient information. It is hard to disambiguate the building or land use category with full certainty by looking at the satellite image alone. See Figure 11 . Models and training setup are as used in [9, 34] , except for the LR schedule, where we experiment with multiple alternatives.Guanaco [12] . This is a subset of the OASST1 dataset [37] containing only the highest-rated paths in the conversation tree. We follow the fine-tuning setup from [12] , except that we use vanilla fine-tuning without any quantization or low-rank adapters.Yelp [5] . This is a subset of the Yelp Dataset Challenge 2015 dataset with 25k reviews in the train set and 5k reviews each in the validation and test sets. The input is a review text and the label is one of 5 classes (1 to 5 stars). Assigning a rating to a review is intrinsically non-deterministic as different reviewers might have different thresholds for the star ratings. This introduces noise in the data.Folktables [14] . Folktables consists of 5 classification tasks based on the US Census: Income, Employment, Health, TravelTime and PublicCoverage. The data is tabular. The available feature columns do not contain sufficient information to predict the targets with full certainty, even if the Census recorded the ground-truth labels with high accuracy. This results in noise.Collab and Reddit [52, 82] . These datasets are from TUDataset [52] , and were originally introduced by Yanardag and Vishwanathan [82] . Collab is a scientific collaboration dataset. The input is an ego-network of a researcher and the label is the field of the researcher (one of High Energy Physics, Condensed Matter Physics and Astro Physics). The Reddit-5k and Reddit-12k datasets (originally called REDDIT-MULTI-5K and REDDIT-MULTI-12K) are balanced datasets where the input is a graph which corresponds to an online discussion thread from the social network site Reddit. Nodes correspond to users and there is an edge if one user responded to another's comment. The task is to predict which subreddit a discussion graph belongs to. Reddit-5k is smaller with 5k examples and 5 classes. Reddit-12k is bigger with 12k examples and 11 classes. Tables 6, 7, and 8 provide detailed results for CIFAR-N, LLM instruction tuning, and other datasets respectively.Throughout the main paper, we use a checkpoint interval of 1 epoch. For small-epoch settings, such as LLM pre-training or fine-tuning, it might be better to checkpoint more frequently, at fractional epochs. In this section, we investigate the impact of checkpoint interval on the best MMLU score, and the epoch at which it is achieved, for the LLM instruction tuning setup of § 7.1.  and 14b show the results. We find that a checkpointing interval of 0.7 epochs gives the best results, with higher and lower intervals performing slightly worse. This makes sense-higher intervals include too few checkpoints for SWA, lower ones include too many weaker checkpoints from earlier in training.Also, we find that the optimal epoch is shifted further at smaller checkpointing intervals (by about 2 epochs when the checkpointing interval is 0.1 epochs), showing that post-hoc reversal is even more important in this setting. This is likely because with more checkpoints being averaged, even more overfitted checkpoints can be accomodated while still increasing the overall performance.Here, we replicate post-hoc reversal on a synthetic dataset with 2 input features, with the aim of visualizing learnt decision surfaces to solidify our intuitions. As per [3, 16] ensembling and SWA help when the data has a "multi-view" structure, or equivalently, the loss landscape has multiple modes. This is hard to achieve for 2D datasets, so instead we simulate the effect by training each MLP on a random 50% subsample of the training data. While our experiments in the main paper use the standard cross-entropy (CE) loss, here we consider two leading training objectives from the label noise literature: (1) SOP [45] and (2) ELR [42] . Tables 9, 10 and 11 compare naive and post-hoc selection strategies for CIFAR-N datasets under CE, SOP and ELR losses respectively. Here again we find that post-hoc selection is superior to naive selection in general. We also note that the differences between CE, SOP and ELR are minimal. This is likely because we use i.i.d. (and therefore noisy) validation and test sets, unlike the original papers which use clean validation and test sets. We find post-hoc reversal to be an important phenomenon when the base curve exhibits performance degradation due to overfitting. However, under some scenarios, the base curve shows a monotonic improvement in performance with additional training (or increasing model size). Examples include:(1) the data has low noise, (2) the training is heavily regularized, and (3) there is an abundance of data, so that a single data point is not repeated enough to cause overfitting. In such cases, post-hoc selection outcomes are similar to naive selection. Since our suggested approach only ensembles models trained for the same number of epochs during post-hoc selection, it does not subsume the naive selection search space, leading to marginally worse performance sometimes, although this can be easily overcome in practice.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: Please see code (including instructions to download data and reproduce the main results) at https://anonymous.4open.science/r/post-hoc-reversal.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material.• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).Thus, our variant of SWA is hyperparameter-free.Ensembling models from possibly unequal epochs is covered in § 6Possibly due to higher model variance with constant LR, beneficial for both ensembling and SWA.Results. Tab. 1 compares naive and post-hoc selection strategies for CIFAR-N and FMoW. Except for some clean label sets, post-hoc selection is always better than naive selection, often with > 2× improvement from post-hoc selection as compared to naive selection. It remains effective with out-6 Alternatively, one can select Tj = arg minT M val T S+T •f (θ j 1 , . . . , θ j T ) as a hybrid between post-hoc selection (within runs) and naive selection (across runs).See, for example, discussion at https://discuss.huggingface.co/t/improvements-with-swa/This may be attributed to (1) picking the same epoch for all runs in post-hoc selection, and (2) generalization error between validation and test sets for the selected epoch. 


Decision: **Summary**

This is an empirical study that shows that performance trends observed during training might reverse after post-hoc transforms like temperature scaling or stochastic weight averaging. This leads to the practical recommendation of *post-hoc selection*, i.e. to choose intermediate checkpoints based on their contribution to post-hoc performance rather than individual-run performance. Results are surprising, e.g. sometimes this method favors overfitting on individual runs.

**Recommendation**

This work is interesting for the research community and has some practical implications. I am happy to support its acceptance.

Reviews: review: paper_summary: paper_summary: The paper discusses the phenomenon of *post-hoc reversal*, where the performance trend is reversed after applying post hoc transforms, namely, temperature scaling (TS), stochastic weight averaging (SWA), and ensembling (Ens).

The paper conducts an empirical study to observe the phenomenon across different epochs, model sizes, and hyperparameters.

They propose a *post-hoc selection* strategy where the optimal epoch/model size/hyperparameters are selected considering the performance after applying post-hoc transforms.

The paper focuses on the noisy data setting and shows experimental gains in that setting for some datasets. 

Experiments have been conducted for the FMoW dataset, CIFAR-10-N, and CIFAR-100-N, as well as some additional experiments on text, tabular, and graph datasets.
strength_weakness: strength_weakness: 1. The paper proposes *post-hoc selection* strategy, simply selecting the optimal epoch (or model size, etc.) after applying already existing (and widely used) post-hoc transforms (SWA, En, TS). They do not propose any *novel* mechanism to tackle *post-hoc reversal*.
2. It is a findings paper; the study is empirical and lacks theoretical insights. Given the empirical nature of the study, the performance gains in some datasets are marginal (e.g., C-100-N Noisy test error, Table 1).
3. For the datasets in Table 5, the post-hoc selection strategy does not consistently provide performance gains, i.e., it performs worse in some cases.
questions: questions: **Questions**
1. Why is post-hoc reversal prominent in the noisy data setting? Can the authors provide intuition or reasoning (other than empirical observations)?
2. Can the authors provide intuitions to (at least) some of the observations? For example, why do SWA and Ens handle double descent, but TS causes it?
3. In Figure 7, what is the SWA ensemble? It has not been discussed in Section 5.
4. For the datasets Yelp, Income, and Reddit-12K, the post-hoc reversal is observed in Figure 7, but the post-hoc selection strategy either shows worse performance or marginally better. Does this indicate that the post-hoc selection strategy is ineffective in these settings? Can the authors provide any reasoning/explanations?

**Suggestions**
* The paper title should include "noisy data setting", as the paper focuses on the noisy data setting.
* Include intuitions/reasonings/explanations (wherever possible) for the observations in the experimental results.
limitations: limitations: Yes.
paper_summary: paper_summary: The authors investigate applying three post hoc transforms, namely temperature scaling (TS), ensembling, and stochastic weight averaging (SWA), to trained models after training, separately and jointly. They oppose their approach to the "naive approach" and provide empirical observations of a phenomenon they refer to as "post hoc reversal."  This phenomenon corresponds to the performance trends of models that change after these transformations. In particular, they focus on the observation in noisy settings of the following observations.

1- Epoch-wise post hoc reversal. The authors show that post hoc transforms reduce overfitting, double descent, and loss-error mismatch. In particular, SWA and ensembling reduce overfitting and flatten double descent peaks.

2- Model-wise post hoc reversal. The model's size influences the improvement in the performance of the post hoc transforms, i.e., larger models tend to perform better under post hoc transforms.

3- Hyperparameter-Wise Post Hoc Reversal. It appears that constant learning rates improve performance better than decaying learning ones and the optimal number of epoch shifts. 

The experiments are broad, and metrics show good non-trivial improvement based on the method.
strength_weakness: strength_weakness: The main weakness of the approach, which is recurrent in most current empirical LLM observations, is that understanding the source of the improvement needs to be clarified. 

In fact, it is not clear why the benefits of post hoc transforms are more pronounced in high-noise settings.

Making assumptions to verify potentially synthetic and controlled datasets could be helpful. As of one, this corresponds to an observation.
questions: questions: If the author can provide a good answer to the question above that would address my concerns.
limitations: limitations: Yes
paper_summary: paper_summary: This paper shows empirical evidence that common approaches to model selection can be improved upon by less greedy alternatives. In particular, authors highlighted what they referred to as post-hoc reversal, when post-training transformations reverse the trends observed in independent training runs. For instance, in certain overfitting situations, transformations such as temperature scaling, weight averaging, and ensembling, reversed growing test error trends and closed the generalization gap to a greater extent than common model selection approaches. This observation then yielded practical recommendations of incorporating post-training transforms into the model selection pipeline, which resulted in improved performance across a number of settings, as well as on smoother error curves that render model selection less noisy. Authors attributed post-hoc reversal to the high variance involved in the training of neural networks, which can be smoothed out by some of the studied transformations. The paper lays out clear practical recommendations as a result of the reported empirical observations, which I believe would be highly relevant to the community.
strength_weakness: strength_weakness: - This requires no action from the authors but just for the record, I have mixed feelings about the presentation choices regarding the use of notation. The text is notation heavy, which makes it precise, but at the cost of readability. The discussion seems simple enough to enable the use of almost plain text only. But I reiterate that I consider this to be mostly matter of writing style, and I expect no effort from authors addressing this comment.

- The bulk of the empirical assessment focuses on a somewhat small scale setting involving variations of the Cifar-10 dataset and relatively small neural networks. While this is obviously due to the high cost involved in replicating these experiments in larger scale settings, it limits the strength of the presented empirical evidence. However, section 6 adds results for other settings including fine-tuning of very large models, which address this concern to an extent.

- The experiments focus on multi-epoch training settings, and it's a bit unclear how those results transfer to now common pre-training situations where very few epochs are used. While one can replace epoch-end checkpoints by checkpoints obtained every k steps, it's unclear how the choice of k affects results, for instance.

- One component that seems to have been left out of the analysis is robustness. Does post-hoc reversal still happens in situations where the test set is somehow shifted relative to the validation set used for post-hoc selection? It might be the case that there's not so pronounced of a gap between post-hoc and greedy/naive selection in such a scenario.
questions: questions: - Would it be possible to replicate a subset of the results reported in section 4 and 5 for a larger scale dataset (e.g., ImageNet)?

- Does post-hoc reversal still happen in situations where the test set is somehow shifted relative to the validation set used for post-hoc selection?

- How does one leverage post-hoc selection in a non-multi-epoch training situation?
limitations: limitations: Please refer to the "Weaknesses" section.

Comments: title: Acknowledgement to Author's Rebuttal
comment: Thank you for the clarifications and extra results.
Thanks to the authors for the detailed response.

Most of my queries have been addressed. I do not have any further questions at this point.