Title: Inconsistency-Aware Minimization: Improving Generalization with Unlabeled Data
Venue: NeurIPS 2025
Accepted: no
Keywords: Generalization

Regularization

Training Method

Deep Learning

Inconsistency
Abstract: Accurately estimating the generalization gap and devising optimization methods that generalize better are crucial for deep learning models, particularly in both theoretical understanding and practical applications. The ability to leverage unlabeled data for these purposes offers significant advantages in real-world scenarios. This paper introduces a novel generalization measure, termed $\textit{local inconsistency}$, developed from an information-geometric perspective of the neural network's parameter space; a key feature is its computability from unlabeled data. We establish its theoretical underpinnings by connecting local inconsistency to the Fisher Information Matrix (FIM) and the loss Hessian. Empirically, we demonstrate that local inconsistency not only correlates with the generalization gap but also exhibits characteristics comparable to $\textit{sharpness}$. Based on these findings, we propose Inconsistency-Aware Minimization (IAM), a regularization strategy that incorporates local inconsistency. We demonstrate that in standard supervised learning settings, IAM enhances generalization, achieving performance comparable to existing methods such as Sharpness-Aware Minimization (SAM). Furthermore, IAM exhibits notable efficacy in semi-supervised learning scenarios, where the local inconsistency regularizer is computed from the unlabeled data portion to further improve model performance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed inconsistency-aware minimization, a new method that restricts the change of the neural network model in the output function space to improve the generalization ability. The method uses the KL divergence under a perturbation to measure the local inconsistency, which can be estimated using unlabeled data in practice. Experimental results on CIFAR 10 and CIFAR 100 confirm that the proposed method can improve the generalization performance over existing methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The proposed metric of local inconsistency and the IAM method provide an interesting perspective on the generalization of neural networks. The core strengths are:\n1. The new IAM methods show empirical improvements upon classical SAM methods across various settings. Especially, the author shows the subgroup behaviour in Figure 1, which is nice to understand why local inconsistency could be a promising replacement to sharpness. \n\n2. The proposed method can be computed using the cheap unlabeled data, making it a promising approach to measure and improve generalization in practice. \n\nWeakness: Overall, the improvement upon existing works needs to be further justified. \n1. A similar inconsistency measure has already been studied previously in [1], and it is unclear whether the proposed metric has advantages in practice. It would be good to make an empirical comparison to this metric in addition to the current high-level justification. \n\n2. The authors have mentioned several drawbacks of using sharpness as the metric for generalization. While Figure 1 is a nice illustration for showing the difference, it would be good to provide some intuition to understand what makes the two metrics different in a geometric perspective, and to what extent it can solve the drawbacks of sharpness in the literature. \n\n[1] R. Johnson and T. Zhang. Inconsistency, instability, and generalization gap of deep neural\n395 network training, 2023. URL https://arxiv.org/abs/2306.00169\n\nQUESTIONS:\n1. The IAM algorithm seems to be computationally more expensive than SAM in [1], it is good to include the computation comparison, given that the performance improvement is rather marginal. In particular, it seems that Algorithm 1 in [1] also requires one trained model only. Can you explain more about the claim in lines 137-138?\n\n2. Is there a reason why KL divergence is preferred to measure the inconsistency instead of other metrics, such as L2 distance? \n\n3. How many benefits can we gain from increasing the number of iterations K in Algorithm 1? It would be nice to provide some experimental evidence on this. \n\n[1] P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur. Sharpness-aware minimization for efficiently\n374 improving generalization, 2021\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- Introduce a novel generalization metric called local inconsistency, defined via small perturbations of the model parameters $\\theta$.\nTheoretically, connect local inconsistency to the largest eigenvalue of the Fisher information matrix.\n- The idea of estimating inconsistency from unlabeled data involves multiple models on the same dataset with different random seeds, proposed by Zhang et al. [15], with extensive theoretical analysis and experiments.\n- Propose the Inconsistency-Aware Minimization (IAM) framework (with variants IAM-D and IAM-S) to directly reduce local inconsistency, which is an alternative method to measure generalization compared to the sharpness measure used in SAM.\n- Evaluate on the CIFAR dataset, comparing IAM with SGD and SAM in various label settings.\n\nSTRENGTHS AND WEAKNESSES:\n## Strength: \n- Introduce new generalization measurement, termed local inconsistency, based on model theta and its perturbation. \n- Wide range of experiments from full labelled, limited sample, and no labelled sample.\n- The paper is easy to follow.\n\n## Weakness:\n- The novelty of IAM in the fully supervised setting is limited: the “inconsistency” term was already introduced and analyzed in Zhang et al. [15], with extensive theoretical analysis and experiments.\n- The authors proposed the $S_p$ that closely mirrors FisherSAM [1] criterion—both measure small-perturbation effects via KL divergence in output space. Even though the perspective leading to this term is not exactly the same, it should be discussed and compared in this work.\n- The update form of IAM-D in formula 6, which perturbs model parameters, is similar to Virtual Adversarial Training (VAT) [2], which perturbs in the input space. \n- While IAM shows promise for semi-supervised and contrastive learning (with few or no labels), the paper fails to discuss or compare its methods against existing work in those areas. Authors only compare IAM to SAM and SGD, which are not strong competitors in those areas.\n- There is no discussion and analysis on the theoretical superiority of IAM (convergence rate and training time compared to other optimization methods)\n\n[1] Kim, Minyoung, et al. \"Fisher sam: Information geometry and sharpness aware minimisation.\" International Conference on Machine Learning. PMLR, 2022.\n[2] Miyato, Takeru, et al. \"Virtual adversarial training: a regularization method for supervised and semi-supervised learning.\" IEEE transactions on pattern analysis and machine intelligence 41.8 (2018): 1979-1993.\n## Suggestion:\n- Apply this method in case of using extra unlabelled samples for training, in which the labelled dataset is ImageNet and the extra dataset is unlabelled (could be CIFAR or Food101, etc) to enrich the training process\n- Analyze this method from other perspectives, such as noisy label or out-of-distribution, which should be more reasonable\n\nQUESTIONS:\n- The authors state IAM-D uses CE on the labeled subset and local inconsistency on both the labeled and unlabeled subsets. What is the exact setting of this experiment (in terms of how these losses are balanced, the schedules for applying each term, etc)?\n- In Self-Supervised Learning (SSL), SGD is the common optimizer to evaluate and is a reasonable baseline. However, modern self-supervised methods often benefit from optimizers like AdamW or SogCLR-style, which often yield stronger generalization.\n- Experiments are limited to CIFAR variants; including larger benchmarks (ImageNet, Food101, etc.) would better demonstrate scalability.\n- After SAM, many approaches have been proposed to improve SAM, and also been pointed out that the sharpness measurement in SAM is not reliable. It would be more appropriate to compare IAM against other works such as ASAM, GSAM, LookSAM, fSAM, etc\n-  Key methods such as FisherSAM—which measures parameter perturbation via the Fisher information—are absent from comparisons and discussion, despite their conceptual overlap with the  $\\delta$ term.\n- The author discussed in section  5.4 that IAM implicitly minimizes output entropy and acts like a regularization, but no experiments currently validate this effect, and compared to the mentioned methods (LR, ER).\n- Odd choice of hyperparameters for $K, \\beta$, and $\\rho$, which are based on grid search and only give the value for the best performance without a sensitivity analysis.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes using the KL divergence between the softmax output of a model at a given input and the output at the same input under perturbed model parameters as a proxy for generalization. A smaller KL divergence indicates greater output stability under weight perturbations, which is taken to reflect better generalization. The proposed objective minimizes this divergence, and since it does not require ground-truth labels, it can be applied to unlabeled data. Empirical results demonstrate that this inconsistency measure correlates with the generalization gap, and that training with the proposed regularizer reduces test error compared to both standard SGD and Sharpness-Aware Minimization (SAM).\n\nSTRENGTHS AND WEAKNESSES:\nThe paper tackles the interesting problem of incorporating Fisher Information into the understanding of generalization, while also addressing the important challenge of integrating unlabeled data into performance evaluation.\n\nHowever, the primary weakness lies in the motivation for the proposed measure. The Fisher Information Matrix (FIM) is a classical statistical tool used to identify datasets that enable the most confident parameter estimation by maximizing Fisher information. The authors express the KL divergence (the inconsistency measure) via a quadratic approximation using the FIM, which is further approximated by the Gauss-Newton matrix, which is an approximation of the Hessian that can be computed without labels, relying solely on the model’s softmax outputs. The maximum of this measure is then represented by the largest eigenvalue normalized by the maximal perturbation radius, computed efficiently via one step of the power iteration algorithm.\n\nAt line 135, the paper states that high instability associated with a high FIM implies uncertainty in the model’s predictions. This is conceptually incorrect: a high Fisher Information Matrix actually corresponds to greater certainty in parameter estimation, not the opposite and it is unclear how this even relates to the predictions uncertainty.\n\nOverall, the motivation remains unclear. If the focus is on unlabeled data, then the Hessian (as a curvature measure) can already be estimated without labels. If the goal is to link this to entropy (as discussed in section 5.4), it appears to contradict the established entropy minimization approaches in semi-supervised learning [1]. Alternatively, it could be viewed as a modification of SAM, where the maximum loss is found using second-order information.\n\nThe theoretical connection between the proposed measure and the generalization gap is based on PAC-Bayes bounds and leverages the known relationship between the loss Hessian and PAC-Bayes generalization guarantees.\n\nEmpirically, the proposed measure is evaluated only against the Hessian trace and maximum eigenvalue, without comparison to more sophisticated metrics such as ASAM or relative flatness. Additionally, the semi-supervised experiments do not benchmark against state-of-the-art approaches like consistency regularization, pseudo-labeling, or entropy minimization.\n\n[1] Grandvalet, Y., & Bengio, Y. (2004). Semi-supervised learning by entropy minimization. Advances in Neural Information Processing Systems, 17.\n\nQUESTIONS:\n1 - How the representation of softmax allows to use it as distribution in FIM, while it is a pseudo-distribution?\n\n2 - How the Hessian is computed in the experiments comparing correlation with generalization gap?\n\nLIMITATIONS:\nThe interchangeable usage of FIM and Hessian and its approximations requires more precise mathematical justification.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel generalization measure called Local Inconsistency, defined as the maximum expected KL divergence between model outputs and parameter perturbations $\\delta$ within an $l_2$​ ball of radius $\\rho$. The local inconsistency can be computed by using only unlabeled data and a single trained model. Besides, the authors theoretically link local inconsistency to the Fisher Information Matrix (FIM) and loss Hessian, showing it approximates the scaled maximum eigenvalue of the FIM. Finally, the authors propose a method called Inconsistency-Aware Minimization (IAM) that incorporates local inconsistency into the objective and demonstrate its effectiveness in standard supervised learning settings, semi- and self-supervised learning scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This paper makes contributions both to theoretical understanding by proposing a novel generalization measure termed local inconsistency that correlates to the generalization gap and to practical applications by proposing a method called Inconsistency-Aware Minimization (IAM) that incorporates local inconsistency into the objective.\n2. The proposed local inconsistency can be computed by using only unlabeled data and a single trained model, making it practical in real applications. \n3. The proposed IAM method is verified in supervised, semi- and self-supervised learning settings, which can be applied in wide scenarios.\n\nWeaknesses:\n1. Theorem 1 is an informal version with unclear assumptions, which may limit its applications. It would be better if the authors could provide a formal version with clear assumptions.\n2.  Datasets in the experiments may be small and easy (CIFAR10 and 100), large datasets such as ImageNet can be considered.\n3. According to the results reported in Table 1, the improvement on CIFAR 10 is marginal. Given that the authors do not report the running time comparisons, it remains questionable whether such a modest improvement justifies the additional cost.\n\nQUESTIONS:\n1. Under what conditions will Theorem 1 hold? Can you give specific conditions to make Theorem 1 clearer and more understandable?\n2. Since the performance improvement on CIFAR 10 is marginal, what is the computational cost of the proposed IAM method?\n3. In semi-supervised learning (Table 2), why does SAM outperform IAM-D on CIFAR-10 at the 20% label rate?\n4. Algorithm 1 initializes randomly, how does the choice of $\\sigma^2$ affect the performance of IAM?\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces _local inconsistency_, a new notion for measuring the generalization error of trained models. Interestingly, local inconsistency does not depend on label data, making this notion usable for self-supervised and semi-supervised learning settings. The authors discuss connections between this notion and the maximum eigenvalue of the Fisher Information Matrix (FIM). They further present a PAC-Bayes generalization bound that depends on local inconsistency. Building on this result, they propose a fast method to approximate local inconsistency inspired by power methods. The authors introduce two new algorithms, IAM-D and IAM-S, which aim to minimize local inconsistency in addition to the empirical risk. In the end, the authors empirically show how consistently local inconsistency is correlated with training error, and also demonstrate the competitive performance of IAM-D and IAM-S on image datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- While the notion of local inconsistency is fairly simple, the experiments show that it is more effective in correlating with generalization error compared to the trace and maximum eigenvalue of the Hessian, especially in large models.\n- The paper is very well written and easy to follow. The authors discuss the connection between local inconsistency, Fisher information, and inconsistency.\n\nWeakness:\n- Despite the multiple levels of approximation, estimating local inconsistency—and hence each iteration of the IAM-D and IAM-S algorithms—still requires $K$ additional gradient evaluations, which is computationally costly.\n- Since local inconsistency and effectiveness of IAM-D and IAM-S are supported mostly experimentally, I would recommend a broader comparison to more recent SAM methods, for example, Li et al. [2024], Tahmasebi et al. [2024], Kwon et al. [2021], Mi et al. [2022], Mordido et. al. [2024] and others.\n- The empirical improvements by IAM-D and IAM-S compared to baselines such as SAM or SGD are very minor. Given that IAM-D and IAM-S require $K$ additional gradient evaluations per iteration for estimating local inconsistency, while SAM and SGD require two and one respectively, it seems that if we allocate the same budget (in terms of gradient evaluations) and train baselines such as SGD and SAM longer, the improvement would be negligible. I recommend that the authors account for gradient evaluations in their experiments and report results using the same number of evaluations for the different methods.\n- Tables demonstrating the additional runtime overhead of IAM-D and IAM-S are also quite helpful.\n\nWhile I like the simple yet effective idea of this paper, given that the nature of the work is primarily experimental, I would, for now, lean toward rejection and reconsider later if my suggestions are addressed.\n\nMinor Suggestion:\n- It seems in equation for it should be $L_{S}(\\theta)$.\n- Typo: S_p(\\theta), caption of subfig 1 of figure 1.\n- Typo: Line 248, $L_{ISAM-D}(\\theta)$.\n\nReferences: \\\nLi et. al. Tilted Sharpness-Aware Minimization, ICML 2025, ArXIv 2024 \\\nTahmasebi et. al. A Universal Class of Sharpness-Aware Minimization Algorithms, ICML 2024 \\\nKwon et al. Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks, ICML 2021 \\\nMi et al, Make sharpness-aware minimization stronger: A sparsified perturbation approach, NeurIPS 2022 \\\nMordido  et. al. Lookbehind-SAM: k steps back, 1 step forward, ICML 2024\n\nQUESTIONS:\n- What is $m$ in equation 5?\n\nLIMITATIONS:\nNot discussed—especially, a discussion on the additional runtime and extra gradient valuations required for estimating local inconsistency would be very helpful.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-06"}