Title: Uncertainty-Informed Meta Pseudo Labeling for Surrogate Modeling with Limited Labeled Data
Venue: NeurIPS 2025
Accepted: yes
Keywords: Surrogate modeling

Semi-supervised regression

Meta pseudo label
Abstract: Deep neural networks, particularly neural operators, provide an efficient alternative to costly simulations in surrogate modeling. However, their performance is often constrained by the need for large-scale labeled datasets, which are costly and challenging to acquire in many scientific domains.
Semi-supervised learning reduces label reliance by leveraging unlabeled data yet remains vulnerable to noisy pseudo-labels that mislead training and undermine robustness.
To address these challenges, we propose a novel framework, Uncertainty-Informed Meta Pseudo Labeling (UMPL).
The core mechenism is to refine pseudo-label quality through uncertainty-informed feedback signals. Specifically, the teacher model generates pseudo labels via epistemic uncertainty, while the student model learns from these labels and provides feedback based on aleatoric uncertainty.
This interplay forms a meta-learning loop where enhanced generalization and improved pseudo-label quality reinforce each other, enabling the student model to achieve more stable uncertainty estimation and leading to more robust training.
Notably, This framework is model-agnostic and can be seamlessly integrated into various neural architectures, facilitating effective exploitation of unlabeled data to enhance generalization in distribution shifts and out-of-distribution scenarios.
Extensive evaluations of four models across seven tasks covering steady state and transient prediction problems demonstrate that UMPL consistently outperforms the best existing semi-supervised regression methods. When using only 10% of the fully supervised training data, UMPL achieves a 14.18% improvement, highlighting its strong effectiveness under limited supervision. Our codes are available at https://github.com/small-dumpling/UMPL.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Uncertainty-Informed Meta Pseudo Labeling, a semi-supervised learning framework tailored for surrogate modeling of physical systems governed by PDEs under limited labeled data. The framework builds upon a teacher-student architecture, where the teacher estimates epistemic uncertainty via EMC Dropout to generate pseudo-labels, and the student learns from these pseudo-labels while modeling aleatoric uncertainty via heteroscedastic regression. A key contribution is the use of uncertainty-informed feedback: the student’s uncertainty guides refinement of the teacher’s pseudo-labels, enabling a closed-loop meta-learning process. The method improves generalization under distribution shifts across multiple tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Integration of dual uncertainty modeling in a meta-learning loop.\n2. Efficient and theoretically grounded uncertainty estimation.\n3. Strong empirical results on PDEs and real-world datasets.\n\n\nWeaknesses:\n1. The key issue here is the contribution. Although this work proposed multiple ways to improve uncertainty estimation in terms of reliability and efficiency. However, the core part of this work is the SSL, which just use the uncertainty as a tool. I think the best way to justify the contribution should focus on SSL instead of the accuract uncertainty. \n2. It is good to achieve a better uncertainty estimation, but why don't just use most up to date uncertainty estimation methods or  calibration methods that achieve better performance on universal accepted benchmarks?\n3. The proposed uncertainty estimation method is not well evaluated on widely-accepted benchmarks, if the author believe the proposed method is better than other uncertainty estimation methods, you can include more experiments to justify it.\n4. The computational overhead analysis is lacking.\n5. How sensitive is the method to the quality of aleatoric uncertainty estimates from the student? The robustness of the method to different PUF thresholds or uncertainty injection strengths need to be better discussed.\n\nQUESTIONS:\nlisted in the weakness\n\nLIMITATIONS:\nYes. The paper includes a discussion on limitations in the appendix and acknowledges challenges such as computational cost and the assumption of reliable uncertainty estimation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a semi-supervised teacher-student framework for emulators with limited simulation data. The teacher generates pseudo-labels using MC-Dropout to predict epistemic uncertainty, while the student learns from these labels while predicting aleatoric uncertainty via variance prediction. The student's uncertainty provides feedback to refine the teacher's pseudo-labels in a meta-learning loop and adds spatial coherence. Evaluation of the system is presented from scientific tasks and distribution shifts.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n* Original feedback closed loop between teacher-epistemic and student-aleatoric \n* The mesh-aware spatial coherence for the pseudo-labels\n* Extensive experimental section on various architectures\n\n**Weaknesses**\n* Questionable uncertainty method: the paper relies on MC-Dropout to estimate the teacher-epistemic, known to struggle when the true uncertainty isn't nicely behaved and bell-shaped (Gaussian) – common in messy real-world physics like shocks or abrupt changes. \n* Reliable uncertainties not shown: there should be some experimental proof that the uncertainty prediction are trustworthy using checks like calibration metrics or reliability diagrams. Aren' t the errors remain high in complex tasks like NSM2D?\n* Potentially overly easy experiments: benchmarks used clean, smooth simulation data without the real-world noise or sharp changes one could encounter. This likely explains why even the basic \"Pseudo Label\" baseline beat supervised learning – a result that often doesn't hold up with noisier or more complex data.  I am not convinced that the method would hold against the hard distribution shifts or more complex noise common in real data\n\nQUESTIONS:\n1. Both epistemic (MC-Dropout) and aleatoric (heteroscedastic variance) uncertainty predictors assume symmetric Gaussian distributions. How does UMPL handle physical systems with asymmetric or heavy-tailed uncertainties (e.g., shock waves, bifurcation points) where these assumptions fail? Did you evaluate on such systems?\n2. Why are standard uncertainty calibration metrics absent? Can you show quantitative proof that the predicted uncertainty reliably correlates with error magnitude (not just direction) in OOD regions?\"\n3. Could you report computing resources?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the challenge of building robust surrogate models for complex physical systems, especially under conditions of limited labeled data, distribution shifts or OOD scenarios. Traditional DNNs and neural operators require large labeled datasets, which are often costly or impractical to obtain in scientific domains but the problem arises as, even though semi-supervised learning offers a way to leverage abundant unlabeled data, it suffers from noisy pseudo-labels that can degrade model performance.\n\nTo tackle this the authors propose Uncertainty-Informed Meta Pseudo Labeling (UMPL), a semi-supervised learning framework that improves pseudo-label quality by incorporating uncertainty estimates in a setup of a teacher-student meta-learning framework where they have roles based on uncertainty. Specifically, the teacher model generates pseudo-labels along with epistemic uncertainty, which reflects the model’s confidence and helps identify uncertain predictions under distribution shifts. The student model however learns from these pseudo-labels in order to estimate the aleatoric uncertainty, capturing inherent data noise and label uncertainty. This separation of uncertainty types aligns with the distinct roles of teacher and student and leads to more robust and interpretable uncertainty estimates. The student provides feedback to the teacher based on aleatoric uncertainty, guiding the teacher to refine pseudo-labels iteratively. This meta-learning loop enhances pseudo-label quality and model generalization, especially in uncertain or underrepresented regions of the data.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is well written and the motivation is clear. Although I am not that familiar with the practical problems and frameworks that solve (or at least tackle how to solve) problems like pseudo-labeling of high-dimensional PDE systems, it seems that the problem is well formulated and the authors motivate it well.\n\nRegarding the contribution of the paper, in their second bullet point the authors claim that the role-aware design leads to more robust and interpretable uncertainty estimates, however without backing it up with any claims. It would be good if the authors could point out or summarize the sections, results (figures/tables) which back this claim up. At the moment it seems a bit unjustified. The same comment stands for the third bullet point, in which the authors claim that the targeted feedback significantly improves the quality of pseudo labels across iterations without claiming why that is the case.\n\nAs I am unfamiliar with the setting of modeling high-dimensional PDE systems I struggle to find the motivation for the choice of relative L_2 error compared to just the standard L2 loss. Can the authors explain the motivation why the loss choice in (1)? Also, why have the authors chosen an encoder embedding? Is this a popular model choice in the student-teacher literature? Is there a reason why one would expect this would work better for modeling PDE systems? Both fall a bit flat unmotivated.\n\nThe same comment stands for their choice of MC Dropout. Although the simplest to implement and most straightforward to train from my experience with BNNs, the authors should argue more why they have chosen it, and not for example SVGD[1], fSVGD[2] or any other Bayesian NN choice, and I think that they should perform benchmarks/baselines to corroborate their choice. In Appendix C the authors give a nice Bayesian interpretation of MC Dropout which I am thankful for, but I was a bit disappointed that they did not deliberate anything about why specifically choosing it over other methods. \n\nRegarding Figure 2. it is not really clear what the point of the figure is, the error between the proposed method and Noisy TS is differently distributed and with different magnitudes and its not clear to me from the plot which one is better.\n\nRegarding using REINFORCE as the solution to minimize the loss in (8) and deriving its gradient, it would be nice if the authors add a few comments regarding the REINFORCE rule; are there any downsides to using it, is it the standard in other literature, what are its benefits and so on. I am not well versed in this literature and to me, although all the design choices that the authors have made towards proposing the UMPL framework, they fall a bit flat. In general, is there any reason why we would expect each of them to work better separately and together, how do they interplay and can the authors show that other choices for the building blocks of UMPL would've resulted in inferior performance? The ablation study in Table 3 was indeed nice and provided one small step towards this but giving at least one or two examples of what happens with other choices of uncertainty estimation (e.g., Bayesian NNs) would've been nice to include. \n\nOne argument towards this could be the superior results that the authors observe in Tables 1 and 2, but these also fall a bit short as I don't know if these results are statistically significant or not. In answers to question 6 of the Q&A the authors mention that they introduce experiment setting and details but I couldn't find how long did it take to run your experiments and whether the authors ran them on GPUs vs CPUs. Furthermore, it is mentioned that the statistical significance is not reported as in fluid dynamics modeling using a fixed random seed shows consistent performance. I believe this claim needs to be backed up and at least commented in the main paper or the appendix. If the results were computationally expensive and you have limited resources this is fine but then this should be commented on too. If you did mention this somewhere in the appendix I would be happy if you can point me to it, but if your experiments did not require extensive computational resources I think that statistical significance needs to be added both to Tables 1 and 2 as well as the L2 error in Table 3.\n\n[1] Liu, Q. and Wang, D., 2016. Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances in neural information processing systems, 29.\n[2] Wang, Z., Ren, T., Zhu, J. and Zhang, B., 2019. Function space particle optimization for Bayesian neural networks. arXiv preprint arXiv:1902.09754.\n\nQUESTIONS:\nAs mentioned above:\n\nAs I am unfamiliar with the setting of modeling high-dimensional PDE systems I struggle to find the motivation for the choice of relative L_2 error compared to just the standard L2 loss. Can the authors explain this motivation?\n\nHow long did it take to run the experiments and did you use GPUs or CPUs? Could you please elaborate more on why do you not report statistical significance? Can you please include significance for at least a couple of experiments in Table 1 or 2? \n\nTypo in Appendix H title: \"experiential settings\"\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the challenge of building surrogate models for physics-governed systems when only a small, costly set of simulation labels is available alongside abundant unlabeled data, proposing a semi-supervised teacher–student framework in which a lightweight Exponential-Moving Monte-Carlo Dropout (EMC-Dropout) teacher predicts system states and provides epistemic uncertainty, while a student network learns from those pseudo-labels, estimates aleatoric uncertainty, and feeds it back to the teacher. Training is further stabilized by Progressive Uncertainty Filtering, which gradually discards the noisiest pseudo-labeled regions, and by Supervised Anchoring, which periodically re-grounds the model on true labels. Evaluated by plugging into four distinct neural operators—DeepONet, MeshGraphNet, Transolver, and AeroGTO—across seven CFD and physics benchmarks spanning partial differential equations, real-world datasets, and large-scale 3-D simulations, the method achieves roughly a 14 % error reduction with only 10 % labeled data and markedly improves out-of-distribution and distribution-shift generalization.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. **Novel teacher–student strategy** – The framework introduces an innovative teacher–student loop that produces more reliable pseudo-labels while simultaneously training the surrogate model.\n\n2. **Architecture-agnostic design** – Because the method is model-agnostic, it can be plugged into virtually any neural-operator or surrogate-model architecture with minimal modification.\n\n3. **Thorough empirical validation** – The authors support their claims with extensive experiments across multiple datasets and architectures, demonstrating consistently strong performance.\n\n\n\n**Weaknesses**\n\n1. **Potential computational burden** – Running two networks, multiple Monte-Carlo dropout passes, and iterative feedback may be resource-intensive. A dedicated section detailing the hardware used, run-times, and memory footprint would clarify the true computational cost.\n\n2. **Clarity and presentation** – Several aspects of the manuscript hinder readability:\n   \n   a. Key variables and symbols should be defined in the main text rather than deferred to the appendix.\n   \n   b. Algorithm 1 should be included in the main body to better understand the whole procedure.\n   \n   c. Although Figure 1 is helpful in principle, its current form does not clearly depict the full workflow and should be redesigned for greater clarity.\n\nQUESTIONS:\n1. Could you elaborate on the computational resources your method requires—e.g., hardware, run-time, and memory overhead—so readers can better gauge its practical cost?\n\n2. Within the teacher–student framework, is there a risk of model collapse, where the teacher and student converge to similar outputs on unseen data yet still generate poor labels relative to the true target distribution? - How to protect against this?\n\nLIMITATIONS:\nYes — the limitations are discussed in Appendix Section K.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-09"}