Title: Preference-Guided Diffusion for Multi-Objective Offline Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Offline Optimization

Multi-Objective Optimization

Diffusion Models

Classifier Guidance
Abstract: Offline multi-objective optimization aims to identify Pareto-optimal solutions given a dataset of designs and their objective values. In this work, we propose a preference-guided diffusion model that generates Pareto-optimal designs by leveraging a classifier-based guidance mechanism. Our guidance classifier is a preference model trained to predict the probability that one design dominates another, directing the diffusion model toward optimal regions of the design space. Crucially, this preference model generalizes beyond the training distribution, enabling the discovery of Pareto-optimal solutions outside the observed dataset. We introduce a novel diversity-aware preference guidance, augmenting Pareto dominance preference with diversity criteria. This ensures that generated solutions are optimal and well-distributed across the objective space, a capability absent in prior generative methods for offline multi-objective optimization. We evaluate our approach on various continuous offline multi-objective optimization tasks and find that it consistently outperforms other inverse/generative approaches while remaining competitive with forward/ surrogate-based optimization methods. Our results highlight the effectiveness of classifier-guided diffusion models in generating diverse and high-quality solutions that approximate the Pareto front well.


Peer reviews:
comments: SUMMARY:
This paper proposes a novel generative approach for offline multi-objective optimization using a classifier-guided diffusion model. The core idea is to guide the generation process using a preference model trained to predict whether one design dominates another. This preference model is not the same as the surrogate model generall used in offline MOO.  This guidance steers the diffusion model toward Pareto-optimal regions, including those beyond the training data.

STRENGTHS AND WEAKNESSES:
Strengths: the paper handles an important problem and uses advanced techs to achieve impressive results.  I like the idea of using preference model to guide diffusion generation. 

Weakness:
1. The paper misses an important citation: Kim M, Gu J, Yuan Y, et al. Offline Model-Based Optimization: Comprehensive Review[J]. arXiv preprint arXiv:2503.17286, 2025.

2. Figure 1 seems never mentioned in the text. 

3. It looks like diversity contribution is dataset-dependent while the preference model is dataset-independent. How can u train a preference model using the diversity contribution or "equally dominant" is very rare?

4. I think the author may misunderstand some point: the preference model is trained on pairs from the offline dataset (time step 0) while used to guide designs across different time steps.

QUESTIONS:
See Strengths And Weaknesses*

LIMITATIONS:
See Strengths And Weaknesses*
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the challenge of offline black-box multi-objective optimization (MOO), where new evaluations are infeasible, and the goal is to infer Pareto-optimal solutions beyond the observed dataset. To this end, the authors propose PGD-MOO, a novel inverse approach that combines diffusion models with preference-guided classifiers. Unlike forward methods that rely on training surrogate models for each objective, PGD-MOO leverages the generative capabilities of diffusion models to learn the mapping from objective values to input designs. Furthermore, a preference classifier model is trained to guide the denoising process toward the solution space. In addition to solution dominance, the classifier innovatively incorporates a diversity criterion to obtain more varied and sparsely distributed solutions. Experimental results demonstrate that PGD-MOO outperforms other inverse methods while maintaining strong generalization capabilities. It achieves solution quality comparable to forward methods, which may not be feasible in many real-world scenarios. Moreover, it exhibits significantly better diversity, enabling a more comprehensive approximation of the Pareto front.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. The paper pioneers a diffusion-based offline MOO method that integrates diversity optimization. It deftly replaces scalarized surrogates with a unified dominance/diversity classifier, thereby streamlining the overall architecture.\n2. The method incorporates crowding distance (a concept borrowed from NSGA-II) into preference labeling, ensuring uniform and comprehensive coverage of the Pareto front.\n3. Through evaluations on a diverse array of continuous offline multi-objective optimization tasks—spanning synthetic scenarios to real-world engineering problems—the proposed method consistently demonstrates superior performance. It excels in terms of both convergence to the Pareto front and the diversity of solutions.\n\n### Weaknesses\n1. The paper provides only a brief description of the diffusion model component. As the foundational element of the proposed method, the diffusion model's training process is not sufficiently detailed, particularly regarding how it is adapted to multi-objective optimization settings. The paper should elaborate on how multi-objective datasets are incorporated into the standard diffusion modeling pipeline and how the multi-objective nature of the problem is addressed during training.\n2. In the experimental section, only one inverse approach is included for comparison, which may not be sufficient to fully demonstrate the advantages of the proposed method. It is recommended to include a diffusion-based baseline without classifier guidance as part of an ablation study to better highlight the contribution of the preference-guided mechanism.\n3. The introduction section transitions rather abruptly into listing the contributions. Additionally, Section 3 offers relatively limited discussion of the proposed method, which appears disproportionate given the repeated exposition of the offline multi-objective optimization background earlier in the text. It is recommended to streamline the problem formulation to reduce redundancy and instead devote more space to elaborating on PGD-MOO, particularly the application and adaptation of the diffusion model.\n\nQUESTIONS:\n1. In practical optimization problems, there are often various constraints. How can the proposed method be extended to handle constrained multi-objective optimization problems and ensure that the generated solutions satisfy constraints?\n2. The training process of diffusion models is often computationally intensive. Are there ways to optimize the training process to reduce computational costs and improve training efficiency, such as through model architecture design or training algorithm improvements?\n3. As the number of objectives grows, Pareto dominance information tends to have a diminished impact. Why not consider incorporating hypervolume (HV) contribution or other metrics instead of relying solely on crowding distance when comparing two designs?\n4. The experiments conducted thus far have focused on small-dimension datasets. How can the proposed method be effectively extended to tackle problems characterized by high-dimensionality and complex constraints?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Preference-Guided Diffusion for Multi-Objective Offline Optimization (PGD-MOO), which uses a preference-trained classifier to guide diffusion sampling in offline multi-objective optimization. A preference classifier is trained on pairwise comparisons in which the preferred design Pareto dominates its counterpart; when two designs are mutually equally dominating, the tie is resolved by favoring the one with a larger crowding distance, so the model learns to value diversity along the Pareto front. The authors demonstrate that PGD-MOO outperforms other baselines in DTLZ and ZDT tasks.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\nThe paper is clearly structured and easy to follow. One of its most compelling aspects is the idea of overweighting more diverse samples over concentrated ones, thereby encouraging the classifier to favor diversity. To the best of my knowledge, this is a novel approach, and it appears to improve the ∆-spread metric effectively. In the synthetic function benchmarks, PGD-MOO shows that a preference-based classifier outperforms the surrogate model approach, as well as multi-head and multi-model baselines.\n\n### Weakness\n- **Narrow experimental coverage.** PGD-MOO is tested on two synthetic tasks (DTLZ, ZDT) and the RE engineering tasks, whereas Xue et al. [1] include a broader set that includes MO-Swimmer, MO-Hopper, MO-Portfolio, and Molecule tasks, characterized by continuously represented data. It would be better to add more continuous representation tasks to the experiment section for more solid demonstrations. \n\n- **PGD-MOO underperforms ParetoFlow** On the RE suite, PGD-MOO underperforms ParetoFlow [2] despite using the same guidance mechanism—the main difference being the classifier. This result suggests that a preference-based classifier may not outperform a surrogate regressor in practice.\n\n- **Limited methodological novelty.** Classifier-guided diffusion sampling was first introduced by Dhariwal et al. [3] and later extended to classifier-free guidance by Ho et al. [4]; these two variants are fundamentally the same in that both condition the reverse process at inference time. For offline optimization, Krishnamoorthy et al. [5] already explored classifier-free guidance for offline model-based optimization. Even if one considers conditioned generation to be a universal technique, the use of preference classifiers is already a well-established idea in RLHF for language models [6]. As such, the novelty of this work remains incremental.\n\n- **Restricted to continuous design spaces.** Because PGD-MOO inherits the continuous DDPM backbone [7], it cannot natively handle discrete variables, whereas ParetoFlow [2] works for both discrete and continuous problems. Adopting a discrete-friendly diffusion variant (or a data-type–agnostic flow) would make the approach more generally useful.\n\n- **Clarity.** The explanation of the training method for the preference model in Section 4.1 could benefit from greater clarity. It seems to be based on the Bradley-Terry model, but this is not made explicit in the text. Providing a more detailed and transparent description would help readers better understand the underlying methodology.\n\n- **Limited ablation about diversity contribution.** The paper provides limited ablation on the *diversity contribution* to the selection of winning samples. To better understand the effectiveness of isolating diversity as a design choice, it would be valuable for the authors to include an ablation study specifically examining its impact on performance.\n\n[1] Xue, Ke, et al. \"Offline multi-objective optimization.\" arXiv preprint arXiv:2406.03722 (2024).\n\n[2] Yuan, Ye, et al. \"ParetoFlow: Guided Flows in Multi-Objective Optimization.\" arXiv preprint arXiv:2412.03718 (2024).\n\n[3] Dhariwal, Prafulla, and Alexander Nichol. \"Diffusion models beat gans on image synthesis.\" Advances in neural information processing systems 34 (2021): 8780-8794. \n\n[4] Ho, Jonathan, and Tim Salimans. \"Classifier-free diffusion guidance.\" arXiv preprint arXiv:2207.12598 (2022).\n\n[5] Krishnamoorthy, Siddarth, Satvik Mehul Mashkaria, and Aditya Grover. \"Diffusion models for black-box optimization.\" International Conference on Machine Learning. PMLR, 2023.\n\n[6] Ouyang, Long, et al. \"Training language models to follow instructions with human feedback.\" Advances in neural information processing systems 35 (2022): 27730-27744.\n\n[7] Ho, Jonathan, Ajay Jain, and Pieter Abbeel. \"Denoising diffusion probabilistic models.\" Advances in neural information processing systems 33 (2020): 6840-6851.\n\nQUESTIONS:\n1. Could you elaborate on why conditional guidance was chosen as the core mechanism for multi-objective offline optimization? In line 205, the paper mentions the goal of generating samples near the Pareto front, including those outside the training data distribution. However, Yuan et al [8] has suggested that conditional guidance can be difficult to tune effectively, and its generalization to out-of-distribution (OOD) regions tends to be limited. In contrast, approaches that fine-tune the diffusion model using proxy models such as DRaFT [9] and DDPO [10] have been shown to generate samples from desirable regions with strong performance. Finetuning methods are more robust in OOD area because finetuning moves the model distribution towards high-rewarding space. Moreover, recent approaches like GFlowNets [11] have been used to mitigate mode collapse in reinforcement learning, offering alternative sampling strategies with improved diversity. While [8][9][10][11] is not for MOO, in light of these alternatives, I am curious about the rationale behind choosing conditional guidance over diffusion finetuning methods.\n\n2. A follow-up question related to the clarity of the methodology: could you confirm whether the preference model was trained using the Bradley-Terry model? This is suggested implicitly, but an explicit clarification would be very helpful.\n\n[8] Yuan, Hui, et al. \"Reward-directed conditional diffusion: Provable distribution estimation and reward improvement.\" Advances in Neural Information Processing Systems 36 (2023): 60599-60635.\n\n[9] Clark, Kevin, et al. \"Directly fine-tuning diffusion models on differentiable rewards.\" arXiv preprint arXiv:2309.17400 (2023).\n\n[10] Black, Kevin, et al. \"Training diffusion models with reinforcement learning.\" arXiv preprint arXiv:2305.13301 (2023).\n\n[11] Liu, Zhen, et al. \"Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNets.\" arXiv preprint arXiv:2412.07775 (2024).\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a diffusion-based generative method for offline MOO, which aims to identify the Pareto-optimal solutions only given the offline dataset $\\mathcal{D}= \\\\{ ( \\mathbf{x}_i, \\mathbf{y}_i ) \\\\} _{i=1}^{N}$. The proposed method first trains a pairwise preference classifier based on NSGA-II selection [1] (integrating both dominance and solution diversity) to determine which design is better, then trains an unconditioned diffusion model. After that, it uses the classifier to guide the denoising procedure. Experimental results demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- This proposed method is clear and well motivated.\n- The code implementation is clean and well constructed.\n\n## Weaknesses\n\n- In line 253, you mentioned that “This approach does not require training surrogate models, thus providing a simple alternative approach to offline MOO”. As in [2], the end-to-end surrogate model is prediction-based MLP regressor. Don’t you need to train an MLP-based preference classification-based surrogate model too? Based on my comprehension, this discussion is not appropriate.\n- Based on my comprehension, your approach is somehow similar to RGD [3] in offline single-objective optimization. Relevant discussion on both this work and the difference between your framework and RGD is needed.\n\nQUESTIONS:\nHow do you set up the noise schedules $\\beta_t$ in your experiments?\n\nLIMITATIONS:\nLimited experimental studies:\n\n- Effectiveness of the classification-based preference guider is not examined. For example, I suggest compare it with weighted sum regression-based predictor [4] or ranking-based preference guider [5] (use ranking loss to train the model).\n- I cannot understand the benefits of the inverse approaches in offline MOO based on the experiments. Upon the same preference classifier, I suggest comparing your diffusion-based method to searching inside the classifier by replacing the selection operator of NSGA-II with the model’s output.\n- Ablation of the guidance weight $w$ is needed, as in [6].\n\nI understand that conducting additional experimental evaluations in all tasks of Off-MOO-Bench [2] is time-consuming during the rebuttal period. I suggest that you compare those in some representative tasks. I will be glad to increase the score once you address my concerns.\n\n## References\n\n[1] A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE TEvC, 2002.\n\n[2] Offline multi-objective optimization. ICML 2024.\n\n[3] Robust Guided Diffusion for Offline Black-Box Optimization. TMLR 2024.\n\n[4] ParetoFlow: Guided Flows in Multi-Objective Optimization. ICLR 2025.\n\n[5] Offline model-based optimization by learning to rank. ICLR 2025.\n\n[6] Diffusion models for black-box optimization. ICML 2023.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}