Title: FraPPE: Fast and Efficient Preference-Based Pure Exploration
Venue: NeurIPS 2025
Accepted: yes
Keywords: Bandits

Pareto Set Identification

Pure Exploration

Fixed Confidence

Sample Complexity

Preference Cone
Abstract: Preference-based Pure Exploration (PrePEx) aims to identify with a given confidence level the set of Pareto optimal arms in a vector-valued (aka multi-objective) bandit, where the reward vectors are ordered via a (given) preference cone $\mathcal C$. Though PrePEx and its variants are well-studied, there does not exist a *computationally efficient* algorithm that can *optimally* track the existing lower bound (Shukla and Basu, 2024) for arbitrary preference cones. We successfully fill this gap by efficiently solving the minimisation and maximisation problems in the lower bound. First, we derive three structural properties of the lower bound that yield a computationally tractable reduction of the minimisation problem. Then, we deploy a Frank-Wolfe optimiser to accelerate the maximisation problem in the lower bound. Together, these techniques solve the maxmin optimisation problem in $\mathcal O(KL^{2})$ time for a bandit instance with $K$ arms and $L$ dimensional reward, which is a significant acceleration over the literature. We further prove that our proposed PrePEx algorithm, **FraPPE**, asymptotically achieves the optimal sample complexity. Finally, we perform numerical experiments across synthetic and real datasets demonstrating that **FraPPE** achieves the lowest sample complexities to identify the exact Pareto set among the existing algorithms.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the Preference-based Pure Exploration (PrePEx) problem, focusing on identifying the set of Pareto optimal arms in a vector-valued bandit. In this problem, the reward vectors are ordered using a given preference cone $\\mathcal C$. The authors propose an algorithm based on a computationally tractable reduction of the minimization problem and the Frank-Wolfe optimizer. They prove both the algorithm's sample and computational complexity to demonstrate its effectiveness. Additionally, the authors validate the algorithm through experiments. Results demonstrate that their algorithm has advantages in both accuracy and efficiency.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n1. The authors design a computationally efficient and statistically optimal PrePEx algorithm that addresses the preference-based pure exploration problem with exponential family distributions.\n2. The authors provide a thorough analysis of their algorithm. Their algorithm's sample complexity and computational complexity are proven.\n3. The authors utilize real-world datasets to validate the algorithm's effectiveness experimentally.\n\n**Weakness**:\n\n1. The paper's definitions should adhere to standard, classical formulations, particularly for fundamental concepts such as partial order and the L-parameter exponential family. Regarding the latter, my assessment suggests that the authors' definition of the L-parameter exponential family deviates from its widely accepted form. This discrepancy could impact the validity of the paper's conclusions.\n\n2. Lemma 2 stands out as the most crucial conclusion in the paper. It purports to demonstrate that the algorithm achieves asymptotic optimality as $\\epsilon$ and $\\tilde{\\epsilon}$ approach zero. However, while $\\epsilon$'s definition is briefly touched upon in the appendix, the definition of  $\\tilde{\\epsilon}$ is absent. This omission prevents me from evaluating whether these two values can indeed be arbitrarily close to zero, which makes it impossible to assess the meaningfulness of Lemma 2's conclusion fully.\n\n3. The authors directly reference results from related papers for the lower bound of the problem.\n\nQUESTIONS:\n1. My assessment suggests that the authors' definition of the L-parameter exponential family deviates from its widely accepted form. This might mean the problem the authors are solving is actually limited to one-parameter exponential families, rather than truly extending to multi-parameter ones as the framing suggests. \n\n2. Could the authors please clarify the precise definition of  $\\tilde{\\epsilon}$  in Lemma 2? Furthermore, it's not immediately clear whether both $\\epsilon$ and $\\tilde{\\epsilon}$ can be arbitrarily close to zero. Gaining clarity on these points would be crucial for me to fully assess if the algorithm achieves asymptotic optimality as claimed.\n\n\nIf the authors can address my concerns regarding Lemma 2, I would consider improving my evaluation of the paper.\n﻿\nSeparately, because I'm not deeply familiar with the Pareto Set Identification problem, I haven't been able to thoroughly check all the proofs, despite spending considerable time on them. Therefore, I'll be looking to other reviewers' insights to help inform any adjustments to my overall assessment.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the design of computationally efficient algorithms for the preference-based pure exploration problem. The authors reduce the corresponding lower-bound optimization problem into a computationally tractable form and solve it using the Frank-Wolfe algorithm. The authors further provide theoretical guarantees for proposed algorithms on computational and sample complexity. Both theoretical results and empirical experiments demonstrate the superiority of their proposed algorithm.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1. This work explores a computationally efficient algorithm for solving the preference-based pure exploration problem. It presents a rigorous theoretical analysis on both the computational and sample complexity of the proposed method. \n\nWeaknesses: \n1. In the problem formulation part of Section 2, the definition of the reward distribution $\\nu_a$ seems somewhat unconventional, as the authors only define its mean $\\boldsymbol{\\mu}_a$. In contrast, (Shukla and Basu, 2024) further define the corresponding covariance matrix $\\Sigma$ to account for noisy feedback. \n\n2. In Figure 1, the authors intend to illustrate how the Pareto frontier varies under different preference cones. However, in Figure 1, the two Pareto frontiers overlap, which easily confuses readers. The authors could consider plotting the Pareto frontiers for both $\\pi/2$ and $\\pi/3$ cones to better illustrate the difference. \n\n3. Some concepts are used without formal definitions, such as the term \"stationary policies\" in line 209. \n\n4. In line 298, the authors claim that the Frank-Wolfe algorithm can handle non-smooth optimization objectives by constructing the corresponding $r$-sub-differential set. Can this approximation method guarantee recovery of the original optimal solution? In addition, please justify the choice of the hyper-parameter $r_t$. \n\n5. A minor writing suggestion: in line 205, the concept of $p$ pure policies has not been formally introduced. To avoid potential confusion, it is advisable to use the notation \\Delta_K \\setminus \\{\\pi^*\\} rather than \\Delta_K \\setminus \\{\\pi_i^*\\}. \n\n6. It is recommended that the authors provide a comprehensive table comparing different preference-based pure exploration methods in terms of algorithmic complexity (including time and space complexity) as well as the upper bounds on sample complexity. This would help readers better understand the strengths and weaknesses of the proposed method in a broader context.\n\n7. The authors could elaborate more on the technical challenges in accelerating algorithms solving the preference-based pure exploration problem, and clarify the theoretical contributions of their proposed approach.\n\nQUESTIONS:\nPlease see the strengths and weaknesses section.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles Preference-based Pure Exploration (PrePEx) in multi-objective bandits, aiming to identify the Pareto-optimal arms efficiently and confidently. The authors propose FraPPE, a computationally efficient and asymptotically optimal algorithm that leverages structural reductions, a Frank-Wolfe optimizer, and a novel stopping rule. Empirical results demonstrate gains in sample efficiency and stability over prior methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n•\tThe paper tackles an interesting and well-motivated problem; namely preference-based pure-exploration.\n\n•\tThe works proposes a practical approach which reduce the computational complexity of solving the tackled problem.\n\n•\tThe paper results are supported by theoretical guarantees and empirical analysis on both synthetic and real-world problems.\n\n\n**Weaknesses:**\n\n•\tAdditional tests on larger-scale or more diverse real-world settings would strengthen the empirical claims.\n\n•\tFor clearer presentation and better contrast with prior work, it would be helpful to summarize the main differences and contributions in a comparison table. \n\n•\t(minor) While multi-armed bandits provide a foundational framework for studying sequential decision-making under uncertainty, describing them as the \"archetypal setup\" of reinforcement learning is somewhat misleading (line 36). Bandits represent a simplified special case without state transitions or delayed consequences, which are central features of reinforcement learning.\n\nQUESTIONS:\nCan the same approach (FraPPE) be extended or scaled to combinatorial pure exploration problems, such as combinatorial bandits?\n\nHow sensitive is FraPPE to mild deviations from its theoretical assumptions in practical scenarios?\n\nCould the authors report or discuss FraPPE's actual runtime in wall-clock seconds (rather than just iteration or sample complexity), especially on the real-world dataset? This would provide practical insight into computational efficiency beyond theoretical complexity.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers pure exploration in a K-armed stochastic bandit problem with L-dimensional rewards. The goal is to identify the Pareto set of policies (distributions over arms) given a fixed confidence level in as few evaluations as possible. The paper builds an approach based on the lower bound on Pareto front identification. Using structural properties, it forms a computationally tractable alternative to the minimax optimization problem in the lower bound. A track-and-stop inspired arm selection and stopping rule is proposed based on the tractable lower bound. Asymptotic optimality of the proposed rule is established. Experiments are carried out to show how the proposed rule compares with SoTA.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\nThe paper is clearly written. The authors did a good job of explaining this complicated pure exploration problem in nine pages. \n\nThis paper solves an important open problem in bandit-based vector optimization by proposing a computationally efficient, asymptotically optimal, fixed-confidence pure exploration algorithm. The proposed algorithm utilizes a tractable reformulation of the intractable optimization problem. Theoretical analysis of the computational complexity demonstrates a significant improvement over existing approaches. The paper proposes a non-asymptotic, asymptotically optimal bound on the sample complexity. Experiments support theoretical claims. \n\n\nWeaknesses: \n\nThere is no major weakness.\n\nQUESTIONS:\n1) It is hard to parse the contribution of Section 3.3. What is the exact reduction in computational complexity when the polar cone representation of alternating instances is used? \n\n2) Please elaborate on how the $ L$-parameter exponential family extends the scalar exponential family. What conditions are required on functions $h$, $\\eta$, and $T$?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses an open problem in preference-based pure exploration (PrePEx) in multi-objective bandits, where the goal is to identify all Pareto optimal arms under a given preference cone with fixed confidence. The authors propose FraPPE, a computationally efficient algorithm that provides an asymptotically optimal solution for arbitrary preference cones and exponential family distributions. The key contributions include: \n- Structural reductions of the optimization problem, avoiding convex hull computations, \n- Application of Frank-Wolfe optimization for efficient lower bound tracking, and \n- Theoretical guarantees of asymptotic optimality with improved computational complexity.\n- Simulation results to support the proposed results.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n**Novel Theoretical and Algorithmic Contributions:** The paper makes significant progress on an open problem posed by Crepon et al. (2024) by providing a computationally efficient and asymptotically optimal algorithm for arbitrary preference cones. The decomposition of the Alt-set without requiring convex hull computations is a significant theoretical advance. The reduction from O(K²L) to O(KL min{K,L}) computational complexity is non-trivial and practically important.\n\n**Problem Relevance and Motivation:** The multi-objective bandit setting with preference cones is well-motivated through concrete applications in clinical trials (COV-BOOST), material design, and RLHF. The inability to scalarize objectives a priori is a genuine challenge in these domains, making the preference cone framework particularly relevant.\n\n**Strong Literature Positioning:** The paper does a great job of placing itself amid related work. The authors clearly articulate the limitations of existing approaches: successive elimination methods that only find approximate Pareto sets, the computational intractability of PreTS, and the restriction of previous optimal algorithms to other distributions. The connection to pure exploration with multiple answers and the Frank-Wolfe literature is also well stated.\n\n**Clear Presentation:** The paper is well-structured with a logical flow from problem formulation through theoretical contributions to algorithm design. The use of illustrative examples aids a deeper understanding of the work. Technical concepts are introduced progressively with appropriate mathematical insights.\n\n**Weaknesses:**\n\n**Limited Experimental Validation:** While the COV-BOOST results are compelling, the experimental section would benefit from more ablation studies: (i) runtime breakdown to validate computational efficiency claims, (ii) scaling effects with varying K and L, and (iv) ablation studies on other parameters.\n\n**Assumptions and Practical Considerations:** The assumption of perfect knowledge of preference cones may be restrictive in practice. Maybe a discussion of robustness to misspecified cones would be a nice add.\n\nQUESTIONS:\n**Questions**\n\n**Preference Cone Specification in Practice:** What are the real-world models with the preference cone C, and what are their typical values? Also, how sensitive is the algorithm's performance to misspecification of the preference cone C? In real applications, domain experts may have uncertainty about their preferences. Have you considered extensions where the cone is learned or refined during exploration?\n\n**Computational Bottlenecks:** Despite the theoretical improvements, conic programs per iteration could still be computationally demanding. Does an experimental time breakdown on which steps of the algorithm provide the time bottleneck make sense?\n\nLIMITATIONS:\nIt would be great if the authors added a dedicated `limitations` section discussing the points touched upon in the Weaknesses and Questions section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-11"}