Title: Constrained Linear Thompson Sampling
Venue: NeurIPS 2025
Accepted: yes
Keywords: Safe Bandits

Linear Bandits

Thompson Sampling
Abstract: We study safe linear bandits (SLBs), where an agent selects actions from a convex set to maximize an unknown linear objective subject to unknown linear constraints in each round. Existing methods for SLBs provide strong regret guarantees, but require solving expensive optimization problems (e.g., second-order cones, NP hard programs). To address this, we propose Constrained Linear Thompson Sampling (COLTS), a sampling-based framework that selects actions by solving perturbed linear programs, which significantly reduces computational costs while matching the regret and risk of prior methods. We develop two main variants: 
S-COLTS, which ensures zero risk and ${\tilde{O}(\sqrt{d^3 T})}$ regret given a safe action, and R-COLTS, which achieves ${\tilde{O}(\sqrt{d^3 T})}$ regret and risk with no instance information. In simulations, these methods match or outperform state of the art SLB approaches while substantially improving scalability. On the technical front, we introduce a novel coupled noise design that ensures frequent 'local optimism' about the true optimum, and a scaling-based analysis to handle the per-round variability of constraints.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces algorithms for safe linear bandits that utilize Thompson sampling-like approaches. The algorithms are evaluated according to their regret (cumulative difference in total reward between optimal actions and played actions) and risk (cumulative constraint violations of played actions). They first introduce an algorithm that ensures $\\tilde{O}(d^{3/2} \\sqrt{T})$ regret and zero risk (i.e. constraint satisfaction for all rounds) under the assumption that there is a known point that is strictly feasible. They then introduce a method with $\\tilde{O}(d^{3/2} \\sqrt{T})$ regret and risk that doesn't require the aforementioned assumption. Notably, both of their methods apply a Thompson sampling to both the reward function and the constraint function. They also provide some numerical comparison of their work with some existing work.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Their algorithms only require an LP per a round, versus prior work that requires an SOCP in each round with the same regret. As such, one expects that there is often substantial computational benefits to their methods.\n- The algorithm design combines some existing ideas in a new way, and introduces some new ideas. For example, it combines a scaling approach similar to what is used in [HTA24] and a linear Thompson sampling-type approach [AG13]. The resampling approach (Algorithm 2) appears to be new to the literature.\n- The analysis goes substantially beyond existing work, as combining the aforementioned algorithm designs makes the analysis substantially more difficult. The analysis ideas introduced in the paper might be more broadly relevant.\n- Relevant work is thoroughly discussed.\n- The paper does a good job of presenting the key challenges that arise in the analysis and design of the algorithms.\n\nWeaknesses:\n- The paper is quite dense, which sometimes made it difficult to follow. I expect that this will be relieved somewhat with the extra page that is allowed for the final version.\n- I discuss some specific minor points in the questions section.\n\nQUESTIONS:\nSome minor points:\n- In line 122, it says that \"we also assume that $m/\\delta = O(poly(d))$ when stating theorems.\" Can you provide some clarification on this? To my knowledge, it is common to take $\\delta = 1/T$ to get expected regret guarantees, which would violate that assumption.\n- I think the presentation of Table 1 (first four rows) is misleading in that it sort of hides the assumption that the known safe action $a_{safe}$ is strictly-feasible (otherwise the given regret bounds are void).\n- Slater's condition is defined in the caption of Table 1, but I don't see that it is used in the table.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-19"}

comments: SUMMARY:
This paper considers the safe linear bandits problem, where the goal is to minimize the regret subject to unknown linear constraints. The paper proposes a sampling-based framework--constrained linear Thompson Sampling--that perturbs the empirical estimates and selects actions by solving a perturbed linear programs. Under the hard constraint with a known safe action, it proposes S-COLTS and achieves $\tilde{O}(\sqrt{d^3 T})$; under the soft constraint, it introduces R-COLTS and achieves the same upper bound. Experiments show that COLTS matches or improves upon the regret of prior methods while dramatically reducing computational cost.

STRENGTHS AND WEAKNESSES:
**Strengths**
- This paper extends the Thompson sampling to the constrained linear bandits problem. The design of the coupled noise is novel and practically important. 
- Both COLTS algorithms avoid the complex optimization problems, yielding significant speedups compared to the baselines.
- Theoretical guarantees are provided for both the regret and risk, matching or improving over the previous works.


**Weaknesses**

- In definition 2, should $B$ be a non-increasing map?
- While I appreciate the authors efforts in expanding the analysis, some sections (e.g., look-back scaling) are intricate and could benefit more from additional high-level intuitions.

QUESTIONS:
- Is it possible to derive problem-dependent bounds on the regret and risk, i.e., the bounds will include some instance-dependent parameters like the mean gaps and the feasibility gaps of the arms? 
- Is it possible to generalize the algorithm to the combinatorial semi-bandits problem, where the hard/soft constraints are required for the pulled arms?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies linear bandits with (linear) safety constraints. The paper proposes a sampling procedure akin the Thomson sampling algorithm to avoid solving expensive problems at each round.\nBy assuming the knowledge of a safe action, the authors propose a hard-constraint algorithm (zero violations) S-colts and prove that regret scales with $O(\\sqrt{d^3 T}\\cdot R(a_{safe})$ where $R(a_{safe})$ is the ratio of the gap divided by the safety margin of the safe action (1+). The algorithm needs to know the safety margin of the safe action (bounds on).\nThen the authors provide an algorithm for the soft constraints (sublinear violations) that does not require knowing $a_{safe}$ and achieves a regret and violations bound of $O(\\sqrt{d^3T})$.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is interesting and relevant to the NaurIPS learning community. The writing is good, even if it becomes a bit too dense in the second half, where I think the authors could do a better job at first describing the S-Colt algorithm at a high level, highlighting what the main argument of its correctness is. I also have a minor issue with the complexity-related terms used in the paper. Sometimes, the authors claim that an algorithm is NP-hard (which is very imprecise) or that an algorithm requires solving an NP-hard problem (which is not proven anywhere and probably unprovable formally). This is a minor misuse of terminology, but I think it has to be fixed. More on this in the questions section of the review.\n\nQUESTIONS:\n1. The formatting of Equation 1 is weird. Consider putting regrets and risk after the definitions, and not in the equations\n2. Line 33: What is an NP-hard constraint?\n3. Table 1: Are you sure that solving OPT-PESS requires solving an NP-hard problem at each round? More precisely, are you sure that you can implement *any* instance of the NP-hard problem while running OPT-PESS (or DOSS)? I strongly believe this is not the case. While I think it is true that these methods require solving a “hard” program, you cannot say that solving these problems is NP-hard. Meaning that either you show a reduction from a generic NP-hard problem to the problem that OPT-PESS requires to solve when applied to a specific constrained bandit problem, or you cannot say that running OPT-PESS requires solving an NP-hard problem. You also have issues with randomization, as the “hard” problems that need to be solved by these methods are randomized, and, as far as we know, the average case complexity of these methods might be polynomial.\n4. Line 84: same issue. Also, problems are NP-hard and not algorithms. What you can say is that the classic implementation of ROFUL requires solving a problem X for which there is no polynomial-time algorithm currently known.\n5. Line 119: Why is the L2 operator norm used? I think constraining the vector for each reward cost is what is standard in the rated literature (BwK, for instance). There, you assume that each vector of costs (column of $\\Phi$) is a vector in between $[-1,1]^d$. I think this is more natural, as normalization across different costs does not seem natural\n6. Definition 2: What are $\\eta$ and $H^i$? This is not clear from the current writing. Additionally, Lemma 3 is somewhat confusing. Consider rewriting a bit here.\n7. What prevents you from extending the results to convex non-linear constraints?\n8. What is the connection with the BwK literature and its extension of bandit with constraints? I think this is much more related than what either community realizes. I can refer here to some papers that I think should be compared, as some recent BwK works handle per-round constraints: \"Contextual Bandits with Packing and Covering Constraints: A Modular Lagrangian Approach via Regression\" and \"Beyond Primal-Dual Methods in Bandits with Stochastic and Adversarial Constraints\".\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies constrained linear bandit and introduces the COLTS framework and its variants with algorithmic and computational merits. However, I have several substantial concerns regarding the clarity and consistency of some key definitions and derivations, which I believe deserve careful attention.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. Constrained bandit linear is an interesting area in the online decision-making literature.\n2. The paper provides a theoretical analysis of the proposed approach, along with experimental results.\n\nWeakness/Areas of improvement:\n\n1. The presentation of the paper is hard to follow. In my humble opinion, I would suggest that the quality and the clarity of the presentation could be improved.\n\n2. Some of the claims in the paper could be more elaborated. For instance, the paper claims the contribution as existing works require solving NP hard constraints avoiding this requirement is the focus of the paper. However, it is unclear from the paper why such an NP-hard constraint arises in those works. Some of the works that are cited study a slightly different problem setting  (specifically in their constraint model), and hence, such a claim must be well-explained. \n\n3. There are multiple undefined notations, some errors, etc., identified in the paper. A careful revision will be needed.\n\nQUESTIONS:\n1. $\\Gamma(a)$ is defined as $\\min_u (\\alpha - \\Phi_\\star a)_+^i$, where the variable $u$ is undefined (line 107). \n\n2. The confidence event $Cont(\\delta)$ holds with probability at least $1 - \\delta$ in Lemma~1. However, since it combines confidence intervals for both $\\theta^\\star$ and $\\Phi_\\star$ over all $T$ rounds, the union bound implies that the probability is at least $1 - 2T\\delta$, not $1 - \\delta$. \n\n3. When ensuring constraint satisfaction, the paper states the condition $1 - \\rho \\leq M_t(b_t) / \\Gamma(a_{safe})$. This inequality is inverted: to ensure the constraint is satisfied, one must have $\\rho M_t(b_t) + (1 - \\rho) \\Gamma(a_{safe}) \\leq 0$ from line 173. This misstatement could affect both the theoretical justification and the algorithm implementation. \n\n4. The legend of Figure~2 does not explain all the components, specifically the dashed lines, which may represent standard deviations or baselines. Clarifying their meaning is important for interpretability and reproducibility.\n\n5. The majority of the safe linear bandit works, and some of them cited in the paper, consider a finite action set, rather than a convex polytope as considered in the paper. While the linear constraint that is typically used in safe bandits is of the form that action in each round, x_t^\\top \\mu^* < C, which is different from the risk definition used in this paper. Can you comment and explain and provide some practical motivation for such a constraint?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}