Title: BraVE: Offline Reinforcement Learning for Discrete Combinatorial Action Spaces
Venue: NeurIPS 2025
Accepted: yes
Keywords: Offline Reinforcement Learning

Batch Reinforcement Learning

Combinatorial Action Spaces

Reinforcement Learning

Discrete Action Spaces

Large Action Spaces

Sequential Decision Making
Abstract: Offline reinforcement learning in high-dimensional, discrete action spaces is challenging due to the exponential scaling of the joint action space with the number of sub-actions and the complexity of modeling sub-action dependencies. Existing methods either exhaustively evaluate the action space, making them computationally infeasible, or factorize Q-values, failing to represent joint sub-action effects. We propose \textbf{Bra}nch \textbf{V}alue \textbf{E}stimation (BraVE), a value-based method that uses tree-structured action traversal to evaluate a linear number of joint actions while preserving dependency structure. BraVE outperforms prior offline RL methods by up to $20\times$ in environments with over four million actions.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed to handle combinatorial action spaces which can be very large in offline RL. The authors proposed to construct a tree to sequentially handle each action dimension relying on node Q values and branch V values. The authors further restricted the tree to only dataset actions to sparsify the tree to lower computational cost.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\\\n\nHandling combinatorial action spaces is of fundamental interest to RL. I think the paper is overall well-written and the tree-based method is principled. Positioning the tree structure in offline RL where only dataset actions are used is also very clever and matches that of conservative offline RL. I also like the idea of $q$ and $v$ values that estimate node and subtree values so that the tree does not need to be fully traversed before an action can be picked. \n\n**Weakness:**\\\nHowever, it seems the paper is not complete in several aspects. I believe the paper can benefit from more explanations both theoretically and empirically:\n1. the tree essentially acts as a policy to output actions given input states. That means for every input state we should maintain a tree and more importantly have all its node and branch values **accurately estimated**. However, this seems to be prohibitive for continuous states since we need to build infinitely many trees. How did you deal with continuous states? Did you apply some binning? When dealing with continuous spaces, the network function $f(\\cdot, \\cdot; \\theta)$ typically maps to a distribution like Gaussian, and we can generally hope the network generalize to similar states, but I don't see how that is possible given the way the tree was constructed in BraVE.\n\n2. for the tree to be useful, we need to have accurate estimates of $q,v$. But because $v$ is a summary of values of all subtrees, how did BraVE ensure that the values from unchosen subtrees (evetually leaf nodes) were accurate? Did you need extra assumptions or structures on the network $f$ to ensure unchosen leaf nodes had reasonable values? More generally, if $q$ and $v$ had estimation errors, how did the errors influence action selection? Because of the tree structure, I would expect that the chance of picking a wrong action depends on tree level. This type of discussion would add to the contribution of the paper.\n\n3. I am afraid that the experiments are limited in diversity. In the first paragrah of the paper the authors mentioned applications like healthcare but they were not touched in the experiments. Moreover, some related papers tested the discretized continuous action spaces, that would also add to the credibility of the paper, see ref [1].  Related to this, I found it very strange that IQL can perform well even with this large discretized action space in supp? Why this is the case?\n\nReferences:\\\n[1] Path Consistency Learning in Tsallis Entropy Regularized MDPs, ICML 2018\n\nQUESTIONS:\nBesides the questions in the weakness section, I am curious that can this sequential traversal approach be applied to softmax action selection?\n\nLIMITATIONS:\nNo societal impact applicable. For limitations, the authors have discussed some limitations of the paper, as the tree traversal indeed relies on a fixed predefined ordering of sub-action dimensions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors address the problem of offline reinforcement learning in environment with combinatorial action spaces. The proposed method represent the action space with a tree structure where each node represents a complete action and each level correspond to an action dimension.  Each node is characterized by a value estimate and a vector of branch values for each subtree, these values are estimated by a neural network. The neural network is trained using a regularized TD learning loss as well as a custom branch value supervision loss.\n\nThe resulting algorithm is evaluated on a synthetic dataset from a n-dimensional navigation problem against an offline RL baseline (IQL) and an offline RL with value factorization (FAS). \n\nThe proposed method has similar performance as FAS for low dimensional action space but outperforms both baselines by a lot as the action space grows.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The idea of representing the value function by a tree is interesting and as far as I know novel and present a strong contirbution.\n\n- The paper is well organized, and fairly easy to follow (except the section on the loss calculation). \n\n- The method shows strong performance on the synthetic dataset compared to the baseline. The experiments seem well-designed and they evaluate the method in controlled settings. The results are quite convincing. Some of the description of how the environments are generated could be improved.\n\n**Weaknesses**\n\n1. Why only evaluating on synthetic data? I understand the importance of the experiments proposed to evaluate the algorithm in a controlled setting but it would have made an even stronger case to add an example on a real dataset.\n\n2. The ablation study is an important part of the analysis and lack discussions. The authors provide curves in appendix but do not discuss the results. I believe such a discussion should be included to the main text. \n\n3. Another aspect that is missing and I think crucial is the analysis of the order of the tree. It seems to me that the way the tree is constructed can impact the result greatly, as some actions might be more important than others, if they appear very deep in the tree the performance might be impacted. In the related work discussion MCTS it is mentioned that a sequential structure need to be imposed, however it seems to me that the tree constructed by brave also need to impose some arbitrary structure on the action space. The authors acknowledge this limitation in the conclusion but still I think it could use a dedicated experiment.\n\n4. Explanation of the algorithm sometimes lack clarity\n\n- The loss computation is a key component of the method but lack some clarity. $\\hat{a}$? is not defined, and it is not described how $v$ is learned. \n- I also did not understand why v, the best value of each subtree, needs to be learned and not computed after the training from the estimated q values. The pseudo code provides a little bit more insights but it would help to make the text consistent. \n- Some of the explanation of the results could use more justification, for example when the authors explain the behavior of FAS on high dimensional problem l258.\n- The experimental setup with sub-action dependencies is hard to understand. The number of pits is said to increase to 100%, does that mean the whole state space is a pit?\n\nQUESTIONS:\n-\tWhy a tree and not a graph?\n-\tWhat is the definition of $\\hat{a}$?\n-\tCould you provide more intuition about why the reward cannot be factorized in your experiments?\n-\tWhat is the impact of the order of the action dimension when constructing the tree in BRAVE?\n-\tHow would BRAVE perform on a non synthetic dataset?\n\nLIMITATIONS:\nMost of the limitations are acknowledged by the authors:\n\n-\tDiscrete actions required\n-\tThe ordering of the action can impact the performance (although no guidance is provided on that)\n-\tIt is an offline method, so it depends on the quality of the dataset", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Brave, a novel offline RL algorithm designed to handle high-dimensional, discrete combinatorial action spaces. The paper says that existing methods are either computationally infeasible due to exhaustive evaluation of vast number of actions, or sacrifices accuracy by factorizing Q-values. Brave addresses this trade-off by imposing a tree structure on the action space, allowing it to evaluate maxQ with a linear number of evaluations while preserving the dependency structures of action dimensions. The method employs a Q-guidance to traverse this tree, and these guidance are trained with a behavior-regularized temporal difference loss. The experiments show Brave is significantly outperforming baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Proposes a novel method to cleverly implement Q-learning on large combinatorial action space\n- Algorithm well explained\n- Significantly outperforms other baselines\n\nWeaknesses\n- While the proposed method is about handling large combinatorial action space, the paper limits its scope on offline RL settings, and the contribution specific to offline setting is minor (adding l2 regularization). Considering that the main contribution can be applied to any RL settings, I believe that the paper should have been focused on standard online setting rather than offline setting.\n- While the paper only says about the FAS approach in the paper, actually there have been a number of different approaches toward large combinatorial action space. If we insist learning with Q-learning-based algorithms, we can either learn an action embedding [1], or use a model-based approach through Q=r+V [2].\n[1] Dulac-Arnold, Gabriel, et al. \"Deep reinforcement learning in large discrete action spaces.\" (2015)\n[2] Vinyals, Oriol, et al. \"Grandmaster level in StarCraft II using multi-agent reinforcement learning.\" nature 575.7782 (2019): 350-354.\n  - If we are OK with other type of algorithms, policy-based algorithms (e.g., PPO) or sequence modelling approaches (e.g., Decision Transformer) will be able to easily solve the problem.\n  - Therefore, I am not very satisfied with the empirical evaluations done in the paper; it is even very hard to get a feeling about whether Brave is working or not, as FAS and IQL are mostly broken and any other types of algorithm, or oracle policy, are not included.\n- Furthermore, all three algorithms experimented have a very different behavior regularizations. IQL - quantile regression, FAS - BCQ like, BraVE - l2 regularization. While the difference between BraVE and FAS on how to handle large combinatorial action space is the main focus here, some of the performance differences may have been originated from these different behavior regularizations; I would say this is not a good way to demonstrate the performance of the proposed method.\n\nQUESTIONS:\n- Can we simply use BraVE as an online algorithm if we do not put l2-behavior-regularization? If yes, why should the paper focus on the offline setting?\n- By having additional binary indicator of whether the action is selected or not, Q-function of BraVE seems to have different neural architecture compared to other algorithms. How is this implemented, and how do you ensure a fair comparison with other baselines?\n- As difficulty of the problem (|A| or |S|) increases, the experiments result does not have a clear trend. For example in table 3, FAS shows much better performance on |S|=15625 compared to |S|=25. What is the reason to this?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors introduce BraVE an offline RL algorithm for combinatorial discrete action spaces with multiple dimensions related to different sub-actions.\n\nThe problem of evaluating the action-value function maximum over a large combinatorial action space $\\max_a Q(s,a)$ (given a state $s$) needed for action-value function learning using TD method is alleviated by using dedicated neural network $f$ and organizing \nactions into a tree structure where, given some fixed ordering over sub-action dimensions, in each tree level one sub-action is decided.\n\nThe trick is that the network $f(s,a)$ is trained to output $Q(s,a)$, where $a$ corresponds to some tree node, but also to output maximum values over sub-trees corresponding to all its child nodes.\n\nThis is done by updating $f$ loss with every $Q$ target computation. \n\nSuch $f$ then allows for fast $\\max_a Q(s,a)$ evaluation (linear in number of action space dimensions). \n\nBesides this basic idea authors include many practical improvements  including:\ntree sparsification (to include only nodes which corresponds to actions actually present in the off-line dataset),\nmasking (when number of sub-actions across sub-action dimensions varies),\ndepth-penalty weight (helping to weight more the errors closer to the root node where decisions are more critical),\nand beam search (for better robustness during inference).\n\nAuthors demonstrate feasibility of the method on their proposed benchmark which allows for strongly dependent sub-actions (depending on a given state) with superior performance over compared baselines.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is well written, the presentation is clear, and the proposed method is well motivated.\n\nAlthough there is not much math, I found no errors and typos in the math formulas.\n\nI found the proposed idea of how to cope with $\\max_a Q(s,a)$ evaluation compelling.\n\nI would  like to see the generalization of the proposed method to the standard online RL setting where probably also tree structure has to be updated due to possible distribution shift.\n\nGenerally, I have no complaints about the paper.\n\nQUESTIONS:\n1) I am missing information on how the $f$ network is initialized\nin order to fit with $\\max_a Q(s,a)$ evaluation. One could  train it with some lower bound on $Q$ values, but it seems to introduce an inconvenient bias. Could you clarify this?\n\n2) From Figure 13a, it seems that increasing beam width brings rather problems (decrease in return). Could you comment on this?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}