Title: Multi-Reward Best Policy Identification
Venue: NeurIPS 2024
Accepted: yes
Keywords: multiple rewards

best policy identification

pure exploration

active exploration

reinforcement learning

sequential decision making
Abstract: Rewards are a critical aspect of formulating  Reinforcement Learning (RL) problems; often, one may be interested in testing multiple reward functions, or the problem may naturally involve multiple rewards. 
In this study, we investigate the _Multi-Reward Best Policy Identification_ (MR-BPI) problem, where the goal is to determine the best policy for all rewards in a given set $\mathcal{R}$ with minimal sample complexity and a prescribed confidence level. We derive a fundamental instance-specific lower bound on the sample complexity required by any Probably Correct (PC) algorithm in this setting. This bound guides the design of an optimal exploration policy attaining minimal sample complexity. However, this lower bound involves solving a hard non-convex optimization problem. We address this challenge by devising a convex approximation, enabling the design of sample-efficient algorithms.  We propose MR-NaS, a PC algorithm with competitive performance on hard-exploration tabular environments. Extending this approach to Deep RL (DRL), we also introduce DBMR-BPI, an efficient algorithm for model-free exploration in multi-reward settings.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe present article extends the track-and-stop approach of Garivier et al. to a multi-reward MDP setup. Given an MDP problem with a finite number of reward functions the aim is to develop an algorithm that learns optimal policies for all reward functions simultaneously. Under (drastic) assumptions the authors present a sample based variant of track-and-stop in the multi reward setup. The algorithm is based on replacing the theoretical model complexity $T^*$ (in a multi reward variant) from Garivier et al. by an upper bound $U^*$ that can be estimated. Estimating $U^*$ during the exploration phase results in a practically implementable termination condition that results in worse complexity than the theoretical termination condition. The algorithm is tested on a few (very similar) tabular examples and compared to simple algorithms. An educated guess is performed to design a deep variant of the algorithm that is tested on a multi reward card pole variant and deep sea. Results beat easily the (terrible) results of the benchmark algorithms used.\n\nSTRENGTHS:\nThe article is extremely dense with information, perhaps it would have been better to split the article in 2 or 3. The theoretical development is an extension of several earlier articles of the authors with very detailed mathematics. While the tabular examples are of limited practical interest (the assumptions are just too strong) the deep variant is interesting. While I am not so sure about the relevance of the tabular setting, the relevance in the deep setting is obvious (and was realised of the authors for cardpole). The ability of generalisation of NN makes it interesting to train the network simultaneously on perturbations of the true reward function in order to learn a more stable policy that might work well for even more reward settings.\n\nThe scientific quality of the article is very high. There is hardly any typo. I appreciate a lot the critic discussion of limitations, this is far above the rather low scientific standard in ML. I could not check the entire proofs of the appendix, what I read seemed solid. \n\nGood article! I am curious to follow up on the future development of the robust DQN variant.\n\nWEAKNESSES:\n- I did not enjoy reading the article very much as I was pushed into first reading other articles to get a rough understanding of what is going on. Even reading the basis of the present article ([17], [40]) was not enough. To get an idea why $T^*$ is considered one needs to go all the way back to [20], and even further. It feels a bit like a community wants to stay among each other, the usual RL researcher is excluded by the style of writing. I would suggest to completely skip the discussion in the end (almost a page) and instead use the space to explain the reader what the technique is about and why the theoretical estimate naturally lead to Algorithm 1.\n- The assumptions are drastic and should be discussed more clearly. I guess the authors have bandit background and skip discussions of issues that are typical for bandits. In particular, assuming knowledge of rewards and/or the reward gaps. While this is not unusual in bandits it is very much in RL. I am perfectly fine with such theoretical results in particular, as the authors implemented an educated guess algorithm in the deep setting that addresses this issue with the reward gaps.\n\nQUESTIONS:\nHere is a number of questions and comments.\n\n- $\\alpha$ should also be an input for the algorithm 1. Why is there no dependence on $\\alpha$ in the theoretical results?\n- How do you compute U in Algorithm 1 without knowing the rewards gaps? You estimate M, but what about the gaps? They are estimated in the deep setup, why not in the tabular setting?\n- I think the algorithms are not sufficiently explained. In the tabular case $M_t$ should be explained in more detail. What shall the reader get from \"use estimate $M_t$ of the model\", \"update $M_t$\" without knowing $M_t$? In the present form the article is hardly comprehensible (even for me as a Mathematician). In that generality (what estimate is used?) I am a bit sceptical about the validity of Theorem 3.3, but as said, I could not try to understand all proofs.\n- The use of Borel-Cantelli in the proofs is not correct as it is. The events are not independent and this direction of Borel-Cantelli requires (pairwise) independence. I am pretty sure the SARSA paper contains the details on how to use a conditional independence version of Borel-Cantelli.\n- I am a bit puzzled about the connection of Theorem 3.1 and 3.3. If I am not wrong, Theorem 3.1 is a bandit theorem, requiring to play independent \"arms\" $(s,a)$ while Theorem 3.3 (the algorithm) is an RL theorem requiring rollouts $(s_0,a_0, s_1,a_1,...)$. The latter is obviously much harder and depends a lot on the exploration method. Unfortunately, I have six 50pages NeurIPS papers on the table and cannot go fully through every proof. Could you add two sentences why the theorem should hold? For instance, SARSA convergence holds by rewriting SARSA as Q-learning plus an error that goes to zero by the exploration condition. Here, the situation is much harder.\n- A crucial point is to replace $T$ by $U$. In line 169 you mention an upper bound for $U$, which has an additional $1/\\Delta^2$ compared to the theoretical bound. Is there a better theoretical bound? The authors should at least discuss briefly in the main text that there is something non-trivial going on (as in the SARSA paper where on-policy exploration is essentially as good as off-policy exploration). As it is it one quickly overlooks the difference in the algorithm to the \"allocation\".\n- I cannot see a scaling in the number of rewards. Is there any theoretical understanding on how $\\tau$ scales in the number of rewards?\n- How does your algorithm compare to just train for one reward after the other? Does that make sense? The training time for card pole seems quite high but I might be wrong. Typically card pole plots show the number of episode (ca. 200-300), not the number of steps. Is it clear your algorithm is faster than just training 5x card pole? How about comparing your algorithm to training 5x card pole with your different reward settings but keeping the neurone network from the reward before (and using some replay buffer to avoid overfitting)? My feeling is that such a naive approach might be better than the benchmarks used. \n- Similar question: Comparing your deep algorithm to performing several deep trainings for different rewards. Is your approach particularly suitable to avoid overfitting? Does your algorithm help for generalisation (as you see in the cardpole example?).\n\nLIMITATIONS:\nThe tabular setting requires drastic assumption (but gives a clean analysis), the deep setting is an interesting educated guess but does not allow a mathematical analysis. I would say that is quite normal for RL, except, perhaps, the deviation of the educated guess Algorithm 2  from the tabular algorithm 1 is pretty big.\n\nThere is no good benchmark to compare to, this makes the numerical results a bit boring. Perhaps a better benchmark would be to compare to training individually the policies. While that might even beat the proposed algorithm in the tabular case (for few rewards) I doubt the same holds for the deep case as the NN would only remember to solve the last reward problem.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of best policy identification for RL with multiple rewards. The goal is to efficiently identify the best policy for given rewards with a high-level confidence. Authors provide an instance-dependent lower bound for the studied problem and introduce a provably-correct algorithm for a convex approximation of the original problem in the tabular setting. Extensions to deep RL is discussed with numerical results.\n\nSTRENGTHS:\nThe authors demonstrate how to efficiently identify optimal policies across multiple rewards. The strengths of this work are summarized as follows:\n\n1. The studied setting is interesting and of practical concern when we seek to optimize the performance across a set of rewards.\n\n2. An instance-dependent lower bound is identified for any Probably-Correct algorithm in the setting of multi-reward best policy identification problem.\n\nWEAKNESSES:\nWhile this paper clearly articulates the idea of performing, here are some problems and potential improvements that authors are suggested to address and consider. More specifically,\n\n1. Environments for the deep RL part is too simple. The Cartpole swing-up and DeepSea environments cannot fully demonstrate the performance of the proposed algorithm in more complex, real-world scenarios. It would be beneficial to include experiments on more challenging benchmark environments for better assessment of scalability and practical applicability.\n\n2. The policy optimality is only considered and defined for stationary deterministic policy (as in definition 2.1), which can be too restrictive. It is not clear when considering the set of Markovian policies (which can be stochastic), whether the proposed lower bound still holds, and whether the performance of the algorithm is still optimal. \n\n3. Theoretical guarantees for the deep RL extension is unavailable. Sample complexity bounds are only provided for tabular settings, which leads to the dependencies on the cardinality of state and action space. And empirical studies for the deep RL settings are not sufficiently convincing due to the simplicity of the environments.\n\n4. In terms of the theoretical results of the lower bound, the proof structure closely follows the prior results (e.g. Marjani et al. 2021, Taupin et al. 2022) in single-reward RL. It is not quite obvious what are the main technical challenges and novelties of extending the prior results (e.g. Marjani et al. 2021) from single-reward RL to multiple-reward RL.\n\n5. While the studied setting can be interesting, the relationship between the MR-BPI problem and reward-free RL as well as multi-objective RL somewhat remains unclear throughout the context. Though discussion has been provided in Section 1 and 5, I am not fully convinced that reward-free RL cannot solve the concerned practical scenario in multi-reward best policy identification problem. Indeed, reward-free RL assumes rewards are unknown, whereas the studied settings assume the knowledge of rewards. As a result, it is not surprising to see that properly utilizing the knowledge of rewards can lead to better performance as shown in the numerical results: the proposed algorithm (MR-NaS) significantly outperforms RF exploration strategies (RF-UCRL, ID3AL). However, reward-free RL is a more general type of algorithms and can be particularly useful in practice when it is hard to accurately learn rewards or when rewards are sparse.  As such, the emphasis of the two settings are rather different. It might not be a fair comparison, and it is desirable to provide the fundamental reasons that can explain such performance improvement in numerical experiments. Therefore, more thoughtful insights should be provided to clearly explain the difference and relationship between these settings.\n\n6. Some minor aspects:\n    - Grammatical errors need to be addressed, e.g. Line 354, line 91.\n    - In line 75-80, if only deterministic policies are been considered, it is more appropriate to write $a_t = \\pi(\\cdot|s_t)$ etc. Do not use the probability simplex notation for policies.\n\nQUESTIONS:\n1. Do we treat MR-BPI as dedicated exploration strategies for multi-objective RL?\n\n2. Most reward-free RL focuses on episodic settings, whereas this paper studies discounted settings for multi-reward MDPs. Is there any particular reason for choosing this (simpler) setting? Do you foresee any technical challenges that can be difficult to resolve in episodic multi-reward settings?\n\n3. In Line 81 - 86 (Set of rewards), for each reward vector, each coordinate $i$ represents the reward for the $i$-th state-action pair, which is a scalar, why do we need the canonical basis $\\mathcal{R_{canonical}}$ of rewards, where each element is a vector? (When you write $\\mathbb{R}^{SA}$, I assume each element is a real number, not a vector). Do you assume for each $(s, a)$, there is a different reward function? Could you provide a concrete example of your definition of \"set of rewards\" with your notation? I assume you intended to say there exists $m$ (global) reward functions, and this set of functions $\\mathcal{R}$ is thus in $\\mathbb{R}^{m \\times SA}$.\n\nLIMITATIONS:\nThere are no unaddressed ethical considerations in the studied context.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the challenge of identifying the best policy in RL when there are multiple rewards. The authors get a lower bound on the sample complexity and design an optimal exploration policy. The authors propose two algorithms: MR-NaS for tabular environments and DBMR-BPI for Deep RL. These algorithms efficiently identify the best policy across multiple rewards in RL.\n\nSTRENGTHS:\nThe paper presents a comprehensive and well-balanced analysis of theoretical and empirical results. The appendix provides supplementary evidence that strengthens the authors' arguments.\n\nWEAKNESSES:\n1. The paper  is inspired by [17], but it would be better to explicitly acknowledge this inspiration in the main part, rather than mention it at Remark C.2. Furthermore, a more in-depth discussion of the challenges in proof caused by the novel forcing policy would strengthen the paper's contribution.\n\n2. Even though more details is covered in the appendix, the paper should provide some details on the convex optimization. For example, a discussion of the computational costs associated with these methods would provide valuable context for readers. Sometimes, the computational cost of convex optimization methods are high. \n\n\n3. The abstract would be better if providing a more impressive motivation for multi-reward RL, emphasizing its significance and potential impact.\n\n4. Didn’t put some innovative aspects in the main text, leaving vital details to be discovered in the appendix. This can lead to important contributions being overlooked.\n\nQUESTIONS:\nPlease address the issues mentioned in the weaknesses.\n\nLIMITATIONS:\nAs noted in the paper, the assumption of a unique optimal policy limits the method's applicability to a broader range of scenarios. The computational cost of convex optimization may prevent it to deal with large-scale dataset.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-29"}