Hedge Your Actions: Flexible Reinforcement Learning for Complex Action Spaces
Authors: Norio Kosaka
Ayush Jain
Xinhu Li
Kyung-Min Kim
Joseph J Lim
Venue: Submitted to ICLR 2023 (2023)
Keywords: Efficient Reinforcement Learning
Large Action Space
Listwise Action Retrieval

Abstract: Real-world decision-making is often associated with large and complex action representations, which can even be unsuited for the task. For instance, the items in recommender systems have generic representations that apply to each user differently, and the actuators of a household robot can be high-dimensional and noisy. Prior works in discrete and continuous action space reinforcement learning (RL) define a retrieval-selection framework to deal with problems of scale. The retrieval agent outputs in the space of action representations to retrieve a few samples for a selection critic to evaluate. But, learning such retrieval actors becomes increasingly inefficient as the complexity in the action space rises. Thus, we propose to treat the retrieval task as one of listwise RL to propose a list of action samples that enable the selection phase to maximize the environment reward. By hedging its action proposals, we show that our agent is more flexible and sample efficient than conventional approaches while learning under a complex action space. Results are also present on \url{https://sites.google.com/view/complexaction}.

Under review as a conference paper at ICLR 2023
HEDGE YOUR ACTIONS: FLEXIBLE REINFORCEMENT LEARNING FOR COMPLEX ACTION SPACES
Anonymous authors Paper under double-blind review
ABSTRACT
Real-world decision-making is often associated with large and complex action representations, which can even he unsuited for the task. For instance, the items in recommender systems have generic representations that apply to each user dif- ferently, and the actuators of household robot can be high-dimensions and noisy. Prior works in discrete and continuous action space reinforcement learning (RL) define retrieval-selection framework to deal with problems of scale. The retrieval agent outputs the space of action representations to retrieve few samples for a selection critic to evaluate. But, learning such retrieval actors becomes increas- ingly inefficient as the complexity in the action space rises. Thus, we propose to treat the retrieval task one of listwise RL propose list of action samples that enable selection phase to maximize the environment reward. By hedging its action proposals, show that our agent more flexible and sample efficient than conventional approaches while learning under a complex action space. Website
INTRODUCTION
An essential goal of reinforcement learning (RL) is to solve tasks humans cannot solve, such as those with innumerable action spaces. For instance, recommender systems have a set of millions of discrete items, and robotic control requires acting in continuous dimensions. In such domains, all the actions cannot be enumerated feasibly to perform RL. Thus, actions are associated with continuous parameterizations representations that enable generalized learning. We consider continuous and large parameterized discrete action spaces as problems of innumerable actions.
Typically innumerable action space tasks are solved in two phases: retrieval and selection (See Fig. D. QT-Opt (Kalashnikov et al. 2018) learns robotic manipulation by retrieving actions from a distribution fitted using the cross entropy method and selecting the action that maximizes learned f-fuctioon. Similarly, Dulac-Arnold et al. (2015) performs large discrete action RL by acting in the space of action representations, retrieving Enearest-neighhee Continuous actor-critic approaches like DDPG and SAC (Lillicrap et al. 2015) Haarnoja et al. 2018) directly learn actor with learning objective of retrieving an action that maximizes Q-function.
and selecting with a Q-function.
However, action retrieval in these approaches takes the form of a single action, a distribution, or local neighborhood defined on the action space. While these forms of retrieval work when the selection Q-function is smooth over the actions, it becomes a limiting factor in complex action spaces. The action space can be imprecise, noisy, high-dimensional, the task (such as in recommender systems), leading to a mismatch between the action representations and their effects on the task. Therefore, enable efficient reinforcement learning in complex and innumerable action spaces, we aim to address the goal of performing robust action retrieval.
independently derived from
Our critical insight to perceive the problem of action retrieval as one of listwise reinforcement learning (Sunehag et al. 2015). Concretely, the retrieval network is considered an RL agent with a modified action space of picking a list action candidates, trained to enable the selection Q- function to maximize the environment reward (Fig. D.. Listwise retrieval improves the efficiency of RL in two ways. Firstly, retrieval lags behind selection during initial training because trained with RL loss. By hedging diversifying the retrieved actions. candidates to maximize, enabling directed exploration of the task. Second, listwise retrieval can learn to adapt the list composition (e.g., diverse similar) training because the actions
selection phase gets better

---

Under review as a conference paper at ICLR 2023
RecSys
List
Discrete Actions
State
Action
State
Retrieval π
Selection
Q
Robot Control
Action
Continuous Actions
Action Retrieval as Listwise RL
Figure I: Large discrete action set and continuous action space tasks cannot be solved with single Q-function because it infeasible to enumerate all the actions. A common approach to (a) retrieve k actions, and (b) select only from those actions. We posit that the retrieval task can be generally seen as listwise RL, where the retrieval agent must learn to output the list of candidates as a whole. This enables flexible learning in complex innumerable action space RL.
constrained to a predefined distribution or local neighborhood form. We show that the flexibility of listwise retrieval is crucial in complex action spaces.
To this end, we propose novel framework FLAIR, Flexible Listwise Actlon Retrieval, that can incrementally build a list of & candidate actions without enumerating all possible list combinations. We extend cascaded DQN (Chen al. 2019c Jain et al. (02T) to continuous action space resulting in cascaded DDPG framework with l actors and k critics. Specifically, action for each list index while considering the state and the partially built list as input. Each cascaded actor is trained to maximize its associated critic's value, while the crities are trained to maximize their environment reward. Overall, each actor-critic pair learns to retrieve a candidate action that can optimize the environment reward when combined with the current list of candidates.
actor outputs candidate
Our primary contribution introducing the problem complex innumerable action spaces in re- inforcement learning. We make the retrieval-selectinn approach flexible by incorporating listwise RL. We demonstrate our proposed method FLAIR learns to perform listwise action retrieval that en- ables flexible and efficient decision-making in innumerable action space tasks, such recommender systems, a novel mine-world environment, and continuous control.
2
RELATED WORK
2.1
LARGE DISCRETE ACTION SPACE RL
Continuous action space is a common problem RL, with several on-policy algorithms like TRPO (Schulman et al.] 2015). PPO (Schulman al. 2017), and off-policy algorithms like
DDPG (Lillicrap et al. 2015). SAC (Haarnoja et al. 2018) proposed for it. Dulac-Aroldd et al. Chandak et al. 2019) address the issue large discrete action spaces in reinforcement learning by learning an actor-critic framework in the space of action representations. Likewise, several approaches are based on the retrieval and selection framework (Tan et al.) 2019 Chen al.) 2019b Tessler et al. 2019 Narasimhan et al. 2015). We show that such conventional approaches suffer in complex action spaces due their predefined constraints on the form of action retrieval.
(2015)
Recently, Hubert et al. (2021) propose model-based planning in innumerable action spaces. Their insight of using a retrieval and selection framework similar ours, to avoid enumerating all pos- sible actions. They propose important sampling-like loss modification to enable learning with a subset of samples. However, unlike our work, this work focuses on the selection network, while their retrieval network is assumed to be specified proposal distribution and not learned. (Van del Wiele et al.] 2220)) learn a proposal distribution that can do arg max Q by supervised learning on tar- gets obtained from stochastic maximization procedure the Q-function. However, this stochastic
2

---

Under review as a conference paper at ICLR 2023
search procedure is susceptible the action space's complexities. Chen et al. (2019a) train fixed tree-structured policy network that reduces Q-function evaluation to logarithmic complexity in the number of actions. However, this cannot support generalization in the action representation space is thus inefficient and unscalable.
2.2 LISTWISE ACTION SPACE
Listwise RL has combinatorial action space in the list size. Also known as slate RL, it has applica- tions in recommender systems (Sunehag al. 2015 le et al. 2019b; Chen al. 2019c). In recent work, Jain et al. (Jain et al. (021)) proposed generalizable policy architecture to perform better list- wise RL varying action spaces by considering interdependencies of the available actions as well as items within a list. Their architecture stands for Cascaded DQN(CDQN; Chen et al. (2019c)), where a set of Q-networks sequentially build a list while passing the intermediate list information onto the next Q-network to work out the action at the following list index. However, CDQN is based on a discrete action space DQN, and thus, unscalable to innumerable action spaces. In this work, we propose how to perform cascaded listwise RL in continuous action space achieve action retrieval for innumerable action space tasks.
2.3 COMPLEX ACTION SPACE
A common form of complex action space arises in tasks that learn action representations inde- pendently from the task. [Tenenhollt Mannor (2019) use expert demonstrations to extract action representations based on co-occurrence of actions. JJin et al. (2020) propose an unsupervised frame- work to obtain action representations by encoding sets action observations. While representation learning is useful for reducing the complexity of action space, it is conventionally done in task- independent manner, which results in potentially challenging representations for action retrieval.
3 PROBLEM FORMULATION
A crucial goal of RL is solve decision-making tasks that humans cannot solve, such as recom- mending the appropriate item to user from a catalog of millions of items. Such problems become incredibly challenging when agent's and environment's 's action interface is complex. For instance, complex robots require nigh-dimensione control and real-wold applications can have environments where a large stochastic noise is applied to an agent's actions. In RecSys, the action representa- tions often come from lask-independent domains, such as unsupervised learning on general item characteristics (Jain al.) 202T] 2020). Generally, we define complex action spaces where the representations of actions available to the agent are not directly amenable to the task it is solving.
3.1 RL WITH INNUMERABLE ACTION SPACE
We consider unified setup for RL with large discrete action sets and continuous action spaces. Specifically, our task follows a Markov Decision Process (MDP), defined by tuple {S, A, T, R, } of states, actions, transition probability, reward function, and a discount factor, respectively. When the action space A is continuous, it has a D D-dimensional continuous parameterization, For discrete action sets such that |A] >> we use D-dimensional action representations Ga E R to denote an action E Å. This encodes the characteristics of the action accessible to the agent, for example, a compressed representation of image and text features for each item in recommender systems. Furthermore, at each time step in the episode, the agent receives state observation 8, E from the environment and acts with a; E A. Then, receives the new state after transition 8141 and a reward rt. The objective of the agent is to learn policy r[aas) that maximizes the expected discounted reward over evaluation episodes, E, [E,r 2",r--1re)
R
3.2 RETRIEVAL AND SELECTION FRAMEWORK
Value- based methods such as Q-learning Watkins & Dayan 1992 Mnih et al., 2015) enable ef- ficient reinforcement learning by being suitalle for off- off-policy training. However, for innumerable action spaces, acting and training Q-functions becomes infeasible due to its reliance on computing arg maxaea Q(s, a). To make it feasible, prior works leam two-stage framework A retrieval
3

---

Under review as a conference paper at ICLR 2023
Reward
Vi
V2
Critic;
Critic;
Critic,
Critica
State
FLAIR employs cascaded DDPG architecture to generate listwise actions as the retrieval agent. Each actor и; outputs continuous action a; while taking in the state and currently built list [01,...Q;-1] as input. Likewise, each critic Q; learns to compute the episodic return based on the state, current list, and the policy's action. The cascaded DDPG architecture of retrieval is trained in conjunction with the selection Q-function.
Figure 2:
stage obtains small subset of candidate action(s), which the Q-function evaluated over in the lection stage. For discrete action space, there is an additional step of finding k-nearest neighbors in the action representation space to retrieve actual discrete actions, based similarity measure such as Euclidean distance (Dulac-Arnol) et al. 2015). In prior work, QT-Opt (Kalashnikov et al.) (018) retrieved Q-function maximizing actions using distribution fitted via cross entropy method, while actor-critic approaches train deterministic actor in DDPG (Lillicrap et al. 2015) stochastic actor in SAC (Haarnoja et al. 2018) to maximize the Q-function directly.
3.3 CHALLENGES OF ACTION RETRIEVAL IN COMPLEX ACTION SPACES
Action retrieval is challenging in complex action spaces when the manifold of action representations difficult to act and learn. In simple RL tasks, the optimal policy (a]s) relatively smooth over the action space. Thus, small values of da. Thus, prior retrieval agents relying on k-nearest-neighhors over discrete action representation queries have wide margin of error regarding action selection. Similarly, continuous actor-critic approaches that output a point action query or a distribution (usually Gaussian) over the action space can learn well.
π (als) 5) and à + da expected to have similar behavior in the task for
However, in RecSys or nigh-dimensiioal noisy robotic control, nearby actions a and a-ôa may have different environmental effects. This because the action representation space may not correspond to the task, and the relevance of actions can even vary depending on the state. For example, laptop and book may be similar or very dissimilar for different users. Therefore, such complex innumerable action spaces, prior methods which assume a predefined structure of outputting over the action space (like k-NN, smoothness, similarity local neighborhood) are more challenging and inefficient to learn.
APPROACH
Our goal is to design an action retrieval framework robust to complex action spaces to address the
challenges in Sec. 3.3 Our central insight is that the problem of retrieval of action samples is cquivalent to reinforcement learning in a listwise action space (Sec. 4.I To enable listwise RL in the continuous space of action representations,
adapt prior work listwise DQN (Chen al.) 2019c; Jain et al. 202T) by learning to incrementally build a list of actions using cascaded actor-
crities (Sec. 4.2

---

Under review as a conference paper at ICLR 2023
4.1 FLAIR: FLEXIBLE LISTWISE ACTION RETRIEVAL
We propose FLAIR, a listwise RL framework for learning to retrieve action samples for RL with complex action space. Listwise RL supports directed exploration during initial training by enabling the selection Q-function to maximize the environment return better. Further, it allows the actor to be more robust to complex action spaces by having more flexibility in the actions it can sample. Together, by learning to sample list of actions, our method can enable more sample efficient RL in discrete and continuous complex action spaces.
4.2 CASCADED ACTOR-CRITICS FOR CONTINUOUS LISTWISE RL
We require a continuous listwise RL agent to sample a list of action representations from the action space. Listwise RL itself is combinatorially complex problem because there can be a large number of lists that can result from various action combinations. In conjunction with the innumerable action space, this makes listwise RL challenging. To tackle this problem discrete action spaces, (Chen) et al. (019c) proposed Cascaded DQN where k cascaded Q-networks learn to incrementally build list candidate actions.
In this work, we extend the idea of cascaded networks to continuous action space listwise RL (Fig. 22. We propose to use cascaded actor-critics that Iearn to output a constant action for each list index. Each actor-critic pair is tasked to learn retrieve action sample complementary to the currently built-up list input to maximize the environment reward.
Policy Architecture: We choose Deep Deterministic Policy Gradients (DDPG) (Lillicrap et al. 2015) as our base continuous action space algorithm, but any continuous action space approach can be extended to cascaded architectures. Specifically, the 7th critic Q-function is trained maximize the environment reward while being conditioned on the list built upto (i - 1)th index. Subsequently, the actor is trained output an action that maximizes the 7 critic, while conditioned on the current list. The actor is trained by backpropagating gradients through the action to the critic, as in DDPG. All actors and critics share list encoder that flexible to the size of actions. For our experiments, we use Deep Set (Zaheer et al. (017)) as the choice of list encoder. More details on the policy architecture and implementation are present in Appendix
ENVIRONMENTS
We evaluate ARDDPG on the following three complex action space scenarios where the agent learns to act in the following task that each of which is equipped with the complex action space; (1) Tools reflecting the breakable object in 2D Grid World to shorten the optimal path the goal, (2) Items reflecting the characteristic of items to recommend items relevant to user, and (3) Actions reflecting the rotation of joints of a robot to perform task. Figure 3 provides an overview of the tasks, the action space construction, and an illustration of the expected behavior. More environment details such as tasks, action representations, and data collection are present Appendix A
5.1 MINING TASK: MINE WORLD
We modify the grid world environment (Chevalier- Boisverr et al. 2018 where an agent navigates a 2D maze while removing mines by appropriate pick-axes to reach a goal as soon as possible. As the action space, the agent always has access to 4 direction movements, which are going up, down, right and left. Further, the agent can use one of the many pick-axe actions to remove mines that are placed in the maze randomly. The digging of mines can result in breaking the mine or another mine, depending on the tool and the type of mine. Given the full observation of the maze, the agent needs to learn to navigate to the goal as soon possible. However, sometimes there will be some mines blocking the way, so the agent also need to apply correct tools to break the mines. Successful digging and reaching the goal yield rewards the agent. Each action is associated with a domain- defined action representation. We make the complex action space, by introducing Gaussian noise in each dimension. So, the manifold of the action representation space noisy and imprecise.
5

---

Under review as a conference paper at ICLR 2023
Discrete tasks
Continuous RL task
MuloCo Reacher
Dig Mine Grid World
Recommender System
Recommend
Agent
Break
Mites
Agent
2
Roach Goal
Action oet
Item
Ground-cuuth
A
Complex
Make Action Representation Space Complex
Increase Sitesize Add noise action representation
Enlarge Dimension
Figure 3: Environment Setup. (Left) In Hine-World, the red agent must remove mines to reach the green goal. (Middle) In simulated recommender system(RecSim), the agent selects a item given the set of items. (Right) In Continuous control tasks, the agent sends the right control signals to the robot to reach the goal.
5.2 SIMULATED RECOMMENDER SYSTEM: RECSIM
Recommender system (RecSys) is a natural application of large discrete action space RL for stance, there can be a large number of news articles videos are recommendable every day. The agent needs to recommend a right item out of a large set items given the user preference informa- tion from the environment. We use RecSim (Ie al. 2019a) to simulate user interactions and equip with the larger set items. In the item-set there are two sizes the item-set 10000 and 5000 items. We assume fully observable environment with the state the preference vector and the action representations as item characteristic. So, the agent needs to find the most relevant item from the large item-set based the given the user preference information each time-step.
5.3 CONTINUOUS CONTROL: REACHER
The 2D Reacher MuloCo environment Todorov et al.] 2012) is simple two-link planar robot tasked to reach randomized target location, originally with 2-dimensional action space. We modify the action space to become complex and 50-dimensional. Specifically, the 50-D action space accessible to the agent, goes through non- non-linear matrix transformation to result in the 2- action space. Further, we introduce a binary state variable that randomly switches one of two matrix transformations at the start of every episode. While the transformation is still smooth, this environment presents a demonstration of how a simple environment can also become complex when the action space is unsuited for the task.
EXPERIMENTS
design experiments to answer the following questions in the context of RL with complex innu- merable action spaces; (1) How effective is listwise retrieval in Complex Large action space com- pared to conventional forms retrieval? (2) How various action retrieval methods scale with different complexities of action representations? (3) Is method sensitive to the choice of list encoder? (4) Does incorporating listwise metrics like diversity improve the performance FLAIR?
6.1 EFFECTIVENESS OF LISTWISE RETRIEVAL IN COMPLEX LARGE ACTION SPACE
We evaluate our proposition against the following baselines, ablations, and variations of our method to validate the effectiveness of listwise retrieval.
6.1.1 BASELINES AND ABLATIONS
DDPG k-NN: follows Dulac- Dulac-Arrood et al. (015)) to train a DDPG-based actor that outputs a single query in the action representation space. k nearest neighbor queries retrieved and given the

---

Under review as a conference paper at ICLR 2023
MineWorld
RecSim
Reacher
Eval Success Rate
Eval Return
3.4-
-19
-96
-123
3.
0.75
1.50
2.25
0.0
0.5
1.0
1.5
150
0.00
1.25
2.5
3.75
5.00
Env Stops
Env Stops
Env Steps
FLAIR (Ours) *** FLAIR Diversity
DDPG2 NN
DDPG2 tSNE
DDPG K-NN
Prior Based
K-N CEM
Joint
FLAIR w/o Cascading
DDPG
DQN (Oracle)
Figure 4: We evaluate FLAIR against baselines on the complex action space. The results are aver- aged over random seeds, and the seed variance is shown with shading.
selection critic. Like the original paper, the actor is trained with gradients from the selection Q network. We adapt this method for continuous action spaces by sampling l actions in Gaussian centered at the selected action query.
In trying to make DDPG k-NN more robust, we include a stronger baseline that trains an extra retrieval critic network, only train the retrieval actor. The selection Q-network is just used to act in the environment.
DDPG2 k-NN:
DDPG2 k-NN t-SNE: One potential solution to complex action space is to perform dimension- ality reduction as a pre-training step over the action space. We apply t-SNE to the provided action representations and subsequently train DDPG2 k-NN. This method not applicable continuous actions.
CEM: This baseline follows QT-Opt Kalashnikov et al. 2018 in replacing the learned retrieval actor with a cross entropy method (CEM) (De Boer et al. 2005). Note that in terms of budget of evaluating the selection Q-function, CEM is very expensive as it requires multiple iterations of fitting a distribution over action samples. This method only applied to the continuous action space environment.
This method evaluates distribution-base retrieval actor in the continuous action space, trained with SAC. The retrieved actions are sampled from the learned distribution, and the actual discrete actions selected the nearest neighbors of each list query.
SAC k-NN:
DDPG: This the standard DDPG baseline for continuous action space tasks.
DQN (Oracle): We provide the result of DQN as the upper bound of our method on discrete action space tasks. It is oracle because has access to the entire list of actions as the retrieved samples.
FLAIR (Ours): We perform listwise action retrieval with a cascaded DDPG architecture.
Joint: Instead of performing cascaded listwise action retrieval, an alternative is to train an actor with joint action space of x D. Then, the action samples can extracted from the joint action at intervals of D dimensions each. This baseline is expected to be hard to learn because it does not utilize the fact that different dimensions are not completely independent, but rather form groups of actions.
FLAIR w/o Cascading: We remove the list action encoder from the cascaded actors and crities, such that each actor-critic pair acts as an independent agent. By initializing the networks with unshared weights, these networks act multi-agent network without any communication. Thus, the cascaded actors are unaware of other actions in the list and cannot perform appropriate listwise RL. This ablation of our method demonstrates that FLAIR's listwise RL. helps improve task performance.
FLAIR Diversity: We augment method with reward to encourage diversity in the list actions for the retrieval agent only. This reward is not necessarily helpful as diversity in the retrieved list can sometimes be lead to interference in task learning.

---

Under review as a conference paper at ICLR 2023
MineWorld
RecSim
1.0
Eval Success Rate
Eval Return
0.6
0.4
0.2
0.0
2
0.00
0.75
1.50
2.25
3.00
0.0
0.5
1.0
1.5
2.0
Env Steps
1a6
Env Steps
1e5
K-NN Variants
FLAIR (Ours)
Diversity
DDPG2 K-NN
DDPG2 N-NN
DDPG K-NN
Prior Based
Other baselines
SAC K-NN CEM
Joint
FLAIR Cascading
(Oracle)
Figure 5: Consistency of our proposition across different action space conditions in terms of (Left) the action-dimension and (Right) the action-set size.
6.1.2 RESULTS
Figure shows the complete experiment results.
Mine World: We observe that FLAIR outperforms the baselines, while FLAIR without Cas- cading is able to achieve some success well. It is complex action space two different kinds of discrete actions available to the agent - navigation and mining actions. Furthermore, the stochastic noise added the action representations makes the task
more complex. There- fore, because of the flexibility in the retrieval actor, FLAIR can learn well in this environment, and achieve close optimal performance as DQN (Oracle). Note that all methods and baselines learned on smaller versions of the maze with simpler action spaces.
FLAIR and its diversity variant outperform all the baselines and reach almost the sam- ple efficiency DQN (Oracle). The diversity reward helps FLAIR to explore in the beginning of training, but is not so helpful later. Joint, FLAIR w/o cascading are in the middle of the perfor- mance levels, while DDPG-KNN variants and SAC k-NN follow next. Crucially, we observe that DDPG2 k-NN outperforms DDPG k-NN, which shows the importance of learning extra critic just for training the retrieval actor, unlike what was originally proposed Dulac-Amold et al. (2015).
RecSim:
Continuous Reacher: FLAIR performs comparatively DDPG, DDPG k-NN and DDPG2 k-NN approaches. Despite the non-linear matrix transformation, the resultant action space is still smooth enough for DDPG and variants learn. DDPG k-NN and DDPG2 k-NN, by virtue of sampling 3 nearest neighbor actions to the continuous action, are more likely to optimize the selection Q-function than DDPG. Therefore, they stay above DDPG throughout the training procedure. Similarly, FLAIR and FLAIR w/o cascading learn faster in the initial stages of learning simply by having more candidates to perform directed exploration with. This demonstrates that listwise retrieval agent offers more flexibility than local neighborhood approach for initial exploration by letting the agnet have more diverse action samples. CEM is unable to learn at all
6.2 ANALYSIS: CONSISTENCY ACROSS DIFFERENT COMPLEXITY OF ACTION SPACE
We evaluate proposition against the baselines to examine the consistency of our result across different difficulty in the action space in the discrete RL tasks by changing the action-set sizes and the action-dimension.
Action-set size: Figure (Right) shows the result of FLAIR compared to the baselines the dif- ferent action-set size from the main experiment. Empirically, observed that the action sizes largely influenced the sample efficiency of the learning. here, we show the representative result with the action-set size equal to 5000 in RecSim simulator. Compared the main experiment
8

---

Under review as a conference paper at ICLR 2023
which uses 10000 actions, all agents improved the performance(above the level of 5) expected since the action-set size contributes to the complexity of the action space. Also, we observed that the margin between FLAIR and the baselines of Joint and DDPG2 k-NN became smaller, which indicates that our agent was able to learn efficiently in the larger and more difficult setup.
Action-set dimension: Figure (Left) shows the result of FLAIR compared the baselines in the different action dimension. order to highlight the effect the change in the action dimension, here we used 30 for the action dimension with the action-set size being 10 whereas in the main experiment
4 we used the dimension of and the 100-tools. As we observed in the action-set size experiment above, all agents improved the performance due to the smaller size. The result shows that the non non-listwise method of FLAIR w/o Cascading deteriorated whereas our method was able to improve the performance.
6.3 DESIGN CHOICES
We analyze the design choices of list encoder and weight initialization of cascaded actor-critic net- works in Fig. 6
RecSim
RecSim
Eval Return
Eval Return
4-
4
2
0.0
1.0
1.5
2.0
0.0
1.5
2.0
Env Steps
1e6
Steps
1e6
DeepSet
Transformer
LSTM
Random
Xavier
Gaussian
Xavier
Figure 6: Validating the design choices of our method. (Left) We observe that DeepSet list encoder slightly outperforms Transformer and LSTM variants RecSim. (Right) We observe that having more diverse weight initializations for different cascaded actor and critic networks enables better performance of FLAIR.
6.3.1 CHOICE OF LIST ENCODER
We compare the performance of FLAIR for difference choices of list encoder architecture on RecSim environment. We observe that Deep Set slightly outperforms transformer (Vaswani et al., 2017) and LSTM list encoders. Because of its simplicity, we use deep set architecture for all our experiments.
6.3.2 CHOICE OF WEIGHT INITIALIZATION
We experiment with different ways to initialize the weights of cascaded actor and critic networks to see if having more diverse network initialization can lead to improved learning of listwise RL behavior. We observe that the standard Xavier initialization was outperformed by Gaussian and Xavier+Gauseee weight initialization. We believe the reasoning for this behavior is an encourage- ment of different actors learn act differently and focus more on the currently built list action.
CONCLUSION
We present FLAIR, listwise action retrieval agent for performing flexible reinforcement learning in complex innumerable action spaces. We demonstrate its applicability to improve exploration initially and flexibility, generally, two discrete action space RL environments, RecSim and Mine World, and one continuous action space Reacher environment.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Lukas Biewald. Experiment tracking with weights and biases. Software available from wandh. com, 2:233, 2020.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01544 2016.
Yash Chandak, Georgios Theocharous, James Kostas, Scott Jordan, and Philip Thomas. Learn- ing action representations for reinforcement learning. In International Conference Machine Learning, pp. 941 941- PMLR, 2019.
Haokun Chen, Xinyi Dai, Han Cai, Weinan Zhang, Xuejian Wang. Ruiming Tang, Yuzhou Zhang, and Yong Yu. Large-scale interactive recommendation with tree-structured policy gradient. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3312-3320, 2019a.
Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed H Chi. Top-k off-policy correction for a reinforce recommender system. In Proceedings of the Twelfth ACM International Conference Web Search and Data Mining, pp. 456-464, 2019b.
Xinshi Chen, Shuang Li, Hui Li, Shaohua Jiang, Yuan Qi, and Le Song. Generative adversarial user model for reinforcement learning based recommendation system. In International Conference on Machine Learning, pp. 1052-1061. PMLR, 2019c.
Maxime Chevalier Bhevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. https://githut com/ maxi httmm//axmmeeb//eemom wwwpeeeccccmmmmiiiiiiiiii.. 2018.
Pieter-Tjett De Boer, Dirk P Kroese, Shie Mannor, and Reuven Y Rubinstein. A tutorial the cross-entropy method. Annals of operations research, 134(1):19-67, 2005.
Gabriel Dulac-Amold, Richard Evans, Hado van Hasselt, Peter Sunehag, Timothy Lillicrap, Jonathan Hunt, Timothy Mann, Theophane Weber, Thomas Degris, and Ben Coppin. Deep rein- forcement learning in large discrete action spaces. arXiv preprint arXiv: 1512.07679 2015.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pp. 1861-1870. PMLR, 2018.
Zhiheng Huang, Wei Xu, and Kai Yu. Bidirectional Istm-crf models for sequence tagging, 2015.
Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Mohammadamin Barekatain, Simon Schmitt, and David Silver. Learning and planning in complex action spaces. In International Conference on Machine Learning, pp. 4476 4486. PMLR, 2021.
Eugene Ie, Chih-wei Hsu, Martin Mladenov, Vihan Jain, Sanmit Narvekar, Jing Wang, Rui Wu, and Craig Boutilier. Recsim: A configurable simulation platform for recommender systems. arXiv preprint 1909. 1900:0447970 2019a.
Eugene Ie, Vihan Jain, Jing Wang. Sanmit Narvekar, Ritesh Agarwal, Rui Wu, Heng-Tze Cheng, Tushar Chandra, and Craig Boutilier. Slateq: A tractable decomposition for reinforcement learn- ing with recommendation sets. 2019b.
Ayush Jain, Andrew Szot, and Joseph Lim. Generalization ing. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Confer- ence Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 4661-4672 PMLR, 13-18 Jul 2020. URL http::/proceedii http://.iggg.......s//..//. jain20b. html
actions in reinforcement learn-
mlr ptwss/v11s//////
Ayush Jain, Norio Kosaka, Kyung-Min Kim, and Joseph Lim. Know your action set: Learning action relations for reinforcement learning. In International Conference on Learning Representa- fions, 2021.
10

---

Under review as a conference paper at ICLR 2023
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforce- ment learning for vision-basee robotic manipulation. In Conference on Robot Learning, pp. 651- 673. PMLR, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1eXiv:1412.6980 6980, 2014.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:/509.000 02971, 2015.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, nature, 51 5187754)::29-533 2015.
Karthik Narasimhan, Tejas Kulkarni, and Regina Barzilay. Language understanding for text-based games using deep reinforcement learning. arXiv preprint /rXiv:/506.08800 2015.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.063477 2017.
Peter Sunehag, Richard Evans, Gabriel Dulac-Arnold, Yori Zwols, Daniel Visentin, and Ben Cop- pin. Deep reinforeement learning with attention for slate markov decision processes with high- dimensional states and actions. arXiv preprint arXiv: 2015.
Tian Tan, Feng Bao, Yue Deng, Alex Jin, Qionghai Dai, and Jie Wang. Cooperative deep reinforce- ment learning for large-scale traffic grid signal control. IEEE transactions on cybernetics, 50(6): 2687-2700, 2019.
Guy Tennenholtz and Shie Mannor. The natural language of actions. arXivv1902.01119, 2019.
arXiv preprint
Chen Tessler, Tom Zahavy, Deborah Cohen, Daniel Mankowitz, and Shic Mannor. Action as- sembly: Sparse imitation learning for text based games with combinatorial action spaces. arXiv preprint Xrv: 000.//90500000.. 2019.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Tom Van de Wiele, David Warde-Farley, Andriy Mnih, and Volodymyr Mnih. Q-learning in enor- mous action spaces via amortized approximate maximization. arXiv preprint 26X1v:2001.01116, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz. Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3):279-29) 1992.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
11

---

Under review as a conference paper at ICLR 2023
APPENDIX
A ENVIRONMENT DETAILS
A.1 MININGENV
The grid world environment, introduced in Sec. 5.1 requires an agent to reach a goal by navigating a 2D maze as soon as possible while breaking the mines blocking the way.
The state space is an 8x8-grid world, and each cell has three dimensions: (cell-type, mine- type-number, direction-numan number). When the corresponding type matched, the second/third dimen- sion would be meaningful. Otherwise, they would be set to be the default value as 0. To be more specific, there are four kinds of cells: empty, mine, goal, and agent:
State:
1. Empty: each dimension equals 0
2. Mine: The cell-type dimension equals 1, the sein-tyye-nummee dimension equals the mine
type id and the it is divided equally into two parts: basic mines and complex mines. There is no significant difference between basic and complex mines except mine id. More specifically, the basic mine id range is [0, ..., 9], and the complex mine id range is [10, 19).
dimension equals 0. The mine set size is equal to 20 and
3. Goal: The cell-typ type dimension equals 2, and all the others equal O.
4. Agent: The cell-type dimension equals 3, the mine-type-numble dimension equals 0, and the dtrectionnnumbll dimension equals to the corresponding direction number. There will only be 4 kinds of directions: facing up, down, left, right.
Ultimately, we will normalize each dimension to [0, 1] with each dimension's 's hkinimum/maxiinee value. And each time when resetting the environment, the category of each mine will be randomly changed.
Termination: An episode is terminated in success when the agent reaches the goal or after a total of 100 timesteps.
Actions: The base action set combines two kinds of actions: navigation actions and pickoaxeetool) actions.
The navigation action set is a fixed set, which contains four independent actions: going up, down, left, and right, corresponding with the direction of the agent. They will change the agent's 'sss direction first and then try to make the agent take one step forward. Note that, different from the empty cell, the agent cannot step onto the mine, which means that the agent is trying to take step towards a mine or the border of the world, then the agent will stay in the same location while the direction will still be changed. Otherwise, the agent can step onto that cell. An agent will succeed if it reaches the goal position.
The size of the pick-axe action set is equal to 100. Each tool has a one-to-one mapping, which means they can and only can be successfully applied to one kind of mine, and either transform that kind of mine into another type of mine or directly break it.
To state it more clearly, the toolset consists of basic and complex tools. Basic tools can only be applied to the basic mines, and will directly break the mine into empty cell. As for the complex tools, they could only be applied the complex mines, and some complex tools can turn the mines into one kind of basic mine. Others can directly break the mines.
Note, basic tools can't turn the basic mines into any other kind of mines, and also impossible for complex tools to turn the complex mines into another kind of complex mines. In addition, it is guaranteed that all mines can be broken, with one or two tools sequentially.
The agent receives large goal reward for reaching the goal. The goal reward is discounted based on the number of action steps taken to reach that location, thus rewarding shorter paths. To further encourage the agent to reach the goal, small exploration reward is added whenever the agent gets closer to the goal, and a negative equal penalty is added whenever the agent gets further to the goal. And also, when the agent successfully applies tool, it will gain small reward. When
Reward:
12

---

Under review as a conference paper at ICLR 2023
Agent
Empty
Basic Mines
Complex Mines
Empty (No mine)
Aation Type
Rovigation Number
Mireell Behore
Mine<ft After
110, 11...
Basic Tools x30
Mine
B1
Complex Tools x110
B2
Goal
1
Navigation actions
18.9]
x4
Mining World Overview
Action Overview
Figure 7: Mining Env Setting Description
the agent successfully breaks a mine, it will also gain small bonus.
Neurrent steps
R(s,a) Goat Raoal
1 AGoal
Nmax steps
Retep (Ddistance before
Ddistance after)
1 correct tool applied Bioel +
1 success fully break mine
RiBomus
where RGval 10, Rstep = 0.1, Riwl = 0.1, R = 0.1, Aaoa = 0.9, Nmux steps 100
4-dimensional vectors manually defined using a mix of number ids, and each dim scaled to [0, 1]. as shown in Graph 7 Dimensions identify the category of skills (navigation, pick-axe), 2 distinguish movement skills (right, down, left, up), 3 is used to denote the mine on which this tool can be successfully applied, and are showing the result of applying this tool. We will normalize the action embedding space to [0, for each dimension.
Action Representations: The action representations
B NETWORK ARCHITECTURES
B. CASCADED ACTOR
The whole actor has a cascaded format and each cascaded actor will receive two pieces of informa- tion: the state observation and the action list generated by previous cascaded actors. For one single cascaded actor, given the concatenation of the input components above, 4-layer MLP with ReLU will process this information and generate one action. And this action will be concatenated with the previous action list. After transformed by optional aetion-list-encodee together with the state information, they become the input of next cascaded actor's input. In the end, the action list will be processed with I-NN to find the nearest discrete action. After this, this action list will be delivered to the selection Q-network.
B.2 CASCADED CRITIC
The critic has one-to-one mapping relationship the actor. The whole critic consists of list of cas- caded critics and each cascaded critic will receive three pieces of information: the state observation, the action list generated by previous cascaded actors, and the action provided by the corresponding cascaded actor. For single cascaded actor, given the concatenation of the input components above, an 2-layer MLP with ReLU will process this information and generate the action's 's value. This value will be used update itself well as the actor with TD-error.
13

---

Under review as a conference paper at ICLR 2023
B.3 LIST ENCODERS
List Encoder: An intermediate list that contains the currently selected actions in between each cas- caded actors is passed on encoder summarize the list-action information.
The raw action representations of candidate actions are passed on to the 2-layer ML. fol- lowed by ReLU. Then, the output of the MLP is processed by a 2-layer bidirectional LSTM (Huang et al. 2015). Another 2-layer MLP follows this to create the action set summary to be used in the following cascaded actor.
Bi-LSTM:
DeepSet: Similar to the Bi-LSTM variant of the summarizer, we employed the 2-layer MLP with ReLU followed by the mean pooling over all the candidate actions to compress the information. Fi- nally, the 2-layer MLP with ReLU provides the resultant action summary to the following cascaded actor.
Transformer: Similar to the Bi-LSTM variant of the summarizer, we employed the 2-layer MLP with ReLU before inputting the candidate actions into a self-attention and feed-forward network to summarize the information. Afterwards the summerization will be the part of the input of the following cascaded actor.
B.4 SELECTION Q-NETWORK
As for selection Q-network, it will received concatenated information of state and one action evaluate the q-value. By comparing the the q-value of each action, the highest q-value action will be chosen the final output action.
C EXPERIMENT DETAILS
C.I IMPLEMENTATION DETAILS
We used Py Torch (Paske et al. 2019) for our implementation, and the experiments were primarily conducted on workstations with either NVIDIA GeForee RTX 2080 Ti, P40, or V32 GPUs on. Each experiment seed takes about 4 hours for Mine World, 8 hours for Reacher, and 6 hours for RecSim, to converge. We use the Weights Biases tool (Biewald, 2020) for plotting and logging experiments. All the environments were developed using the OpenAl Gym Wrapper (Brockman et al. 2016). We use the Adam optimizer (Kingma & Ba 2014) throughout.
C.2 HYPERPARAMETERS
The environment-specific and RL algorithm hyperparameters are described in Table !
C.3 HYPERPARAMETER TUNING
We found methods to be sensitive to the following hyperparameters and sweeped the sociated values: actor-Ir (0.01, 0.001, 0.0001), critic-Ir (0.01, 0.001, 0.0001 actor decay { [100K, 200K, 500K, IM, 2M, 3M J. critic decay { [100K, 200K, 500K, 1M, 2M, 3M). Different environment settings need different combination of the hyper parameters.
14

---

Under review as a conference paper at ICLR 2023
Hyperparameter
Mine World
Reacher
RecSim
Environment
max episode steps number of epochs
100
250
1500/2500
8000
2000
number of environments action dim
5
10
30/4
30/60
number of all actions obs encoder applied state(obs) dim
14/104
10000
True
False 11
False
3
30/60
RL Training
0.001
0.001
0.001
WOLP actor Ir
0,001
0.001
0,0001
WOLP critic Ir
0.001
0.001
0.001
Q-network hidden dimension
128 128
400_300
64_32
candidate list len
4
batch size
256
256
256
buffer size
500000
1000000
1000000
Table 1: EanirommentPoliyysseefffee Hyperparameters
15


Reviews:
[{"review_id": "w4dUvFD8Iuz", "reviewer": "Reviewer_CZhi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors explore a listwise action space RL approach to tackle situations with enormous, discrete action spaces or complex continuous action spaces.  Complex continuous action spaces are defined as those which have high variability in the action dimensions or have a convoluted action space consisting of a large number of features with relatively little power.\n\nThe approach utilizes stacked actors and stacked critics.  The stacked actors produce a list of actions.  The stacked critics utilize the outputs of all preceding actors when creating a training signal for the actors.\n\nThe list of actions created by the actors are candidates evaluated by a selection network.  The action selected by the selection network is the selected action for the ensemble.", "summary_of_the_review": "An interesting approach which addresses a real-world problem with RL (although somewhat abstract).  Missing some analysis that would help justify the need for this approach.", "strength_and_weaknesses": "I think the stacked actors is a great way to produce a list of candidate actions and selecting a winner using a final Q-function selection network is a solid approach.  The paper does a good job of justifying this approach.\n\nThe graphic in Figure 2 does a good job of showing the flow of the approach.\n\nI was surprised that the upstream actor outputs are fed into the critic along with the corresponding actor.  It would have been nice to see some justification of this design decision: is this needed and does it help?  Ensembles of critics (or Q-functions) can be pretty diverse and unstable and, I'd think, reliable in occupying different parts of solution space to produce diverse learning signals.  I also didn't understand how these other actions affect the critic output or how this is driving the ensemble to have more diversity (assuming this is the purpose).\n\nIn my view, the biggest weakness is that this work left one assumption unstated and unexamined.  This assumes that the selection network has the ability to represent the potentially volatile changes in Q-value for nearby action for a given state.  Is there any work showing that the function modeled by an actor is more smooth than the function represented by the critic?  This work didn't provide any analysis that the selection network was capable of representing substantial changes in Q-value from nearby actions.  As I understand this work, this analysis would significantly increase the contribution to the literature.\n\nI recommend looking at Gradient Boosting in crowd ensembles for Q-learning... by Elliott et al or \"The wisdom of the crowd: reliable deep reinforcement learning through ensembles of Q-functions\" in TNNLS journal.  I think the work proposed in this paper is similar would work in areas where this work may not.  May be a nice item of related work.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "tbC-RGzpBfr", "reviewer": "Reviewer_ieEx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method to solve large action space control tasks via a two step RL procedure. The authors first perform action retrieval over a fixed space of action representations and then perform a selection procedure over the retrieved actions via a trained value-function. The proposed method is evaluated over a benchmark of 3 discrete and continuous tasks, including a simulator of a recommender system.", "summary_of_the_review": "This paper clearly is not ready for publication, especially in a venue like ICLR. \nFirst the method in provides limited novelty, as it is a modification of a previously published algorithm, with CDQN (cascaded DQN) by using DDPG as base learner instead of DQN. While this would be fine if the method clearly improved performance over the original algorithm, or provided some theoretical/intuition over the importance of this modification, the authors fail to do any of that. Moreover, The experimental evaluation also is extremely lacking as discussed in the section on Weaknesses.", "strength_and_weaknesses": "Weaknesses:\n\nAs i understood, the paper proposes FLAIR, according to the authors a novel method for performing action retrieval in complex action spaces. Indeed the authors claim in their main contribution is introducing the problem of complex innumerable action spaces in reinforcement learning. Then they solve this task by proposing FLAIR. This claim is clearly not true, because right after making the claim of introducing this setting, the authors continue with a related work section discussing other methods that solve this task, most notably Cascaded DQN (CDQN) the base method from which FLAIR is derived.\n\nThe authors makes unsupported claims throughout the  paper like:\n\"FLAIR make robust actions retrieval\" - Robust in what sense? is it guaranteed to include always the optimal action? Is it guaranteed to retrieve in high probability? Is robust is defined in the paper?\n\"FLAIR enables directed exploration of the task\" - How does it do this? Directed exploration has a clear definition in RL and an immense body of work both in discrete and in continuous action spaces. The authors repeat this claim several time in the paper but never offer any intuition as to why, or any empirical evidence in support of this.\n\n-- Reproducibility is a main issue of this paper---\nThe paper does not contain a single equation. No loss functions are defined, for none of the K critics or K actors described in the methodology section. How is a reader supposed to reproduce the presented results when no detail on even the loss functions is given?\n\nNovelty is also a weakness. In my opinion the paper is just a modification of Cascaded DQN with DDPG as the learning algorithm. I cannot even completely verify this, as no pseudocode of the method was provide, no loss functions were defined and no source code was given.\nNow this would be fine, if clear empirical performance difference were observed and explained between the two versions, or if some intuition was given as to why DQN is not suitable, except for the fact that it cannot be applied to continuous actions. \n\nThe empirical evaluation is also extremely lacking. The method makes not technical or theoretical contributions, so the bar for empirical contributions is increased. The authors use 3 benchmarks, 2 with discrete actions and one continuous actions. The choice of baselines to compare is also questionable. The authors modified CDQN to recover FLAIR and make the choice not to compare the two algorithms, so we cannot even verify if replacing DQN with DDQN has any empirical effect. Moreover, the authors include the performance of DQN as an oracle, claiming that their method couldn't possibly reach the performance of DQN since they do not have the full action space and can only pick a subset of the actions. And this is clearly the main issue with the experimental evaluation. The whole point of the setting is that methods that directly work in discrete actions spaces like DQN would fail here because the number of actions is so large. DQN does have access to the complete action space, but the authors should use benchmarks where this hinders learning. Showing benchmarks where DQN already achieves good performance clearly does not make FLAIR or any action retrieval method for RL needed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "kY37_FLzkP", "reviewer": "Reviewer_GAf6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a new method for complex and large action space. The proposed method uses listwise RL for the retrieval task which is further used for action selection. ", "summary_of_the_review": "The paper has several weaknesses and it is not ready for publication. See weaknesses for justification.", "strength_and_weaknesses": "Strengths:\n- The paper motivates the problem very well. The abstract and introduction are highly aligned with the problem targeted. \n- The paper uses three simulation environments for training and evaluation of the proposed method and several baselines are used for performance comparison.  \n\nWeakness: \nThere are several weakness on writing, explanation, and evaluation. The justification is below: \n1. Results: \n- The paper motivates the problem very well but fails to justify using proper experiments. For example, in section 6.1.2, authors claim that their proposed is better than all baselines. However, it is not seen in neither Figure 4 nor Figure 5 and DQN (oracle) seems to be performing best among all. The y-axis of both the figures is eval success rate or eval return means higher the AUC better the algorithm. This is not seen in any of the figure. \n- The results are not extensive and hence it is difficult to conclude that the method is better. Only Figure 4 & 5 are given as performance comparison and no quantitative results are available. Figure 6 is for the representation learning and doesn't directly come under contributions. \n- The retrieval selection framework is not very common so author should justify all steps of how retrieval works in their method. I couldn't find sufficient information on the same. \n\n2. Method: \n- It's difficult to find why the method would work best. The approach is explained in sec 4 which is not sufficient. For example, why would you be using cascaded actor-critic as compared to normal actor-critic. Not clear at all, why the design choices are used?\n- The authors have cited several papers on which the proposed methods is based (for example, Deep Set, Cascaded DQN, listwise DQN, etc.). These citations make the paper highly abstract and not sufficient for the reader to understand the proposal. I would suggest to make the proposed method explanation in the main text itself in Sec 4. \n\n3. Writing: \n- The title is to hedge your actions. I have not found any useful information where hedging of actions is justified. \n- Sec 3.2, framework 1 should be Figure 1. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8zDOJFhC9bl", "reviewer": "Reviewer_oShM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method to deal with complex ('innumerable') action spaces in RL. The method is based on one from the recommender system literature, cascaded DQN, where a list of action candidates is produced by an internal actor trained through RL, from which the final one is selected. The proposed method is tested on a few different domains, where it is shown to outperform all chosen baselines except a DQN, which is considered the 'oracle'.", "summary_of_the_review": "The paper presents an interesting idea, explained clearly and placed well in the context of the existing literature. The experimental evaluation is strong on some points (baselines, ablations), but does not fully manage to convince due to the relatively short nature of the training runs. Furthermore the gravity of the problem (and hence the necessity of the proposed solution) is not supported numerically. Hence, the paper leaves open some important questions. If those could be addressed, I would increase my score.", "strength_and_weaknesses": "- Strength: the proposed method looks like it has wide applicability.\n- Strength: the paper contains extensive baselines and ablations.\n- Strength: the paper addresses a relevant problem.\n- Weakness: the learning curves shown in figures 4 and 5 don’t seem to have converged, and don’t seem overly long (a few million environment steps), leaving open the question of longer-term performance.\n- Weakness: There are no experiments where the ‘oracle’ DQN is not a feasible solution (it is presented for all experiments, and outperforms all other baselines and the proposed method). There is also no analysis to show where that would happen, leaving open the question how necessary the proposed method is. More discussion of required network sizes and sample efficiency (and hence possibly also compute complexity) would be welcome.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject