Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors explore a listwise action space RL approach to tackle situations with enormous, discrete action spaces or complex continuous action spaces.  Complex continuous action spaces are defined as those which have high variability in the action dimensions or have a convoluted action space consisting of a large number of features with relatively little power.

The approach utilizes stacked actors and stacked critics.  The stacked actors produce a list of actions.  The stacked critics utilize the outputs of all preceding actors when creating a training signal for the actors.

The list of actions created by the actors are candidates evaluated by a selection network.  The action selected by the selection network is the selected action for the ensemble.
strength_and_weaknesses: I think the stacked actors is a great way to produce a list of candidate actions and selecting a winner using a final Q-function selection network is a solid approach.  The paper does a good job of justifying this approach.

The graphic in Figure 2 does a good job of showing the flow of the approach.

I was surprised that the upstream actor outputs are fed into the critic along with the corresponding actor.  It would have been nice to see some justification of this design decision: is this needed and does it help?  Ensembles of critics (or Q-functions) can be pretty diverse and unstable and, I'd think, reliable in occupying different parts of solution space to produce diverse learning signals.  I also didn't understand how these other actions affect the critic output or how this is driving the ensemble to have more diversity (assuming this is the purpose).

In my view, the biggest weakness is that this work left one assumption unstated and unexamined.  This assumes that the selection network has the ability to represent the potentially volatile changes in Q-value for nearby action for a given state.  Is there any work showing that the function modeled by an actor is more smooth than the function represented by the critic?  This work didn't provide any analysis that the selection network was capable of representing substantial changes in Q-value from nearby actions.  As I understand this work, this analysis would significantly increase the contribution to the literature.

I recommend looking at Gradient Boosting in crowd ensembles for Q-learning... by Elliott et al or "The wisdom of the crowd: reliable deep reinforcement learning through ensembles of Q-functions" in TNNLS journal.  I think the work proposed in this paper is similar would work in areas where this work may not.  May be a nice item of related work.
clarity,_quality,_novelty_and_reproducibility: The approach is novel and is intriguing.  The work appears reproducible and uses accessible test domains.  The approach is explained well.  The authors include implementation level details.
summary_of_the_review: An interesting approach which addresses a real-world problem with RL (although somewhat abstract).  Missing some analysis that would help justify the need for this approach.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method to deal with complex ('innumerable') action spaces in RL. The method is based on one from the recommender system literature, cascaded DQN, where a list of action candidates is produced by an internal actor trained through RL, from which the final one is selected. The proposed method is tested on a few different domains, where it is shown to outperform all chosen baselines except a DQN, which is considered the 'oracle'.
strength_and_weaknesses: - Strength: the proposed method looks like it has wide applicability.
- Strength: the paper contains extensive baselines and ablations.
- Strength: the paper addresses a relevant problem.
- Weakness: the learning curves shown in figures 4 and 5 don’t seem to have converged, and don’t seem overly long (a few million environment steps), leaving open the question of longer-term performance.
- Weakness: There are no experiments where the ‘oracle’ DQN is not a feasible solution (it is presented for all experiments, and outperforms all other baselines and the proposed method). There is also no analysis to show where that would happen, leaving open the question how necessary the proposed method is. More discussion of required network sizes and sample efficiency (and hence possibly also compute complexity) would be welcome.

clarity,_quality,_novelty_and_reproducibility: - Clarity: The paper is generally well written and easy to read. There are some aspects where it could be improved:
  - ‘Hedge’ is only mentioned three times, two of which are in the title and abstract, where it is not explained; the third time, in the introduction, it appears to be a synonym for diversifying. Please consider using a different title, or strengthening the connection to the text. The main topic of the paper doesn't seem to be diversity/hedging, but rather dealing with innumerable action spaces in RL, so this title might not be ideal.
  - The second sentence of the abstract is not clear to me, possibly because I am not sufficiently familiar with the recommender systems literature. However, presuming that I am part of the target audience, please clarify why the representations of the items in a recommender system don’t apply to all users in the same way.
  - The acronym ARDDPG is not introduced, please fix (is it Action Retrieval DDPG?).
  - The caption of figure 5 does not make clear how the experiments presented there differ from those in figure 4 (action set size
- Quality: See strengths and weaknesses.
  - Writing quality: please language edit sections 2.1-2.3
- Novelty: the paper extends the explicit treatment of complex action spaces through cascading DQN from Chen et al. to continuous action spaces, and makes the case for introducing the list-wise RL approach to the wider RL literature.
- Reproducibility: no concerns.
summary_of_the_review: The paper presents an interesting idea, explained clearly and placed well in the context of the existing literature. The experimental evaluation is strong on some points (baselines, ablations), but does not fully manage to convince due to the relatively short nature of the training runs. Furthermore the gravity of the problem (and hence the necessity of the proposed solution) is not supported numerically. Hence, the paper leaves open some important questions. If those could be addressed, I would increase my score.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposed a new method for complex and large action space. The proposed method uses listwise RL for the retrieval task which is further used for action selection. 
strength_and_weaknesses: Strengths:
- The paper motivates the problem very well. The abstract and introduction are highly aligned with the problem targeted. 
- The paper uses three simulation environments for training and evaluation of the proposed method and several baselines are used for performance comparison.  

Weakness: 
There are several weakness on writing, explanation, and evaluation. The justification is below: 
1. Results: 
- The paper motivates the problem very well but fails to justify using proper experiments. For example, in section 6.1.2, authors claim that their proposed is better than all baselines. However, it is not seen in neither Figure 4 nor Figure 5 and DQN (oracle) seems to be performing best among all. The y-axis of both the figures is eval success rate or eval return means higher the AUC better the algorithm. This is not seen in any of the figure. 
- The results are not extensive and hence it is difficult to conclude that the method is better. Only Figure 4 & 5 are given as performance comparison and no quantitative results are available. Figure 6 is for the representation learning and doesn't directly come under contributions. 
- The retrieval selection framework is not very common so author should justify all steps of how retrieval works in their method. I couldn't find sufficient information on the same. 

2. Method: 
- It's difficult to find why the method would work best. The approach is explained in sec 4 which is not sufficient. For example, why would you be using cascaded actor-critic as compared to normal actor-critic. Not clear at all, why the design choices are used?
- The authors have cited several papers on which the proposed methods is based (for example, Deep Set, Cascaded DQN, listwise DQN, etc.). These citations make the paper highly abstract and not sufficient for the reader to understand the proposal. I would suggest to make the proposed method explanation in the main text itself in Sec 4. 

3. Writing: 
- The title is to hedge your actions. I have not found any useful information where hedging of actions is justified. 
- Sec 3.2, framework 1 should be Figure 1. 
clarity,_quality,_novelty_and_reproducibility: Clarity: 
- The paper is clearly written and motivates the problem very well. 

Quality:
- The paper lacks proper explanation of the method and the evaluation is also not sufficient. 

Novelty:
- The proposed method is novel to the extend of a new method in retrieving actions. The method is original to some extent. 

Reproducibility: 
- It's difficult to see if the results are reproducible as sufficient information is not provided. Appendix has implementation details but the method is not clear in the main text.
summary_of_the_review: The paper has several weaknesses and it is not ready for publication. See weaknesses for justification.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a method to solve large action space control tasks via a two step RL procedure. The authors first perform action retrieval over a fixed space of action representations and then perform a selection procedure over the retrieved actions via a trained value-function. The proposed method is evaluated over a benchmark of 3 discrete and continuous tasks, including a simulator of a recommender system.
strength_and_weaknesses: Weaknesses:

As i understood, the paper proposes FLAIR, according to the authors a novel method for performing action retrieval in complex action spaces. Indeed the authors claim in their main contribution is introducing the problem of complex innumerable action spaces in reinforcement learning. Then they solve this task by proposing FLAIR. This claim is clearly not true, because right after making the claim of introducing this setting, the authors continue with a related work section discussing other methods that solve this task, most notably Cascaded DQN (CDQN) the base method from which FLAIR is derived.

The authors makes unsupported claims throughout the  paper like:
"FLAIR make robust actions retrieval" - Robust in what sense? is it guaranteed to include always the optimal action? Is it guaranteed to retrieve in high probability? Is robust is defined in the paper?
"FLAIR enables directed exploration of the task" - How does it do this? Directed exploration has a clear definition in RL and an immense body of work both in discrete and in continuous action spaces. The authors repeat this claim several time in the paper but never offer any intuition as to why, or any empirical evidence in support of this.

-- Reproducibility is a main issue of this paper---
The paper does not contain a single equation. No loss functions are defined, for none of the K critics or K actors described in the methodology section. How is a reader supposed to reproduce the presented results when no detail on even the loss functions is given?

Novelty is also a weakness. In my opinion the paper is just a modification of Cascaded DQN with DDPG as the learning algorithm. I cannot even completely verify this, as no pseudocode of the method was provide, no loss functions were defined and no source code was given.
Now this would be fine, if clear empirical performance difference were observed and explained between the two versions, or if some intuition was given as to why DQN is not suitable, except for the fact that it cannot be applied to continuous actions. 

The empirical evaluation is also extremely lacking. The method makes not technical or theoretical contributions, so the bar for empirical contributions is increased. The authors use 3 benchmarks, 2 with discrete actions and one continuous actions. The choice of baselines to compare is also questionable. The authors modified CDQN to recover FLAIR and make the choice not to compare the two algorithms, so we cannot even verify if replacing DQN with DDQN has any empirical effect. Moreover, the authors include the performance of DQN as an oracle, claiming that their method couldn't possibly reach the performance of DQN since they do not have the full action space and can only pick a subset of the actions. And this is clearly the main issue with the experimental evaluation. The whole point of the setting is that methods that directly work in discrete actions spaces like DQN would fail here because the number of actions is so large. DQN does have access to the complete action space, but the authors should use benchmarks where this hinders learning. Showing benchmarks where DQN already achieves good performance clearly does not make FLAIR or any action retrieval method for RL needed.
clarity,_quality,_novelty_and_reproducibility: The paper has extremely limited novelty, it's fairly clearly written but completely lacks reproducibility. There is not a single Equation in this paper, not a single loss function defined or analyzed. 
summary_of_the_review: This paper clearly is not ready for publication, especially in a venue like ICLR. 
First the method in provides limited novelty, as it is a modification of a previously published algorithm, with CDQN (cascaded DQN) by using DDPG as base learner instead of DQN. While this would be fine if the method clearly improved performance over the original algorithm, or provided some theoretical/intuition over the importance of this modification, the authors fail to do any of that. Moreover, The experimental evaluation also is extremely lacking as discussed in the section on Weaknesses.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:

This paper presents a method of using complex innumerable action spaces in RL, by cascading DDPG learners, one per dimension.  The work is presented in contrast to standard DQN approaches which struggle with large finite action spaces, or conventional policy gradient methods that rely on smoothness in the joint action space.  The paper argues that these conditions do not hold in some less conventional RL problems (such as providing a slate of choices in a recommender system, or unfavorable formulations of Mujoco control problems).  The paper shows the proposed method does have traction on these unconventional problems.

Strengths:

The method of cascading DDPG learners appears new, and the experimental results are promising on the chosen problems.  The overall approach is simple enough to understand so that others can build upon it.  The reviewers also appreciated that the paper tackled complex action spaces in RL.

Weaknesses:

Major concerns were around the main claims of the paper and the lack of care around words that have a common technical meaning in the RL community.  Additional concerns were raised by around the lack of care in the choice and description of experiments.

The paper's main claim was of contributing complex innumerable action spaces ("Our primary contribution is introducing the problem of complex innumerable action spaces in reinforcement learning"). However, these terms were not formally defined.  Moreover using the informal problem description in the text, this is also an over-claim since prior work (not discussed in the review period) also tackles what seems to be on the surface as complex innumerable action spaces (e.g. Metz et al (2017) https://arxiv.org/abs/1705.05035).  That work went in a different direction, by discretizing continuous spaces with a sequential approach.  It would be good to have a technical definition to know what is claimed or not claimed about when the method presented in this paper is expected to work.

Another concern was on the misuse of technical terms (oracle when the method is intended to outperform the oracle, the use of exploration and robustness).  Another concern was the choice of experimental domains where all actions can be enumerated (so DQN still works adequately, so the experiment does not demonstrate the intended problem setting).  

Finally, the experiment description in the original paper was found to be lacking.  Reviewers felt they had inadequate insight into when the proposed method is expected to perform well or poorly.  The experiments also raise concerns. The use of 3 seeds is poor statistical practice and computing a variance over 3 seeds is troubling.  There are no error bars on the reacher example.  

justification_for_why_not_higher_score: After discussion with the reviewers, there was a consensus that this paper is not ready for publication.  The reviewers did feel the approach has promise, but a better description of the problem setting and more careful experiments are desired.
Decision: Reject