Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a novel framework for learning action metrics from offline data called Behavioral Metric of Actions (BMA).  The goal of this metric is to position the actions in a latent space that is pertinent relative to the task being learned.  This is particularly useful in environments with a large number of actions where exhaustive exploration is unfeasible, and therefore generalization over actions is necessary.  BMA has two design principles: behavioral similarity between actions, and data-distributional.  Behavioral reflects the fact that two actions that have a similar induced transition with a similar reward should be considered relatively equivalent.  The data-distributional relationship separates actions that appear within the offline dataset from ones that are absent (or at least absent for a particular state).  The authors show that by learning an embedding with these principles they are able to train off-the-shelf algorithms on offline RL tasks in cases where learning would be impossible without an action representation.  They also show that compared to other action representations their approach performs better.
main_review: ## Goals
This paper sets out to define a new action metric for large action spaces which aids in learning on both on-line and off-line scenarios.  They provide a reasonable approach that is demonstrated to perform empirically.

## Strengths
* Generalization over states has been one of the main powerful aspects of function-approximated RL, however as actions are often without any direct feature representation they have often been treated as a an arbitrary discrete set.  Providing representations over actions is key to scaling RL up to large systems, and is therefore an important problem to tackle.

* The proposed approach makes good intuitive sense, and the conclusion that the embedding line up to correlate with the value function is an important one, as this has been my personal intuition for a while: as far as the policy is concerned with optimizing the return, two actions are similar iff they have the same expected value.

* The ablations are clear, especially around the value of p which is nice to see.

* The embedding visualizations are nice on the maze, however it's a bit surprising that the embedding from Chanak don't look better as they have figures in their paper which seem more well-distributed for a maze environment.

* Overall the bounds make sense and are intuitive, and looking over the proof the general idea seems good, I am however not the right person to give them a true seal of approval.

## Weaknesses
The paper has a couple important issues, more in the form than the function:
* Many figures are small, with very small text for the legends, pleas try and make the text bigger and consume more surrounding whitespace with your figures as much as possible.

* There is a lot of space spent on presenting the problem, not leaving enough for clear experimental results and discussion.  Perhaps it's because I'm already convinced of the importance of this problem, but I think for example that Figure 2. doesn't bring much to the table and actually confused me more than anything at first.  

* Implementation details are sorely lacking: what is your nearest-neighbour lookup? How do do you actually set up your embedding network? I'm not entirely sure I'd be able to replicate this, please provide more concrete information on the actual configuration of the network and any tricks to get the loss to actually converge.  For example in Algorithm 2 in the appendix you mention estimating the transition distributions, but then you train them afterwards, I'm sure this makes sense somehow but it's not quite clear to me and I would appreciate it being clearer :)


## Paper Readability
Overall the approach is legible but there are some concerns that I would like addressed:
* There are no descriptions for Figure 5 and Figure 6, please provide a descriptive legend.
* All figures have very small text and have a lot of surrounding whitespace, they are overall hard to read and I had to zoom in a lot to try and understand them.
* Overall I would suggest perhaps restructuring Sec. 4.2 some to be easier to follow.
* One small NIT is that you talk about data-distribution separability, but there is really only one distribution (the expert's), and then you talk about OOD detection.  I believe what you're trying to do is OOD detection through some mechanism, whereas distrubition separability seems like there is perhaps two sources of training data.  Is there a reason not to just call this OOD detection throughout? This would reduce some confusion I had. 
* For the OOD detection can you elaborate in just one sentence what Fujimoto et al.'s approcah is for I_B?  I didn't have time to go through that paper to see what they did and a one-liner would have made things clearer.

Thanks again for this work and hopefully we can clear up some of the form-related nits!

summary_of_the_review: I believe this paper should be accepted on the merit of having:
* A pertinent problem: action embeddings for large action spaces.
* A well thought-out approach to tackle this problem: Using a set of reasonable priors on action similarity to provide a general embedding for actions given an existing dataset of behavior on the environment.
* Convincing experimental results: Both in the on-line and off-line case the approach works well, and is shown to work better than other action embedding schemes.

However it has issues in its form:
* Clearly missing description on Figures 5 and 6.
* Hard-to-read figures for results.
* Not always easy to follow explanation of the method.

If the authors can manage to clear up at least most of the form issues I will support strong acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the action representation learning for offline RL with large discrete action spaces. It proposes a new metric to capture the similarity between actions, called Behavioral Metrics of Actions. The new metric has two components, one is a behavioral metric that groups actions that share the similar MDP property (i.e., reward and transition model), and the other offline specific one is distributional metric, which uses a binary variable to capture whether the two actions coming from the same data distribution. Empirical studies show superior performance over the baselines, and the ablation study demonstrates the necessity of all of the components in the metric.

RL with large discrete action spaces is of interest in some real-world applications, such as recommender systems and dialogue systems, this paper gives a way to do the action abstraction to further reduce the action space and hence increase the effective sample size in any downstream learning task.
main_review: Strength:
- I like the setting this paper studies, how to learn efficient action representation, which seems a necessary task when we face large discrete action spaces, which is also the case for some real-world applications. 
- The paper is well-written and it is easy to follow.
- The experiments are well done. I do like the ablation part, which demonstrates the effectiveness and necessarity of different components.

Weakness:
- The novelty of this paper seems very limited. For the behavior metric, it just seems like doing some bisimulation metric under the action space, while there are already tons of literature studies on this over the state space. This is a simple and trivial extension to the action space.
- Regarding the behavior metric, it would be great if you can also the effectiveness in the online setting since this is not a component specific to offline RL.
- Specific to offline RL, it adds a distributional penalty. Though some theoretical results are given, I do not gain much insight on it regarding how this distributional penalty helps there. Theorem 1 and 2 seem also to hold without this distributional term in the metric. How this helps theoretically, compared with not adding it. 
- Also for Figure 7, is there any explanation that the metric will give more compact clustering than the reconstruction-based method, and why is this the case?


summary_of_the_review: This paper is well-written and the experiments are well-done to illustrate the effectiveness of the proposed metric. However, the novelty of this paper is pretty low, as a bisimulation type metric is well known. Though adding a new distributional component in the offline setting helps, it does not well-justified at all. Given this, I recommend a weak reject for the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a framework for learning Behavioral Metrics of Actions that combines both behavioral relation and data-distribution relation between actions to help the learning of offline RL algorithms on large discrete-action tasks. 

main_review: Strength:

I think the paper is well-written and easy to follow, the motivation of learning a representation specifically for offline RL learning is very clear. The empirical evaluations show good improvement of the proposed method than other existing ones. 

Weakness:

I have the following concerns/questions
1. Since [1] is also working on using pseudometric for representation as mentioned in Section 2, it would be a more proper baseline to compare with. Besides, I'm particularly interested in this comparison because I think one of the difference between [1] and BMA is that the distance defined in BMA is conditional on the states, while the distance in [1] is defined jointly by states and actions. There has been discussions about the conditional distribution and joint distribution in other areas such as imitation learning [2], so I'm interested to see the comparisons. 
2. The hyparameters of the alternatives in Figure 5 are all default parameters from their prior settings, which make the comparisons less convincing since the tasks are very different. It would be more convincing if a grid search over the hyperparameters as indicated in their original papers can be applied. 
3. It would be also interesting to see how the performances will be changed with respect to the value of gamma since the transition distributions can be challenging to estimate when the action dimension is large. 

Minor point: 

1. Eq(6) looks confusing because $||e_s^i - e_s^j||$ and $d(a_i,a_j|s)$ are the same by definition.
2. Will the complexity of eq(1) be a problem when the action space is very large? (e.g., on recommendation system situation, the action space can be million scale)

[1]: R. Dadashi, S. Rezaeifar, N. Vieillard, L. Hussenot, O. Pietquin, and M. Geist. Offline reinforcement learning
with pseudometric learning, 2021.
[2] Ghasemipour, Seyed Kamyar Seyed, Richard Zemel, and Shixiang Gu. "A divergence minimization perspective on imitation learning methods." Conference on Robot Learning. PMLR, 2020.
summary_of_the_review: In summary, I think the proposed method BMA is interesting and the evaluation is promising. It would be great to add a few experiments as mentioned above to make more fair comparisons and obtain a more convincing conclusion.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a pseudo-metric between actions, called Behavioral Metric of Actions (BMA), for offline RL in environments with large discrete action spaces. This metric considers both the data-distributional relation and the behavior relation. The distance between two actions drawn from similar distributions with similar rewards and transition probabilities is small. This paper also provides theorems about the continuity and generalization of Q-function using the proposed pseudo-metric. The experimental results show that training a policy in the proposed action representation space has significantly improved performance for offline RL algorithms in environments with large discrete action spaces.
main_review: Strength:
  - This paper is well-written to follow. It clearly explains the problem, the motivation, and the proposed method.
  - It provides theorems on the continuity and the generalization of Q-function.
  - The proposed algorithm on environments with large discrete action spaces can improve other offline algorithms' performance, and the algorithm using the learned representation space outperforms other representations.

Weakness:
  - The proposed BMA is a simple extension of the bisimulation metric [1, 2] for offline RL. The pseudo-metric is a bisimulation metric for actions plus an indicator function that outputs one when the two actions are drawn from the identical distributions and 0 otherwise. 
  The minor thing is that the explanation of Theorem 2 is too similar to that of Theorem 2 of [2]. I think that the explanation should be rewritten.
   
Typo:
  - In eq (8), the reward difference term
  - In the 5.1 Experimental results section on page 8, "annotated by BMA-CQL and BMA-CQL, respectively."

[1] N. Ferns, P. Panangaden, and D. Precup. Metrics for finite Markov decision processes. In UAI, 2004.

[2] A. Zhang, R. McAllister, R. Calandra, Y. Gal, and S. Levine. Learning invariant representations for reinforcement learning without reconstruction.

summary_of_the_review: This paper has interesting experimental results, but the proposed method and the theory are not novel enough to accept.
review_title: null

Meta-review: The paper proposes a new pseudometric for action representations.  The reviewers generally liked the work and the rebuttal helped to clarity may concerns.  However, the degree of novelty of the approach remains a concern.  In addition, a technical error was discovered by a reviewer in the revised paper.  Hence the paper is not ready for publication.
Decision: Reject