Venue: NeurIPS 2022 Workshop Offline_RL

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors present TAP (Trajectory Autoencoding Planner), a method for planning in high-dimensional action spaces. Their method consists of an auto-encoder architecture employing a causal transformer for both the encoder and decoder and vector quantization for the latent codes as introduced in VQ-VAE. Additionally, TAP learns a prior distribution over the discrete latent space using an auto-regressive model. When planning, TAP performs a beam search by first sampling latent codes from the prior and subsequently reconstructing trajectories using the decoder and selecting the best trajectory based on a criterion that includes penalizing out-of-distribution plans under the prior.

Overall I thought the method was well motivated and the empirical results in high-dimensional action spaces are a significant improvement over prior model-based and model-free offline methods. I also appreciate the attention to decision latency and TAP’s ability for near-constant time scaling in decision latency as the action-space dimensionality increases. This paper aligns well with the workshop and I can easily recommend acceptance.

### Other Comments

- Figure 1 uses $v_i$ for return-to-go whereas the text uses $R_i$.
- The appendix has the related work section copied twice.

review_title: A method for planning in high-dimentional action spaces with low latency overhead

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: ### Summary

This paper proposes an encoder-decoder architecture over trajectory that
implicitly allows modelling future trajectories conditioned on the state
and a latent action space. The architecture, Trajectory Autoencoder
Planner (TAP), allows for planning that scales better with respect to
the dimensionality of the action space. The autoencoder is used with
planning by searching over discrete latent actions that have high
probability and high return. Empirically find that TAP is quick and
insensitive to growing action dimension, while surpassing existing
model-based methods and model-free actor-critic methods.

### Strengths

-   Interesting idea, leveraging the strengths of sequence to sequence
    models while implicitly solving an important problem in RL.
-   The experimental results are strong. Despite having strong and
    well-researched baselines, the TAP architecture strongly outperforms
    while being more efficient than other transformer-based architecture
    and traditional model-based planning methods.

### Weaknesses

-   The results, while strong and compelling, are missing confidence
    intervals on the baselines to truly make for a fair comparison.
-   Discussion around temporaility in the raw actions and the latent
    actions, is at times confusing and unclear. I think an example
    grounded in a simple environment would help clarify these details.

### Decision

Overall, this paper is well-written, interesting and relevant to the
Offline RL workshop. For a future submission, I would encourage the
authors to include statistical errors on the baselines for a sound
comparison, as well as a simple example to outline the effective
temporality of the latent actions.

### Detailed Comments

-   Section 1, Paragraph 2 (Actions and temporality): The connection
    between raw actions and the temporal structure of the environment is
    interesting. However, abstract representation of actions, including
    latent codes, do not necessarily escape this restriction. Abstract
    actions like options can be at a coarser grained temporality, but
    this may not be the case for an abstract planner over latent codes.

-   Section 2.1 (latent policy): The trajectory conditioned on a
    particular state is a function of the true policy being followed.
    Not only does the policy determine the state and action
    distribution, it will also determine the rewards-to-go. However, the
    encoder-decoder architecture does not account for this, and places
    the burden on encoding the current structure in the latent actions.
    I think this is an interesting perspective, but it should be made
    more explicit. An interesting small-scale experiment could observe
    the learned latent actions in an MDP that transitions forwards to
    the next state deterministically versus the same MDP that can
    transition backwards to the previous state. In the first MDP, the
    latent action codes are strictly just "latent actions" whereas in
    the second MDP the latent action codes also double as a policy
    encoding.

-   Section 2.1, Paragraph 3 "Firstly, the latent-action model only
    captures possible actions in the support of the behaviour policy":

    This is not obvious to me, because the encoding-decoding
    relationship is not exact there could still be a mismatch in the
    actions of the behavior policy and the actions in the latent-action
    model.

-   Section 2.1, Pargraph 3 "In contrast, planning in the latent action
    space can be much more flexible, as a latent action does not have to
    be tied to a particular step of the transition."

    This could be true, but this is an artifact of the encoder-decoder
    being at the trajectory level, and does not guarantee that any given
    latent code has a coarser temporal resolution than the raw actio. It
    would be interesting to see what the average effective temporality
    is of any given latent code.

-   Section 3, Table 1: The results presented in this table are
    compelling for the proposed TAP method. It is unclear where the
    results on the baselines are from, are they taken from a particular
    paper? While the proposed method includes a confidence interval on
    the results, this is lacking for the baselines, which does not allow
    for a sound statistical evaluation.

review_title: Elegant application of sequence-to-sequence models that implicitly address a key problem in RL

Decision: Accept