Learning Value Functions from Undirected State-only Experience
Authors: Matthew Chang
Arjun Gupta
Saurabh Gupta
Venue: ICLR 2022 Poster (2022)
Keywords: Reinforcement Learning
Offline RL
Offline RL without actions

Abstract: This paper tackles the problem of learning value functions from undirected state-only experience (state transitions without action labels i.e. (s,s',r) tuples). We first theoretically characterize the applicability of Q-learning in this setting. We show that tabular Q-learning in discrete Markov decision processes (MDPs) learns the same value function under any arbitrary refinement of the action space. This theoretical result motivates the design of Latent Action Q-learning or LAQ, an offline RL method that can learn effective value functions from state-only experience. Latent Action Q-learning (LAQ) learns value functions using Q-learning on discrete latent actions obtained through a latent-variable future prediction model. We show that LAQ can recover value functions that have high correlation with value functions learned using ground truth actions. Value functions learned using LAQ lead to sample efficient acquisition of goal-directed behavior, can be used with domain-specific low-level controllers, and facilitate transfer across embodiments. Our experiments in 5 environments ranging from 2D grid world to 3D visual navigation in realistic environments demonstrate the benefits of LAQ over simpler alternatives, imitation learning oracles, and competing methods.

Published as a conference paper at ICLR 2022
LEARNING VALUE FUNCTIONS FROM UNDIRECTED STATE-ONLY EXPERIENCE
Matthew Chang" Arjum Gupta" Saurabh Gupta University of Illinois at Urbana- Chhama-Chaamaign {mc48, jrinn2,, Laurabhg/@illinn edu
ABSTRACT
This paper tackles the problem of learning value functions from undirected state- only experience (state transitions without action labels i.e. (s. tuples), We first theoretically characterize the applicability of Q-learning in this setting. We show that tabular Q-learning in discrete Markov decision processes (MDPs) learns the same value function under any arbitrary refinement of the action space. This theoretical result motivates the design of Latent Action Q-learning or LAQ, an offline RL method that can learn effective value functions from state-only experi- ence. Latent Action Q-learning (LAQ) learns value functions using Q-learning on discrete latent actions obtained through latent-variable future prediction model. We show that LAQ can recover value functions that have high correlation with value functions learned using ground truth actions. Value functions learned using LAQ lead to sample efficient acquisition of goal-directed behavior, be used with domain-specifii low-level controllers, and facilitate transfer across embod- iments. Our experiments in environments ranging from 2D grid world 3D visual navigation in realistic environments demonstrate the benefits of LAQ over simpler alternatives, imitation learning oracles, and competing methods.
INTRODUCTION
Offline batch reinforcement learning focuses on learning goal-directed behavior from pre- recorded data of undirected experience in the form of (st, (s, $141, rz) quadruples. However, in many realistic applications, action information is not naturally available (e.g. when learning from video demonstrations), or worse still, isn't even well-defined (e.g. when learning from the experi- ence of agent with different embodiment). Motivated by such cases, this paper studies if, and how, intelligent behavior can be derived from undirected streams of observations: (5t, St+1, r.).
At the face of it might seem that observation-only data would be useless towards learning goal- directed policies. After all, to learn such a policy, we need to know what actions to execute. Our key conceptual insight that while an observation-only dataset doesn't 't tell us the precise action to execute, i.e. the policy x (als); it may still tell us which states are more likely to lead us to the goal than not, i.e. the value function V(s). For example, simply by looking at someone working in the kitchen, we can infer that approaching the microwave handle is more useful (i.e. has higher value) for opening the microwave than to approach the buttons. Thus, we can still make use of observation- only data, if we focused on learning value functions opposed to directly learning goal-directed policies. Once we have learned a good value function, it can be used to quickly acquire or infer behavior. Using learned value functions as dense rewards lead to quick policy learning through some small amount of interaction in the environment. Alternatively, they could be used to directly guide the behavior of low-level controllers that may already be available for the agent (as is often the case in robotics) without any further training. Furthermore, decoupling the learning of value functions from policy learning enables deriving behavior for agents with different embodiment as long as the overall solution strategy remains similar. Thus, the central technical question is how to learn good value function from undirected observation streams. Is it even possible? If so, under what conditions? This paper tackles these questions from theoretical and practical perspective.
denotes equal contribution. Project wehsite: https://mathhwwhanng www.egggghhhhhhaieee action glearning site/. We assume rt observed. Reward can often be sparsely labeled in observation streams with low effort.

---

Published as a conference paper at ICLR 2022
We start out by characterizing the behavior of tabular Q-learning from Watkins (1989) under missing action labels. We note that Q-learning with naively imputed action labels is equivalent to the TD(O) policy evaluation, which serves simple baseline method for deriving value function. However, depending on the policy that generated the data, the learned values (without any action grounding) can differ from the optimal values. Furthermore, it is possible to construct simple environments where the behavior implied by the learned value function is also sub-optimal.
Next, we present more optimistic result. There are settings in which Q-learnin can recover the optimal value function even in the absence of the knowledge of underlying actions. Concretely, we prove that if we are able to obtain an action space which is a strict refinement of the original action space, then Q-learning in this refined action space recovers the optimal value function.
This motivates a practical algorithm for learning value functions from the given undirected observation-only experience. We design latent-variable future prediction model that seeks to ob- tain a refined action space. It operates by predicting $(+1 from 8t and discrete latent variable from a set of actions Á (Section 4. Training this latent variable model assigns a discrete action å to each (st, $1+1) tuple. This allows to employ Q-learning to learn good value functions (Sec- tion 4.2). The Iearned value function is used to derive behavior (Section 4.3) either through some online interaction with the environment, or through the use of domain specific low-level controllers.
The use of a latent action space for Q-learning allows us to exceed the performance of methods based on policy evaluation (Edwards & Isbell, 2019), which will learn the value of the demonstration policy, not the optimal value function. Additionally, it side-steps the problem of reconstructing high- dimensional images faced by other state-only value learning methods (Edwards et al., 2020). Other approaches for learning from state-only data rely on imitating the demonstration data, which renders them unable to improve on sub-optimal demonstration data. See Section 7 for more discussion.
Our experiments in five environments (2D grid world, 2D continuous control, Atari game Freeway, robotic manipulation, and visual navigation in realistic 3D environments) test our proposed ideas. Our method approximates refinement of the latent space better than clustering alternatives, and in turn, learns value functions highly correlated with ground truth. Good value functions in-turn lead to sample efficient acquisition of behavior, leading to significant improvement over learning with only environment rewards. Our method compares well against existing methods that learn from undirected observation-only data, while being also applicable to the case of high-dimensional observation spaces in the form of RGB images. We are also able to outperform imitation learning methods, even when these imitation learning methods have access to privileged ground-truth action information. Furthermore, our method is able to use observation-only experience from one agent to speed up learning for another agent with a different embodiment.
2 PRELIMINARIES
Following the notation from Sutton & Barto (2018), our Markov decision process (MDP) is spec- ified by (S, A, p. 7),) where is state space, A is an action space, 7 is the discount factor, and p(s', r|s, a) is the state/reward joint dynamics function. It specifies the probability distribution that the agent ends up in state s', receives a reward of r on executing action from state s.
Offline or batch RL (Lange al., 2012: Levine et al., 2020) studies the problem of deriving high reward behavior when only given a dataset of experience in an MDP, in the form of a collection of quadruples (s, a, s)r In this paper, we tackle harder version of this problem where instead we are only given a collection of triplets (s, .r), i.e. experience without information about intervening actions. In general, this dataset could contain any quality of behavior. In contrast to some methods (see Section 7), we will not assume that demonstrations in the dataset are of high quality, and design our method be robust to sub-optimal data. Using such a dataset, our goal is to learn good value functions. A value function under a policy x is function of states that estimates the expected return when starting in s and following π thereafter.
In this paper, we will focus on methods based on Q-learning (Watkins, 1989) for tackling this prob- lem. Q-learning has the advantage of being off off-policy, i.e., experience from another policy (or task) can be used to learn improve a different policy for different task. Q-learning seeks to learn the optimal Q Q-function Q' (s, a) by iteratively updating Q(+,, 4x) to the Bellman equation. This process converges to the Q" under mild conditions in many settings (Watkins, 1989).
2

---

Published as a conference paper at ICLR 2022
With Ground Truth Actions (Vg)) MSE:0.000
With One Action (V act) MSE:0.064
With 4x Refined Actions (Vax-gt) or) MSE:0.000
With Obfuscated Actions
(Vimpure -ad) MSE:0.017
1.0
0.9
0.8
0.7
0.6
0.5
0.4
We visualize the learned value function when using different action labels for Q-learning: ground truth actions, one single action, 4x refined action space, and obfuscated actions. We also report the mean squared error from the optimal value function. Arrows show the behavior induced by the value function (picking neighboring state with highest value). We note that a) ignoring the intervening actions works poorly both in terms of value function estimates and the induced behavior, b) refined actions don't 't cause any degradation in performance, c) noisy actions that don't 't adhere the refinement constraint cause degradation in performance. See Section 3.2 for more details.
Figure 1:
3 CHARACTERIZING Q-LEARNING WITHOUT TRUE ACTION LABELS
We characterize the outcome of Q-learning in settings where we don't have ground truth intervening actions in the offline dataset being used for Q-learning. Without action labels, one could simply assign all transitions the same label. In this case, Q-Learning becomes TD(0) policy evaluation. The induced value function isn't 't the optimal value function for the MDP, but rather the value according to the policy that generated the dataset. Depending on the dataset, this could be sub-optimal.
Next, we study if labeling (s, r'r) samples with actions from different action space Å to construct a new MDP could aid learning. More specifically, can the optimal Q-function for this new MDP, as obtained through Q-learning on samples (8, ,,,r) labeled with actions from Å, be useful in the original MDP? We show that under the right conditions the value function learned under the altered action space Å is identical to the value function learned for the original MDP.
3.1 OPTIMALITY OF ACTION REFINEMENT
Assume we have Markov Decision Process (MDP) M specified by (S, A, P. 7). Let the action space A be composed of actions a1, 02 03, ..., On E A. We are interested the value learned under a modified MDP, M composed of (S, p) We will show that if the actions and transitions Å and & are a refinement of A and p. then the value function learned on M, Vir identical to the value function learned on M, VM. We define actions and transitions in M to be a refinement of those M when, a) each state, for every action in A, there is at least one action A which functionally identical in the same state, and b) in each state, for each action in A is represented by at least one action in Å in that state.
Definition 3.1 Given a discrete finite MDP, M specified by (S, A,p) and MDP, M specified by (S, A p). M is a refinement of M when V y p(s', rls,, à) p(s rls,a), and SES acA e
3 y p(s), r(s, p(s)rrls,a)
Note that this definition of refinement requires a state conditioned correspondence between action behavior. Actions do not need to have to correspond across states.
Theorem 3.1. Given a discrete finite MDP, M which is a refinement of M (Definition 3.1) then both MDPs induce the same optimal value function, i.e. V. V. (s) (s).
We prove this by showing that optimal policies under both MDPs induce the same value function.
Lemma 3.2. For any policy wM on M, there exists policy aa on M such that V (s) V (s), ys, and for any policy π is on M there exists policy was on M such that V a
As
X (s)
V (s) Vs.
3

---

Published as a conference paper at ICLR 2022
Labeling
Action
Latent
Mukinay
G-Learning
V(8)
Low--Lovel
Controllar
Environment
Densified
RL
Figure 2: Approach Overview. Our proposed approach Latent Action Q- Q-Learning (LAQ) starts with dataset of (s, (s,)',r) triples. Using the latent action learning process, each sample is assigned latent action à. Q-learning on the dataset of quadruples produces value function, V()). Behaviors are derived from the value function through densified RL, or by guiding low-level controllers.
For this lemma we introduce the notion of fundamental actions, which are actions which correspond to sets of actions which have the same state and reward transition distributions in given state. We utilize the equivalence of fundamental actions between MDPs to construct a policy in the new MDP which induces the same value function given policy in the original MDP. We provide proofs for Theorem 3.1 and Lemma 3.2 in Section A.2.
3.2 GRIDWORLD CASE STUDY
We validate these results in a tabular grid world setting. In particular, we measure the error in learned value functions and the induced behavior, when conducting Q-learning with datasets with different qualities of intervening actions. The agent needs to navigate from the top left of 6 x grid to the bottom right with sparse reward. We generate data from fixed, sub-optimal policy to evaluate all methods in an offline fashion (additional details in Section A.7). We generate 20K episodes with this policy, and obtain value functions using Q-learning under the following 4 choices for the intervening actions: (1) Ground truth actions (Vg.). g. (2) One action (Vane-act, ammounts to TD()) policy evaluation), (3) 4x refinement of original action space (V4x-g). We modify the data so that each sample for a particular action in the original action space is randomly mapped to one of actions the augmented space. (4) Obfuscated actions (Vimpure-act). Original action with probability 0.5, and a random action with probability 0.5.
Figure shows the learned value functions under these different action labels, and reports the MSE from the true value function, along with induced behavior. In line with our expectations, V4x-gt which uses refinement of the actions is able recover the optimal value function. Vone-act fails to recover the optimal value function, and recovers the value corresponding to the policy that generated the data. Vimpure-act: under noise in action labels (non-refinement) also fails to recover the optimal value function. Furthermore, the behavior implied by Vímpure-act and Vone-act is sub-optimal. We also analyze the effect of the action impurity on learned values and implied behavior. Behavior becomes increasingly inaccurate action impurity increases. More details in Section A.4.
4 LATENT ACTION --LEARNING
Our analysis in Section 3 motivates the design of our approach for learning behaviors from state- only experience. Our proposed approach decouples learning into three steps: mining latent actions from state-only trajectories. using these latent actions for Q-learning to obtain value functions, and learning policy to act according to the learned value function. As per our analysis, if learned latent actions are state-condine conditiend refinement of the original actions, Q-learning will result in good value functions, that will lead to good behaviors. Refer to Algorithm for details.
4.1 LATENT ACTIONS FROM FUTURE PREDICTION
Given a dataset D of observations streams .... Ot, 0t+1s the goal in this step is learn latent actions that are a refinement of the actual actions that the agent executed 2 We learn these latent actions through future prediction. We train future prediction model fe. that maps the observation of at time 1, and a latent action å (from set Á of discrete latent actions) to the observation 0++1 at time t + 1, i.e. fo(or, â). f is trained to minimize a loss / between the prediction fo(or. â) and the ground truth observation 0++1- à is treated as a latent variable during learning. Consequently, fo is trained using a form of expectation maximization (Bishop, 2006). Each training sample (ot, o++1)
We use the terms state (st) and observation (ot) interchangeably.
4

---

Published as a conference paper at ICLR 2022
is assigned the action that leads to the lowest loss under the current forward model. The function fo is optimized to minimize the loss under the current latent action assignment, More formally, the loss for each sample (o,, 0++1) is: [(o,, 01+1) min I (folor, o.+1).). We minimize E(o L(or, 0t+1) over the dataset to learn fe.
Latent action Ôr for observation pairs (or, 0t+1) are obtained from the learned function fo as: arg ai lem. We use L2 loss in the observation space -iw-dimeniional states, or images).
Á l (fo â), 0t-1). Choice of the function fe and loss vary depending on the prob-
4.2 Q-LEARNING WITH LATENT ACTIONS
Latent actions mined from Section 4.1 allow us to complete the given (or, 0t+1, Tr) tuples into (or, de, 0t41, rx) quadruples for use in Q-learning Watkins (1989). As our actions are discrete we can easily adopt any of the existing Q-learning methods for discrete action spaces (e.g. Mnih et al. (2013)). Though, we note that this Q-learning still needs to be done an offline manner from pre- recorded state-only experience. While we adopt the most basic Q-learning in our experiments, more sophisticated versions that are designed for offline Q-learning (e.g- Kumar et (2020); Fujimoto et al. (2019)) can be directly adopted, and should improve performance further. Value functions obtained from the Q-functions as V (()) = m Q(s, a).
4.3 BEHAVIORS FROM VALUE FUNCTIONS
Given a value function, our next goal is to derive behaviors from the learned value function. In general, this requires access to the transition function the underlying MDP. Depending on what assumptions we make, this will be done in the following two ways.
Densified Reinforcement Learning. Learning a value function from state-only experience can be extremely valuable when dense reward function for the underlying task is not readily available. In this case, using the learned value function can densify sparse reward functions, making previously intractable RL problems solvable. Specifically, we use the value function to create a potential-based shaping function F(s, s') = V(s') (s)), based on Ng et al. (1999), and construct an augmented reward function (ss a, s') r(s, a, s') + F(s,s". Our experiments show that using this densified reward function speeds up behavior acquisition.
Domain Specific Low-level Controllers. In more specific scenarios, it may be possible employ hand designed low-level controllers in conjunction with a model that can predict the next state s' on executing any of low-level controllers. In such situation, behavior can directly be obtained by picking the low-level controller that conveys the agent to the state s' that has the highest value under the learned (s). Such technique was used by Chang al. (2020). We show results in their setup.
5 EXPERIMENTS
We design experiments assess the quality of value functions learned by LAQ from undirected state-only experience. We do this in two ways. First, we measure the extent to which value func- tions learned with LAQ without ground truth information agree with value functions learned with Q-learning with ground truth action information. This provides direct quality measure and allows us compare different ways of arriving at the value function: other methods in the literature (D3G (Edwards et al., 2020)), and simpler alternatives of arriving at latent actions. Our second evaluation measures the effectiveness of LeQ-learned value functions for deriving effective behavior in differ- ent settings: when using it as a dense reward, when using it to guide low-level controllers, and when transferring behavior across embodiments. Where possible, we compare to behavior cloning (BC) with privileged ground truth actions. BC with ground truth actions serves performance of state-only imitation learning methods (BCO from Torabi et al. (2018a), ILPO from Edwards et al. (2019), etc.) and allows to indirectly compare with these methods.
upper bound on the
Test Environments. Our experiments are conducted in five varied environments: the grid world environment from Section 3, the Atari game Freeway from Bellemare et al. (2013), 3D visual nav- igation in realistic environments from Chang et al. (2020); Savva et al. (2019), and two continu- control tasks from Fu et al. (2020)'s D4RL: Maze2D (2D continuous control navigation), and
5

---

Published as a conference paper at ICLR 2022
S
G
2D Grid World
Freeway
3D Visual Nav.
2D Maze
Kitchen
Figure 3: We experiment with five environments: 2D Grid World, Freeway (Atari), 3D Visual Navigation, Maze2D (2D Continuous Control), and FrankaKitchen. Top right corner of Maze2D and FrankaKitchen, shows the embodiments for eessdimmbdi transfer (ant and hook, respectively).
Table We report Spearman's correlation coefficients for value functions learned using various methods with DQN, against a value function learned offline using ground-truth actions (DQN for discrete action environments, and DDPG for continuous action environments). The Ground Truth Actions column shows Spearman's 's correlation coefficients between two different runs of offline learning with ground-trute actions. See Section 5.1. Details on model selection in Section A.11.
Environment
D3G
Single Action
Chustering
Clustering (Diff)
Latent Actions
Ground Truth Actions
2D Grid World
0.959
0.093
0.430
1,000
0.985
1,000
Freeway
(image input) (image imput)
0.961
3D Visual Navigation
0.641
0.722
0,827
0.927
0.991
2D Continoous Control
0.673
0.673
0.613
0.490
0.844
0.851
Kitchen Manipulation
0.854
0,858
0.818
0.815
0.905
0.901
FrankaKitchen (dexterous manipulation in a kitchen). For Maze2D and FrankaKitchen environ- ments, we also consider embodiment transfer, where we seek to learn policies for an ant and hook respectively from the observation-only experience of point mass and the Franka arm. Together, these environments test our approach on different factors that make policy learning hard: continuous control, gigh-dimensional observations and control, complex real world appearance, 3D geometric reasoning, and learning across embodiments. Environments are visualized in Figure more details about the environments provided in Section A.6.
Experimental Setup For each setting, we work with a pre-collected dataset experience in the form of state, next state and reward triplets, (ot, 01+1, Tt). We
latent-variable forward model (Section 4.1) and label triplets with latent actions to obtain quadruples (o,, â,, r). We perform Q-learning on these quadruples obtain value functions V(s), which are used acquire behaviors either through densified RL by interacting with the environment, or through use of domains-specific low-level controllers. We use the ACME codebase (Hoffman et al., 2020) for experiments.
Latent Action Quality. In line with the theory developed in Section 3, we want to establish how well our method learns refinement of the underlying action space. To assess this, we study the state-conditioned purity of the partition induced by the learned latent actions (see definition in Sec- tion A.9). Overall, our method is effective at finding refinement of the original action space. achieves higher state-conditioned purity than single action and clustering. In high-dimensional image observation settings, it surpasses baselines by a wide margin. More details in Section A.9.
5.1 QUALITY OF LEARNED VALUE FUNCTIONS
We evaluate the quality of the value functions learned through LAQ. We use reference the value function V Vgt-ects obtained through offline Q-learning (DDPG for continuous action cases) with true ground truth actions iee (ot, az Ot+ rt).² For downstream decision making, we only care about the relative ordering of state values. Thus, we measure the Spearman's 's rank correlation coefficient between the different value functions. Table reports the Spearman's 's coefficients of value functions
3 Offline DDPG in the FrankaKitchen environment was unstable. To obtain a reference value function, we manually define a value function based the distance hetween the end-effector and the microwave handle (lower better), and the angle of the mierowave door (higher better). We use this the reference value function.

---

Published as a conference paper at ICLR 2022
Gridworld
2D Continuous Navigatiion
Frenway
1.0
Normaized Return
01.8
Stocceos Date
25
0.6
0.6
Reward
20
15
0.4
0.4
10
022
0
5000
10000
15000
20000
0.0
0.5
1.0
3.5
2.0
0.00
0.25
0.50
0.75
1.00
1.25
Environment, Steps
Environment Steps
1
Environment Steps
168
Sparse Reward
BC (around truth actions)+RLL RL
Densified Reward (D3G)
BC (ground truth actions)
Densified Reward (Ours)
Densified Reward (Ours BCQ)
Densifled Reward (TD(O1)
We show learning curves for acquiring behavior using learned value functions. We com- pare densified RL (Section 4.3) with sparse RL and BC/BC+RL. See Section 5.2 for more details. Results are averaged over 5 seeds and show + standard error.
Figure 4:
obtained using different action labels: single action, clustering, latent actions (ours), and ground truth actions. We also report Spearman's 's correlations of value functions produced using D3G (Ed- wards et al., 2020). In all settings we do Q-learning over the top 8 dominant actions, except for Freeway, where using the top three actions stabilized training. Our method out performs all base- lines in settings with high-dimensional image observations (3D) Visual Navigation, Freeway). In state based settings, where clustering state differences is a helpful inductive bias, method is still on- par with, or superior to clustering state differences and even D3G, which predicts state differences.
5.2 USING VALUE FUNCTIONS FOR DOWNSTREAM TASKS
Our next experiments test the utility of LAQ-learned value functions for acquiring goal-driven be- havior. We first describe the 3 settings that we use to assess this, and then summarize takeaways.
Using value functions as dense reward functions. We combine sparse task reward with the learned value function
potential funetion (Section 4.3). We scale up the sparse task rewards by a factor of so that behavior is dominated by the task reward once policy starts solving the task. Figure 4 measures the learning sample efficiency. We compare to only using the sparse reward, behavior cloning (BC) with ground truth actions, and BC followed by spare reward RL. Using value functions to learn behavior of an agent with different embodiment. Decoupling the learning of value function and the policy has the ad advantage that learned value functions can be used to improve learning across embodiment. We demonstrate this, we keep the same task, but change the embodiment of the agent in Maze2D and FrankaKitchen environments. Note that we do not assume access to ground truth actions in these experiments either. For Maze2D, the point- mass is replaced with 8-DOF quadrupedal ant. For FrankaKitchen, the Franka arm is replaced with a position-controlled hook. We may need to define how we query the value function when the embodiment (and the underlying state space) changes. For the ant in Maze2D, the location (with 0 velocity) of the ant body is used to query the value function learned with the point-mass. For the hook in FrankaKitchen, the value function is able transfer directly as both settings observe end-effector position and environment state. We report results in Figure 5.
Using value functions to guide low-level controllers. Learned value functions also have the advantage that they can be used directly at test time to guide the behavior of low-level controllers. We do this experiment in context of 3D visual navigation in scan of a real building and use the branching environment from Chang et al. (2020). We follow their setup and replace their value functions with ones learned using LAQ in their hierarchical policy, and compare the efficiency of behavior encoded by the different value functions.
LAQ value functions speed up downstream learning. Learning plots in Figure 4 show that L.QQ- leamned value functions speed up learning in the different settings over learning simply with sparse rewards (orange line vs. blue line). In all settings except Freeway, our method not only learns more quickly than sparse reward, but converges to a higher mean performance.
LAQ discovers stronger behavior than imitation learning when faced with undirected experi- ence. An advantage of LAQ over other amitation-leamiin based methods such as BCO (Torabi et al., 2018a) and ILPO (Edwards et al., 2019) is LAQ's ability to learn from sub-optimal or undirected experience. To showcase this, compare the performance of LAQ with behavior cloning (BC)
7

---

Published as a conference paper at ICLR 2022
Table We report Spearman's correlation coefficients for value functions learned using either DQN or BCQ, against value function learned offline using BCQ with ground-truth actions. The Ground Truth Actions column shows Spearman's 's correlation coefficients between two different runs of offline learning with ground-truth actions. See Section 5.1.
Environment
Single Action
Clustering
Clustering (Diff)
Latent Actions
Ground Truth Actions
2D Continuous Control (DQN)
0.664
0.431
0.312
0.807
0.765
Continuous Control (BCQ)
0.719
0.927
Navigation Transfer to Ant
Kitchen Manipulstion Transfer
G
Noward
Latert Q-isering
0.5
Deneiiteed frwand 800)
0.5
G
Fird Rlwwrrd (03G)
0.4
Deneitted Reward (TTac))
Success Rote
Polecy Foskustion Yers-ur
Success
0.3
Interaction
Samples
SPI.
Vaneect (Chang el al., 2020)
0
0.53
0.1
Vauseract
0
0.57
Vaan-a
0
0.82
0.0
0.0
Vinenoaa (Chang et al., 2020)
40K
3
0.6
0.8 1.0
Environment Steps
1e5
Enviromment
lse
Figure 6: Visualization of trajecto- ries and SPL numbers in the 3D vi- sual navigation environment.
Figure 5: Behavior acquisition across embodiments. Re-
sults averaged over 50 seeds and show + standard error.
with ground truth actions. Since BCO and ILPO recover ground truth actions to perform behavior cloning (BC), BC with ground truth actions serves as an upper bound on the performance of all methods in this class. Learning plots in Figure 4 shows the effectiveness of LAQ over BC and BC followed by fine-tuning with sparse rewards for environments where the experience undirected (Maze2D, and GridWorld), For Freeway, the experience is fairly goal-directed, thus BC already works well. similar trend can be seen in the higher Spearman's 's coefficient for LAQ vs.. Vane in Table 1. LAQ discovers stronger behavior than imitation learning when faced with undirected data.
LAQ is compatible with other advances in batch RL. Although LAQ uses the most basic Q- Learning our offline value learning method, it is compatible with recent more advanced offline RL value-learning methods (such as CQL (Kumar et al., 2020) and BCQ (Fujimoto et al., 2019)). We validate by simply swaping to using (discrete) BCQ with our latent actions. Figures. 4 and 5 show that LAQ with BCQ is the strongest method, outperforming ours with DQN, and D3G, on Maze2D and embodiment transfer environments. Analysis of Spearman's 's correlations in Table shows the same trend as before with latent actions: better than single actions, and clustering variants. Note also that use of BCQ leads to value functions with better Spearman's 's correlations than DQN.
LAQ value functions allow transfer across embodiments. Figure shows learning plots of agents trained with cross-embodiment value functions. LAQ--densiiied rewards functions, speed-up learn- ing and consistently guide to higher reward solutions than sparse task rewards, or D3G.
LAQ compares favorably to D3G. We compare LAQ and D3G (a competing state-only method) in different ways. D3G relies on generating potential future states. This is particularly challenging for image observations, and D3G doesn't 't show results with image observations. In contrast, LAQ maps state transitions to discrete actions, and hence works with image observations as our experiments show. Even in scenarios with oiw-dimensional state inputs, LAQ learns better value functions than D3G, evidences by Spearman's correlations in Table 1, and learning plots in Figure 4 and Figure 5.
We report the SPL. for 3D navigation using value functions combined with low-level controllers in Figure 6. We report the efficiency of behavior induced by LAQ learned value functions as measured by the SPL metric from Anderson et al. (2018) (higher is better). The branching environment has two goal states, one optimal and one sub-optimal. The demonstrations there-in were specifically designed to emphasize the utility of knowing the intervening actions. Simple policy evaluation leads sub-optimal behavior (SPL of 0.53) and past work relied on using an inverse model to label actions (Chang et al., 2020) to derive better behavior. This inverse model itself required 40K interactions with the environment
LAQ value functions can guide low-level controllers for zero-shot control:
8

---

Published as a conference paper at ICLR 2022
for training, and boosted the SPL to 0.95. LAQ is able to navigate the optimal goal (w/ SPL 0.82) but without the 40K online interaction samples necessary to acquire the inverse model. also performs better than clustering transitions, doing which achieves an SPL of 0.57. The improvement borne out in visualizations in Figure 6. LAQ correctly learns to go to the nearer goal, even when the underlying experience came from a policy that preferred the further away goal.
DISCUSSION
Our theoretical characterization and experiments in five representative environments showcase the possibility and potential of deriving goal-directed signal from undirected state-only experience. Here we discuss some scenarios which are fundamentally hard, and some avenues for future research.
Non-deterministic MDPs. Our theoretical result relies on refinement where state-action tran- sition probabilities are matched. However, the latent action mining procedure in LAQ results in deterministic actions. Thus, for non-deterministic MDPs, LAQ will be unable to achieve a strict refinmment, leading to sub-optimal value functions. However, note that this limitation isn' specific to our method, but applies to any deterministic algorithm that seeks to learn from observation only data. We formalize this concept and provide a proof in Section A.3.
Constraining evaluation of (s) to within domain. L.Q learns a value function V(s) over the set of states that were available in the experience dataset, and as such its estimates are only accurate within this set. In situations where the experience dataset doesn't span the entire state space, states encountered at test time may fall out of the distribution used to train Vss), leading to degenerate solutions. In Section A. we discuss a density model based solution we used for this problem, along with alternate parameterization of value networks which helps avoid degenerate solutions.
Offline RL Validation. Validation (e.g. when to stop training) is a known issue in offline RL (Gul- cehre et al., 2020). Like other offline RL methods, LAQ suffers from it too. LAQ's use of Q-learning makes it compatible to recent advances (Kumar et al., 2021) that tackle this validation problem.
7 RELATED WORK
Our work focuses batch (or offline) RL with state-only data using a veriable future predic- tion model. We survey works on batch RL, state-only learning, and future prediction.
Batch Reinforcement Learning. As the field of reinforcement learning has matured, batch RL (I ange et al., 2012; Levine et al., 2020) has gained attention as a component of practical systems. A large body of work examines solutions the problem of extrapolation error in batch RL settings. Advances in these works are complementary to our approach, as substantiated by our experiments with BCQ. A more detailed discussion of batch RL methods can be found in Section A.I.
State-only Learning. Past works have explored approaches for dealing with the lack of actions in offline RL when given goal-directed undirected state-only experience. Works in the former category rely on high quality behavior in the data, and suffer on sub-optimal data. Past work on state- only learning from undirected experience relies on either domain knowledge or state reconstruction and only show results with low dimensional states. See Section A. for continued discussion.
Future Prediction Models. Past work from Oh et al. (2015); Agrawal al. (2016); Finn et al. (2016) (among many others) has focused on building action conditioned forward models in pixel and latent spaces. Yet other work computer vision studies video prediction problems (Xue al., 2016; Castrejon et al., 2019). Given the uncertainty in future prediction, these past works have pursued variational (or latent variable) approaches to make better predictions. Our latent variable future model is inspired from these works, but explore its applications in novel context.
Latent MDP Learning One way to interpret our method is that of learning an approximate MDP homomorphism (Taylor et al., 2008; Ravindran & Barto, 2004). Other works have explored learning latent homorphic MDPs. These methods tend to focus on learning equivalent latent state spaces (Li et al., 2006; Givan al., 2003). Most similarly to work (van der Pol et al., 2020) also learns latent action space, but relies on ground truth action data to do so.
Acknowledgement: This paper is based on work supported by NSF under Grant 2005-2007035.

---

Published as a conference paper at ICLR 2022
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pp. 104-114, PMLR, 2020. 19, 20
Pulkit Agrawal, Ashvin V Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physics. In NeurlPS, 2016.
Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, and Amir Zamir. On evaluation embodied navigation agents. arXiv preprint arXiv: 1807.06777 2018. 8
Arthur Argenson and Gabriel Dulac-Arnold, Model-based offline planning, 2021. 13
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, jun 2013. 5
Christopher M Bishop. Pattern recognition and machine learning. springer, 2006. 4
Lluis Castrejon, Nicolas Ballas, and Aaron Courville. Improved conditional vrmns for video pre- diction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7608-7617, 2019. 9
Matthew Chang, Arjun Gupta, and Saurabh Gupta, Semantic visual navigation by watching youtube videos. In NeurIPS, 2020. 5, 7, 8, 13, 19, 20, 24
Ashley D. Edwards and Charles L. Isbell. abs/1905.07866 2019. 2, 13
Perceptual values from observation. CoRR,
Ashley D Edwards. Himanshu Sahni, Yannick Schroecker, and Charles L Isbell. Imitating latent policies from observation. In ICML,, 2019. 5, 7,
Ashley D. Edwards, Himanshu Sahni, Rosanne Liu, Jane Hung. Ankit Jain, Rui Wang, Adrien Ecoffet, Thomas Miconi, Charles Isbell, and Jason Yosinski. Estimating g(s, 8') with deep deter- ministic dynamics gradients. In ICML, 2020. 2, 7, 13
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. arXiv preprint arXiv: 160.00077 2016. 9
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020. 5, 13, 19
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In ICMI, 2019. 5, 8, 13
Robert Givan, Thomas Dean, and Matthew Greig. Equivalence notions and model minimization in markov decision processes. Artificial Intelligence, 147(1-2):163-223, 2003. 9
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Tom Le Paine, Sergio Gómez Colmenarejo, Kon- rad Zolna, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cosmin Paduraru, Gabriel Dulac- Arnold, Jerry Li, Mohammad Norouzi, Matt Hoffman, Ofir Nachum, George Tucker, Nicolas Heess, and Nando deFreitas. unplugged: Benchmarks for offline reinforcement learning, 2020. 9, 13
Matt Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang. Kate Baumli, et al. Acme: research framework for distributed reinforcement learning, arXiv preprint 20Xiv:2006.00979 2020.
Ashish Kumar, Saurabh Gupta, and Jitendra Malik. Learning navigation subroutines by watching
videos. In CoRL, 2019a. 13
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off off-policy q-learning via bootstrapping error reduction. In NeurlPS, 2019b. 13
10

---

Published as a conference paper at ICLR 2022
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In NeurlPS, 2020. 5, 8, 13
Aviral Kumar, Anikait Singh, Stephen Tian, Chelsea Finn, and Sergey Levine. A workflow for offline model-free robotic reinforcement learning. In CoRL, 2021.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning, In Reinforce- ment learning, pp. 45-73. Springer, 2012. 2,9
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tu- torial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020. 2,
Lihong Li, Thomas J Walsh, and Michael L Littman. Towards a unified theory of state abstraction for mdps. ISAIM, 4:5, 2006. 9
Ajay Mandlekar, Fabio Ramos, Byron Boots, Silvio Savarese, Fei-Fei Li, Animesh Garg, and Dieter Fox. IRIS: implicit reinforcement without interaction at scale for learning control from offline robot manipulation data. In ICRA, 2020. 13
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 1r21v:131 2013. 5
Andrew Y. Ng, Daishi Harada, and Stuart J. Russell. Policy invariance under reward transformations: Theory and application reward shaping. ICML, pp. 278-287. Morgan Kaufmann, 1999. 5
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis, and Satinder Singh. Action-conditionn video prediction using deep networks in atari games. arXiv preprint arXiv: 2015. 9
Ilija Radosavovic, Xiaolong Wang, Lerrel Pinto, and Jitendra Malik. State-only imitation learning for dexterous manipulation. CoRR, abs/2004.046571 2020. 13
Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn, Offline reinforcement learning from images with latent space models. CoRR, ab572012.11547,, 2020. 13
Aravind Rajeswaran, Chelsea Finn, Sham Kakade, and Sergey Levine. Meta-learning with implicit gradients. arXiv preprint arXiv:1909, 040000 2019. 13
Balaraman Ravindran and Andrew G Barto. Approximate homomorphisms: A framework for non- exact minimization in markov decision processes. 2004. 9
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Brik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A platform for embodied Al research. In ICCV, 2019. 19, 20
Pierre Sermanet, Kelvin Xu, and Sergey Levine. Unsupervised perceptual rewards for imitation learning. In RSS, 2017. 13
Lin Shao, Toki Migimatsu, Qiang Zhang, Karen Yang, and Jeannette Bohg. Concept2robot: Learn- ing manipulation concepts from instructions and human demonstrations. In RSS, 2020. 13
Avi Singh, Larry Yang, Chelsea Finn, and Sergey Levine. End-to-end robotic reinforcement learning without reward engineering. RSS, 2019.
Shuran Song, Andy Zeng, Johnny Lee, and Thomas Funkhouser. Grasping in the wild: Learning 6dof closed-loop grasping from low-cost demonstrations, Robotics and Automation Letters, 2020. 13
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249, 2,
Jonathan Taylor, Doina Precup, and Prakash Panagaden. Bounding performance loss in approximate mdp homomorphisms. Advances in Neural Information Processing Systems, 2008. 9

---

Published as a conference paper at ICLR 2022
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. In I/CAI, 2018a. 5, 7, 13
Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation. CoRR, abs/1807.06158, 2018b. 13
Faraz Torabi, Garrett Warnell, and Peter Stone. Adversarial imitation learning from state-only demonstrations. In AAMAS, pp. 2221.2231. International Foundation for Autonomous Agents and Multiagent Systems, 2019a. 13
Faraz Torabi, Garrett Warnell, and Peter Stone. Imitation learning from video by leveraging propri- oception. arXiv preprint arXiv: /tt.///000..93 00335, 2019b. 13
Elise van der Pol, Thomas Kipf, Frans A. Oliehoek, and Max Welling. Plannable approximations MDP homomorphisms: Equivariance under actions. In International Conference on Autonomous
Agents and Multiagent Systems, AAMAS, 2020.
Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. 1989. 2, 5,
Annie Xie, Avi Singh, Sergey Levine, and Chelsea Finn. Few-shot goal inference for visuomotor learning and planning. In Conference Robot Learning, pp. 40-52. PMLR, 2018. 13
Tianfan Xue, Jiajun Wu, Katherine L. Bouman, and William Freeman. Visual dynamics: Proba- bilistic future frame synthesis via cross convolutional networks. arXiv preprint arXiv:1607.02588 2016. 9
12

---

Published as a conference paper at ICLR 2022
A APPENDIX
A. RELATED WORK CONTINUED
Discussion continued from Section
Batch Reinforcement Learning. In recent times, Gulcehre et al. (2020) and Fu et al. (2020) pro- pose datasets and experimental setups for studying offline RL problems. large body of work examines solutions to the batch RL problem. Researchers have identified that extrapolation enror, phenomenon in which batch RI. algorithms incorrectly estimate the value of states/actions not present in the training batch, is major challenge, and have proposed methods to tackle it, e.g. BCQ (Fujimoto et al., 2019), BEAR (Kumar et al., 2019b), IRIS from (Mandlckar et al., 2020), and CQL (Kumar et al., 2020) among many others. In contrast to these model-free methods, Argenson & Dulac-Arnold (2021); Rajeswaran et al. (2019); Rafailov et al. (2020) learn forward predictive model from the batch data and use it for model predictive control. These methods all approach the traditional batch RL problem, while we consider different and harder setting in which the action labels are unavailable. Aforementioned advances in offline RL complementary to our work. Offline value learning approaches (such as CQL and BCQ) can serve as a drop-in replacement for Q-learning our pipeline and improve our results. In fact, our experiments with BCQ substantiate this.
State-only Learning. In line of work which studies learning form goal-directed state-only experi- ence, researchers use imitation learning- -based techniques (Radosavovic et al., 2020; Torabi et al., 2018a; Edwards al., 2019; Kumar al., 2019a), learn policies that match the distribution vis- ited states (Torabi et al., 201 2019a;b), or use demonstrations to construct dense reward functions (Shao et al., 2020; Sermanet al., 2017: Singh et al., 2019; Xie et al., 2018; Edwards & Isbell, 2019). These methods make strong assumptions about the quality and goal-directed nature of the experience data, and suffer in performance when faced with low-quality or undirected experience.
Instead of goal-directed experience perience. Past work in this area employs Q-learning to learn optimal behavior from sub-optimal data (Chang al., 2020; Song et al., 2020; Edwards et al., 2020). Chang et al. (2020) and Song et al. (2020) use domain specific insights. Edwards et al. (2020) rely on being able to generate the state and only demonstrate results environments with low-dimensional states. Instead, our work maps transition tuples to discrete latent actions and can thus easily work with high-dimensional observations such RGB images.
work tackles the problem of learning from undirected ex-
13

---

Published as a conference paper at ICLR 2022
A.2 PROOFS
Proof. Proof for Lemma 3.2. We will start by showing that for any policy "M on M, there exists policy
M such that V (s) VM (s), for all s.
M
To do this we need to introduce the idea of fundamental actions, which are classes of actions which have the same state and reward transition distributions in a given state. If we have a fundamental action b, corresponding some state and reward distributions, let a(b, s) A give the set of actions in A that have the matching state and reward transition distributions,
A p(s's rls, a) = riss,rls,ccn als,,(())))) aEn(h.s)
Similarly, let (b)b,) S Á give the set of actions in Å belonging to in state In any given state, there are at most min(|A] |Al) fundamental actions for M and the union of all actions belonging to all fundamental actions gives the set of actions that make up the original action space. For our MDP let's denote (s) as the set of fundamental actions in the state 8 for M, and (s)) as the set of fundamental actions in the state for M. Let [(s, a), and (s,, a) be functions which return the set of actions which correspond the same fundamental action containing as a in state s for M, and M respectively. This means,
U a(b,s) ((s
A. y a(b, s) (a( s) and BbBB(s) s). (1) beB
aελ,s
With this notation out of the way value in M as TM does M. We do this by constructing " such that the probability distributions over each fundamental action is equivalent. We define this policy
construct policy xxr from TM, which achieves the same
TM (als) [3(s,a)
Σ
(āls).
aEL(c.a)
Following Sutton & Barto (2018) can define the value function for given policy V (s)
ES [G:]S: = s].
V (8) Σ (als)
Epss' rls, (rr (s')
acA
s/ir
From the properties Eq. 1, we know sum over fundamental actions will count each action exactly once, so we can write this sum as
AM M
Σ Σ
(al(s
Σ r|s, + TM (s')..
(2)
BEB(s) aEa(b.s)
substituting the constructed policy we have
VTM (8) = Σ Σ M
Σ (als))))
Σ EM(s,rls,a) ~VZ (s'))
(B(s
be B(x) aEa(b.s)
aEB(a.a)
since the third sum is over A(s, a) with E a(b, s), we can rewrite this sum over à E a(b, giving
V (s)
Σ Σ
Σ (ā(s) Epss' rp(strris.d)rr +VV" (s')].
[8(ssa)] dea(b,s)
bEB(s) aEa(b.s)
For the final sum, from the definition of fundamental actions, we know p(s', r|s, a) = p(s', r|s, o(b, p(s', rls, a(b, 8)1), so rewrite that term
EMs
Ms alb, V (s')]-
14

---

Published as a conference paper at ICLR 2022
Crucially, in the second sum (over a(b,s) B(s, a) is the same for all E a(b, s). the term in brackets can be treated as a constant. Similarly, since [3(s,a)] = |a(b, s) we can simplify the second sum leaving
RMM M
(s)
(āls) bEB(s) dca(b.s)
Σ
Σ
Esss a(b, 7 (5))
s',r
This gives the definition of V (s) using the same decomposition equation Eq. 2, meaning M
(M
(s)
V M
(s).
One can show the opposite direction, that policy there exists policy TM for any policy aM such V (s) V (s), with a symmetric construction.
Note that Lemma 3.2 holds true regardless the stochasticity of the policy, as the constructed policy matches probability of the original policy for taking each fundamental action.
Proof for Theorem 3.1. Under the MDP M, Q-learning will learn the same optimal value function value function as learned under MDP M, iee. V V.V (+)
Proof. This follows from Lemma 3.2 contradiction. Assume there is state s' where (s) * (s). Two cases must be considered. In the first case, V (s) Vūr We will notate the optimal policy in M as t (i.e. V = V ) and the optimal policy in M as MM- We know there must exist a policy Thr such that v (s) = V # (s) from Lemma 3.2. We arrive at a contradiction because
Ms
V (s) > V (s), so The could not have been the optimal policy on M. The other direction follows symmetric argument. If
V (s), we know there must be a policy Ths in M such that ViT (s) = V (s). This implies T (s)
" (s)
M
i2 (s), meaning that 7 could not have been the optimal policy. Since Q-learning is known to converge to the optimal value function Watkins (1989), Q Q-learning on M will converge to the original value function Vūr-
15

---

Published as a conference paper at ICLR 2022
A.3 STATE ONLY LEARNING FROM STOCHASTIC MDPs
Here we will briefly prove that problem of deducing the optimal value function of an MDP M, from some state-only experience dataset D, cannot be solved in general for non-deterministic MDPs. We notate as the space of all complete state-only datasets, and as the space of all value functions. By complete state-only dataset, we mean datasets in which all possible transition triples (s, r) appear. Note that exactly one complete dataset exists for each MDP. We denote DM as the complete dataset corresponding to an MDM M.
In short, the idea is to construct a single state-only dataset which ambiguously could have been produced from two different MDPs which have different optimal value functions. Since no function deterministic function can model two outputs from the same input, no such function can exist.
Theorem A.I. For any function f D + V which maps from the set of of state-only datasets, D. a ser of value functions. There exists some MDP M, with corresponding dataset DM such that fD) # VM-
$ M ((DM) = Vм-
Proof. We proceed by contradiction, assume any D V such that
(Mf(PM) = VM-
If we can construct Mi and M2 such that VM, X V25, and D such that D DM,, = DM2 then f cannot produce the correct value function for f(D) in all cases, from the definition a function. We will now show very simple construction which exhibits this property. Consider MDP M1, with 3 states, 81: 8t, and actions 41 Q2 8, is terminal, and gives reward 1, other states give reward 0. The transition dynamics of the two actions are given below.
Table: a transitions for M
Table: transitions for M
$1
$2
$1
0
1
0
0
0
$2 0
0
0
For simplicity, assume M1 has discount factor of 0.9. This means V*(a1) V*222 1.
We define M2 as identical to M1, only with stochastic transitions. Table: @1 transitions for M2
Table: transitions for M₂
82
82
8;
$1
0
1
0
0.9
0.1
82 0
0
0
0
In this MDP, with discount factor of 0.9, V*(22) but V (s1) = 0.1 0.9 * 0.9 0.91.
Since both M1 and M2 share the same states, actions, and possible transitions, DM, = DM So we have satisfied our condition that D DM, DM.- and VM, V*M2. Thus, no can model the value functions for both MDPs in general.
16

---

Published as a conference paper at ICLR 2022
A.4 GRID WORLD BEHAVIOR PURITY
Figure S7 plots the MSE in value function and the proportion of states with correct implied actions function of the noise in the action labels.
Proportion of correct actions
0.020
1.0
0.9
MSE
0.010
0.005
0.7
0.000
0.84
0.86
0.88
0.90
0.92
0.94
0.96
0.98 1.00
0.84
0.86
0.88
0.90
0.92
0.94 0.96 0.98 1.00
State based purity
State based purity
Figure S7: We plot correctness of value function estimate and behavior as function of state-based purity of the intervening actions used for Q-learning. Left plot shows the mean squared error (lower better) between the obtained value function and the optimal value function. Right plot shows fraction of states in which the learned value function induces the optimal action (higher is better). Both value function and behaviors become worse as state- based purity decreases.
17

---

Published as a conference paper at ICLR 2022
A.5 LAQ IN STOCHASTIC MDPs
We have conducted experiments on stochastic MDPs in the gridworld environment. Specifically, we do this by adding sticky actions, such that with certain probability (called the stickiness parameter). the environment executes the previous action as opposed to the given action. We run LAQ with data from this stochastic environment and examine a) purity of latent actions, b) quality of learned value functions, and c) correctness of implied behavior.
1.00
purity
0.95
0.99
0.90
0.98
0.97
0.85
0.80
0.96
0.95
0.94
60
40
Stickiness parameter
Stickiness parameter
Average value states
1.04
0.850
0.825
1.00
0.800
0.775
Latent actions
0.96
Ground truth actions
60 Stickiness parameter
40
20
40
Stickiness parameter
Figure S8: As function of stochasticity in environment, we plot: state-based purity of the latent actions (top left), Spearman's 's rank correlation between LAQ learned value function and ground truth value function (top right), correctness of implied behavior by the LAQ learned value function (bottom right), and average state-value of LAQ learend value function, and the ground truth value function (bottom left).
As expected, Figure S8 (left) shows that the state-based purity of the learned latent actions falls off as the stochasticity (stickiness) increases. However, these impure latent actions have little effect the Spearman's rank correlation (top right). The Spearman's 's rank correlation does decrease but remains reasonably high even when the stickiness parameter is set 95%. While the ordering of the values of the different states doesn't change by much, we note increasing amount of over-estimation in the values (bottom right). This expected as with increasing stochasticity, there is an increasingly large gap between what LAQ thinks it can control, and what it can actually control. However, the behavior implied by LAQ is still always correct (bottom left), as the over-estimation uniform over all states. Thus, it is possible to get good performance from LAQ in some stochastic environments. However, is possible to also construct simple scenarios where LAQ, and for that matter any deterministic algorithm, will suffer as we describe in Section A.3,
18

---

Published as a conference paper at ICLR 2022
A.6 ENVIRONMENT DETAILS
2D Grid World. We use the environment and data from Section 3.2. We use the (x, y) coordinates our state for this environment. We use multi-layer perceptron for Jo and L2 loss for I.
2D Continuous Control Navigation. We use the Maze2I) data from D4RL. Fu et al. (2020). Ob- servation space here is (z, y, U2i vyy i.e. location and velocity the agent, and action space is force along and y directions. We use a frame skip of 3. We use a multi-layer perceptron for fo and L2 loss for In the embodiment transfer variant, we swap the agent for 4-legged ant, also from D4RL. The ant embodiment is far more challenging to control, with an 8-d action space and 29-d observation space.
Atari Game We work with Freeway. We generated our own data using the protocol described in Agarwal al. (2020). We turned off sticky actions and store frames both the default resolution (84 x 84) and full resolution (224 x 224). Other settings are same as is typical: stack last 4 frames to represent the observations, and use frame-skip of We use convolutional encoder-decoder model for fo and predict raw future observations. We use L2 distance in the pixel space as our loss function
3D Visual Navigation. We work with the branching environment from Chang et al. (2020) in the Al Habitat Simulator Savva et al. (2019), and use the provided dataset of first-person trajectories as our pre-recorded experience dataset. Following Chang et al. (2020), we employ low-level controllers for deriving behaviors from the learned value functions. We use a convolutional encoder-decoder for fo, and 1.2 loss in pixel space as I.
Kitchen Manipulation. In this task 9-DOF Franka arm can interact with several different objects physically simulated kitchen. The observations 24-d, containing the end-effector position and state of various objects in the environment (microwave door angle, kettle position, etc.). The action space represents the joint velocity. cross-embodiment experiments we replace the Franka arm with hook. The observation space remains the same while the action space becomes 3-d position control.
For experiments in kitchen manipulation, we utilize the GMM based solution described in Sec- tion A.12. For all environments, throw the provided action labels when learning our models.
19

---

Published as a conference paper at ICLR 2022
A.7 DATA COLLECTION
2D Grid World. The setting for this experiment is a 6 x 6 grid with 8 actions, corresponding to moving in the 4 cardinal directions and 4 diagonal directions. The agent starts in the top left (0.0), and gets reward () everywhere, except for in the bottom right, (5.5). Reaching the bottom right gives the agent reward and terminates the episode. In the starting square (0,0), the agent has probability 0.5 of moving right, and probability 0.5 of moving down. When the agent is on the top or bottom edge of the maze (row 0 or row =) moves right with probability 0.9, and takes random action with probability 0.1. When it is on the left or right edge of the grid (column = 0 or column = 5), moves down with probability 0.9 and takes a random action with probability 0.I. Otherwise, is in the interior of the grid, where it takes action away from the goal (randomly chosen from: up, left, or up-left) with probability 0.9, and one of the remaining actions otherwise (randomly chosen from: down, right, down-right up-right, down-left). This policy is rolled out for 20,000 policies to generate the data for methods described in main paper Section 3.2 and Figure 1.
Atari Game Data for the Freeway environment had to be additionally collected, native high resolution resolution images of the episodes from Agarwal et al. (2020) are not readily available. For this, re-generated the data using the protocal described in Agarwal al. (2020), without sticky actions. Then, the high resolution data was collected by taking the action sequences from the re-generated dataset and executing them in an emulator, while storing the native resolution images. Because we re- re-generated the data without sticky actions, the high resolution episodes are perfect recreations of the low resolution episodes.
Visual Navigation. The data used for the visual navigation experiment was generated using the same protocol as Chang al. (2020). Agents are tasked reach one of two goals (Gnear and G in visually realistic simulator Savva et al. (2019). Navigating to Gnear is the optimal path as it is nearer the agent's starting location.
The dataset contains types of trajectories, Ti, T2, and T3. representing 50%, 49.5%, and 0.5% of the trajectories in the dataset respectively. Ti takes sub-optimal path Gnears T3 takes the optimal path to Guear- and T2 navigates to Gfar-
2D Continuous Control (Maze2D) We use the Maze2D dataset from D4RL as is.
Kitchen Manipulation. The data used for the Kitchen Manipulation environment comes from the purtial version the D4RL FrankaKitchen dataset. To facilitate cross-embodiment transfer we convert the state representation to contain end-effector location instead of joint angles.
20

---

Published as a conference paper at ICLR 2022
A.8 ALGORITHM
Algorithm LAA
1: Given dataset D of (or, 0++1, triples for each epoch do for sampled batch (ot, 0,+1, r) ~ D do
2:
Latent Action Mining
3:
4:
L(o;, ot+1) & aen A ((fo(s, &), 8 -oo-VooLoo,, (
5:
Section 4.1
6: end for
7: end for
8: let 0(+1) arg min (folor, a), 01+1)
9: D ((0r, ot+1, ((0, r: 0411) (++,,) E D}
Latent Action Labeling
10: for each epoch do
11: for sampled batch (or, 0c+1, r, à) D do
Q-learning with Latent Actions
12:
Q-learning update to learn Q ss a) 13: end for
14: end for
15: V(8) max ((s, a)
21

---

Published as a conference paper at ICLR 2022
A.9 LATENT ACTION QUALITY
We first measure the effectiveness of our latent action mining process by judging the extent to which
the induced latent actions a refinement of the original actions. We measure this using the state- conditioned purity of the partition induced by the learned latent actions.
In given state, for any latent action a, there must be some ground truth action which most frequently mapped to à. We define the purity of â as the proportion of the most frequent action among all actions mapped to â. For example, in a given state if a set of actions [0, 10,0,00 0, 1, were mapped to latent action å, then O is the most frequent action mapped to â and thus the purity of à would be 0.6. For given state, the purity of an entire set of latent actions is the weighted mean of purity of individual latent actions. Overall purity is the average of all state wise purities weighted by how often the states appear in the dataset.
We extend this definition of cane-conditioned purity to continuous or nigh-dimensional states by measuring the validation accuracy of a function g that trained to map the nigh-dimensional state (or observation) and the associated latent action à to the actual ground truth action Training such a function induces an implicit partition of the state space. Learning to predict the ground truth action from the latent action á within this induced partition estimates the most frequent ground truth action, and accuracy measures its proportion, i.e. purity. This exact procedure reduces to the above definition for discrete state spaces, but also handles continuous and high-dimensional states well. For continuous action spaces, we measure the mean squared error in action prediction instead of classification accuracy.
Table S3 reports the purity (and MSE for continuous action environment) obtained by our proposed future prediction method. For reference, we also report the purity obtained when using single action that maps all samples into a single cluster, and 2 clustering methods that cluster either concatenation of the two observations i.e. [o++1; or the difference between the two observations i.e. [ot+1 -0g]- We use 8 latent actions for all environments except the FrankaKitchen environment for which we use 64 because of its richer underlying action space.
In general, our forward models are effective at generating a latent action space that is state- conditioned refinement of the original action space. This is indicated by the improvement in state- conditioned purity values over using a single action or naive clustering. For the 2D Grid World and 2D Continuous Navigation, clustering the correct space (state difference vs. state concatenation) works well as expected. But future prediction model, which directly predicts 8141 and doesn't 't use any domain specific choices, is able to outperform the corresponding clustering method. We also observe large improvements over all baselines for the challenging case of environments with high-dimensional state representations: Freeway and 3D Visual Navigation.
Table S3: We report the nateecoonditioned action purity (higher is better, MSE for continuous tion case where lower is better), of latent actions for different approaches: single action, clustering concatenated observations, clustering difference in observations, and the proposed future prediction models from Section 4. We note the utility of the future prediction model for the challenging case of Freeway and 3D Visual Navigation environments. See Section A.9 for a full discussion.
Enviromment
Observation Space
Action Space
Purity Metric
Single Action
Clustering [o.,0--]]
Clustering [dee1 0z]
Future
Prediction
2D Grid World
an location
Discreto,
Purity
0.827
0,851
1.000
0.998
Freeway
210 x 160 image
Discrete. 3
Purity
0.753
0.778
0.773
0.907
3D Visual Navigation (Branching)
224 x 224 image
Discrete, 3
Purity (1)
0.783
0.839
0.859
0.928
2D Continouu Control
zy loc. & vel State
Contimoous,
MSF (L)
2.207
2.188
0.325
0.905
Kitchen Manipulation
Continouus,
MSE (L)
0.015
0.015
0.015
0.014
22

---

Published as a conference paper at ICLR 2022
A.10 ANALYSIS OF LEARNED LATENT ACTIONS
We analyze the latent actions learned for the Freeway environment.
We visualize future prediction model's predictions for the Freeway environment in Figure S9. In line with our expectations, the one action future prediction model and the latent action future prediction model are both able reconstruct the background and the vehicles perfectly. At the same time, the one action future prediction model fails to reconstruct the agent accurately, whereas the latent action future prediction model is able reconstruct the agent almost perfectly. This provides evidence for the effectiveness of our proposed latent action mining approach at discovering pure action groundings.
Ground Truth 0z
Ground Truth 0ze1
With One Action
With Latent Actions
00
ACRVSSION
REBPSSEN
Figure 89: We visualize the Freeway future prediction models' reconstructions for o++1- From left to right, the ground truth o,, the ground truth 1 the reconstruction for 0:41 by the future prediction model with one action, the reconstruction for 0+ by the future prediction model with latent actions. The agent is circled each image.
Furthermore, we visualize the learned latent actions for Freeway Figure $10. In Free- way, the agent axis, and consequently, the environment action space only has three actions: move up, move down, and no-op. This means that the agent's 's y-displacement between the current frame and the previous frame directly corresponds to the ground truth action taken. In this visualization, we visualize the the chosen latent action (by color: blue, orange, or green) as a function of the agent's s y-position in the current frame (x- axis) and the y-displacement relative to the pre- vious frame (y-axis). Note that as mentioned the paper, we learn the value functions over the top three most dominant actions to stabilize training; for this reason, the visualization only consists of three latent actions.
0.94
only move along the y-
Change Agent p-position
0305
Relative Previous
-2.00
8.5
0.6
8.7
0.5 02
Agent y y-position
Figure S10: Visualization of the latent actions learned in Freeway. X-axis is the agent's y- position, y-axis is the displacement of the agent (which is indicative of ground truth action).
Within each vertical region marked by the red bars, we see that the latent actions are split into distinct clusters based the agent's y-displacement. Because the agent's y-displacement directly corresponds to the ground truth action taken, this indicates that given the agent's y-position, the learned latent actions encode information about the ground truth action. Hence, qualitatively confirm that the state-based purity of the latent actions is high.
23

---

Published as a conference paper at ICLR 2022
A.11 VALUE FUNCTION DETAILS
The numbers reported in Table are the 95th percentile Spear- man's correlation coefficients over the course of training. training is stable and converges, this corresponds to taking the final value, and in the case that training is not stable and diverges, this acts as a form of early stopping. We take the 95th percentile as opposed the maximum to eliminate outliers.
Value Function Model Selection
Below present visualizations of value functions learned from LAQ
Freeway. We visualize the value functions learned using our latent actions for Freeway. Figure S11 plots the values over the course an episode. In Freeway, the agent has move vertically up and down the screen to cross a busy freeway, receiving reward when it successfully gets to the other side. In single episode, the agent can cross the freeway multiple times; each time the agent makes it to the other side, the agent's location is reset to the original starting location, allowing the agent to attempt to cross the freeway once again. For this reason, we see the value increase as the agent gets closer to the other side of the road, and then drop soon its position resets to the starting location. As evident, the peaks of the learned value function correspond highly to the environment reward.
Figure S11:
We visualize the value predicted by our learned value function over the course of one episode of Freeway. The red points correspond to when the agent receives reward from the environment.
3D Visual Navigation. Figure $12 visualizes the learned value map in the 3D Visual Naviga- tion branching environment from Chang et al.
(2020). As depicted in Figure 6, there two goals: Gnear and G far- The figure on the right illustrates that the value function learned by uti- lizing our latent actions correctly assigns high value to the regions surrounding the two goal locations, and low value elsewhere. Addition- ally, we learn the value functions with DQN us- ing dataset obtained by using a sub-optimal policy which prefers to go to the goal further away rather than the goal close by. Despite this sub-optimality, the learned value function cor- rectly assigns higher value the nearby goal than the goal which is further away.
10
30
60
14
20
60
80
100
Figure S12: Visualization of the learned value
function for 3D Visual Navigation.
24

---

Published as a conference paper at ICLR 2022
2D Continuous Control. Figure S13 shows visualization of the value function learned us- ing latent actions for the 2D Continuous Con-
trol environment. While the observation space of this environment is (ær, y, Ma; fy) L.e. loca- tion and velocity of the agent, we produce this visualization over just the location of the agent. Based on just the location of the agent, the op- timal value function would be a monotonically decreasing function as the distance from the goal location increases. In this particular en- vironment, the goal of the agent is to get to the top-right corner of the maze. The visualization shows that the learned value function produces high values around this goal region at the top- right corner of the maze, and gradually lower values the farther away you go. This figure vi- sually is in line with our quantitative results in Table I which show that the value function learned using latent actions in the 2D Continuous Control environment highly correlates to that learned using ground truth actions.
5
0.8
1-
0.2
Figure S13: Visualization of the learned value function for 2D Continuous Control.
Kitchen Manipulation. We visualize the val- as a function of both the angle that the mi- crowave door makes with respect to its starting
1.15
Distance between End -effecttoo
and Microwave Door Handle
state as well as the distance between the end-
1.10
effector and the microwave door handle in Fig-
100
ure $14. The task here is to open microwave door: the agent receives binary reward when the angle that the microwave door makes with
1.00
respect its starting state (i.e. closed mi-
crowave door) above a threshold (approx. 0.7 radians), and zero otherwise. As expected, see that the predicted value of a state increases the angle of the microwave door increases. The end-effector position does not start out at the microwave door handle, but rather is ini- tialized a fixed distance away from the han- dle. a result, the agent must first move the end-effector to the microwave door handle, be- fore interacting with the door handle to open it. In addition to this, we hypothesize that as the distance between the end-effector and the mi- crowave door handle decreases, the predicted value should increase. While not as obvious as the relation between the microwave door angle and the value, we some indication that the distance between the end-effector and the microwave door handle decreases, the value increases.
0.90
0.35
Microwave Door Angle (radians)
Figure S14:
Visualization of the learned value function in Kitchen Manipulation. The x-axis the microwave angle in radians, the y-axis is the distance between the end-effector and the mi- crowave door handle, with the colors correspond- ing to the magnitude of the values.
25

---

Published as a conference paper at ICLR 2022
A.12 DEALING WITH EXTRAPOLATION ERROR
Given the reward structure utilized in densified reinforcement learning (see Section 4.3), if there exist states which produce spuriously high values of (s)), the optimal policy in the MDP may erroneously seek these states. Here we present two ways to combat this phenomenon: 1) Fit a density model to constrain high reward regions to the distribution used train V(s)
2) Re-R-paa Re-parameterize the value function such that out of distribution states are likely to have low value.
Density Model: We add an additional component the reward computation incentivize behavior to remain within the region of the state space covered by the demonstrations (where the learned value function in-distribution). For the kitchen manipulation environment we build a density model over the end effector position using 2 component Gaussian mixture model. If a state has a less than 1% probability according to the GMM density model, then we assign Vs)) O, for update described in Section 5.2. We apply this shaping all methods in Figure 4 for the kitchen environment. We give sparse reward the same benefit of this shaping by giving reward -1 outside of the GMM distribution, and 0 inside. Sparse task reward remains the same.
Value function re- re-parameterization For sparse reward tasks, we can parameterize the final value prediction - |] (l) Where é is neural network based featurization of the state. This ensures high values are only predicted when the featurization of the state close to the zero vector. States which are of distribution may induce random features, but these will result in lower predicted values.
Results the kitchen manipulation environment presented in Figure S15. We can that all methods solve the task poorly in this environment when value functions are unconstrained. Using density based model (GMM) or the "I-norm" parameterization of the value function allow the task to be solved consistently.
Kitchen Manipulation Unconstrained
Kitchen Manipulation
Manipulation "1-norm"
0.6
0.6
0.6
0.5
0.5
0.5
Success Rate
0.4
0.4
Success Rate
0.4
0.
Success
0.3
0.3
0.2
0.2
0.2
0.1
Environment Steps
Ervironment Steps
Environment Steps
Sparse Reward
Densified Reward (Ours + BCQ)
Densified Reward (TD(O))
Densified Reward (Ours)
Densified Reward (D3G)
We show learning curves for acquiring behavior using different methods for deal- ing with out distribution samples in the kitchen manipulation environment: unconstrained (left), density model based reward shaping (center), and "I-norm" parameterization of the value function (right). Results are averaged over 5 seeds and show 95% confidence intervals. See Section A. for details.
Figure S15:
26


Reviews:
[{"review_id": "yLzFkIjIkTA", "reviewer": "Reviewer_Fa3L", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found the problem setting to be both technically and conceptually interesting. In addition to its motivation, I found the paper well written and easy to read overall. Ideas are introduced carefully (with the small exception of some details in Lemma 3.2), and I believe this paper will be broadly accessible to the RL community. The two presented theoretical results are quite compelling in their own right, if quite natural in hindsight. Some of the claims made (the paragraph headers in the experimental section) seemed too strong given the data. For instance, in the paragraph \"LAQ value functions allow transfer across embodiments\", it is stated that \"LAQ-densified rewards functions, speed-up learning and consistently guide to higher reward solutions than sparse task rewards, or D3G\". Looking at Figure 5, the confidence intervals are large enough that I believe it would be difficult to definitively conclude that orange strictly outperforms purple. I don't believe the overlapping intervals detracts from the paper, but I do believe that the paper would be stronger if such claims were of appropriate strength given the data.\n\nI have no major concerns about the paper, and only one primary question. I include several writing suggestions below. I am not intimately familiar with state-only learning in RL, so unless this idea has already been examined elsewhere in the literature, I believe this paper is ready for publication (pending a few changes suggested below, and any other reservations that other reviewers might have).\n\n**Major Comments/Questions**.\n- [Q.1] In the discrete action case, it was unclear to me whether $n$ is known. That is, the number of actions in the true environmental model. This seems important for determining how to perform the latent action inference.\n- [C.1] Including the pseudocode for LAQ would benefit the paper a lot.\n\n**Minor Questions**\n- Does Lemma 3.2 quantify over stochastic policies, or only deterministic ones? Ah, in looking at the proof in the appendix, it seems that these are necessarily _stochastic policies_. It would be helpful to state this explicitly in the Lemma in the main text.\n\n**Writing Suggestions**\n- An extremely minor point, but I consider $(s,a,r,s')$ to be such a canonical ordering that the initial $(s_t, a_t, s_{t+1}, r_t)$ read oddly to me.\n- \"bellman equation\" --> \"Bellman equation\"\n- It would be helpful to define \"refinement\" formally. So, use a definition block and state it precisely.\n- I found the idea of a \"fundamental action\" to be quite sharp and added a lot of clarity to thinking about the proof. It is not strictly necessary, but if you found a way to make space, it might be nice to include a brief proof sketch for Lemma 3.2 in the main paper that highlights this idea.\n- \"induce same value in every state\" --> \"induce the same value in every state\n- \"in 2 ways.\" --> \"in two ways.\"\n- In the \"Experimental Setup\" paragraph, we move to using $o$ in place of $s$. I did not quite understand this move, and would suggest sticking with $s$.\n- \"in 5 representative environments\" --> \"in five representative environments\"\n\n(Appendix)\n- \"at at most\" --> \"at most\"\n- \"be a functions which\" --> \"be functions which\"\n- I did not quite understand this: \"we know a sum over action in each fundamental action\", but perhaps changing to the following will help: \"we know a sum over fundamental actions\"\n- A minor point, but many equations in the appendix do not have punctuation. I would recommend adding commas and periods to equations where appropriate.\n- In Proof of Theorem 3.1: \"First, is the case where\" --> \"In the first case,\"\n- This is just personal taste, but I prefer $\\neg \\exists$ notation to $\\not \\exists$ (I am referring to Theorem A.1)", "summary_of_the_paper": "This paper studies the offline reinforcement learning problem based only on trajectories without actions. That is, given a set of experiences of the form $(s_t, r_t, s_{t+1})$, how might a learning algorithm proceed? The paper takes care in motivating the problem, both from a conceptual perspective (we might not know the action interface of an agent), and technically (there is good reason we might have access to such trajectories of experience, and learning seems feasible). Given this setting, the paper's main focus is twofold: First, to establish a simple variant of Q-learning that is well-behaved under this learning problem. This motivates Theorem 3.1/Lemma 3.2, which in turn inform the design of Latent Action Q-Learning (LAQ). Then, the second focus of the paper is on exploring the behavior of LAQ in a variety of domains and settings. The experiments are spread across five distinct environments of different kinds (gridworlds, Atari's Freeway, and so on), and each experiment examines a different characteristic of learning. ", "summary_of_the_review": "This is a clean idea, presented carefully, with interesting theory and well conducted, insightful experiments. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "QMOtBWH-neT", "reviewer": "Reviewer_YoH4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths\n---------\n\n-   An interesting approach to the problem of learning from action-less\n    experience. The three part procedure of estimating latent actions\n    with EM, doing Q-learning with the learned actions and using the\n    value function on downstream tasks is promising and seems new.\n\n-   The theoretical arguments that underpin the proposed method are\n    mostly well thought out. There are some confusing parts, that I\n    detail below, but I do think the overall approach is correct and the\n    theory cleanly motivates the proposed algorithm.\n\nWeaknesses\n----------\n\n-   The significance of the contribution is unclear. This is in part due\n    to the obscure nature of the problem (of learning with action-less\n    experience), partly because the related work is not well\n    contextualized earlier in the paper and partly due to the\n    experimental evaluation.\n\n-   The experiments, while somewhat comprehensive, are not particularly\n    compelling. The results are mostly evaluated using correlation of\n    the learned value function and on downstream tasks, both of which\n    makes assessing the overall effectiveness difficult without\n    extensive ablation. It is not clear to me that correlation is a good\n    performance criterion in this case. Furthermore, the results on\n    downstream tasks are mostly inconclusive with overlapping confidence\n    intervals.\n\nDetailed Comments\n-----------------\n\n-   In Section 1 Paragraph 2, you motivate the method by stating that\n    the learned value function can be helpful in many downstream tasks,\n    including control. This is true, but the state-value function can be\n    learned without actions already. The paragraph does not communicate\n    the advantage of learning the action-value function with imputed\n    actions instead of the state-value function.\n\n-   Section 1, Paragraph 3 \"Thus, the central technical question is how\n    to learn a good value function from undirected observation streams.\"\n\n    Sentences such as the one above should specify \"action-value\"\n    function, because, as I mentioned above, a state-value function can\n    be directly learned. This is partially addressed in the rest of\n    paragraph 3 and in Section 3, but to say that Q-learning without\n    actions is policy evaluation seems incorrect. Q-learning cannot be\n    applied without actions, unless you impute the actions. A\n    state-value function can still be learned using TD learning, which\n    seems like the point that is being made in this paragraph.\n\n-   Section 3, Regarding the choice of 4x - ground truth. Is balancing\n    the actions in the larger set important? If one action is repeated\n    15 times and the other ground truth actions are only repeated 1\n    times, would that make the problem harder? Another more pressing\n    issue is if the actions are not a refinement, such as if only two\n    latent actions are permitted. This is more relevant in practice,\n    where we may not know the right action space apriori.\n\n-   \"For downstream decision making, we only care about\n\nthe relative ordering of state values.\"\n\nI don't understand this statement. For downstream tasks the relative\nordering of the action-values is the pertinent quantity, but I don't\nthink an ordering of state-values is meaningful. Since this is the\nprimary reason for using spearman's rank correlation, I do not see how\nmeasuring correlation would be a useful measure of success.\n\n-   How is correlation being measured exactly? And is this truly the\n    measure of success? If correlation is used to overcome some constant\n    bias in the state-values, then this implicitly assumes that the bias\n    is uniform across states which may not be the case. Learning\n    action-value functions offline leads to many issues in the quality\n    of the value predictor which may not manifest itself in the control\n    policy (such as divergent action-values for out-of-sample or\n    out-of-distribution actions).\n\n-   Ground truth actions are said to be an upper bound on the\n    performance of your method, but in Table 1 for Kitchen manipulation\n    the proposed method outperforms this upper bound. How is this to be\n    interpreted in light of the stability comments in footnote 2? Based\n    on footnote 2, it seems like the value function should be constant\n    and not change between runs (i.e. shouldn't the correlation should\n    be 1?).\n\n-   The results on downstream tasks would benefit from TD0 as a\n    baseline. While the value function learned is the value of the\n    behavior policy (and not the optimal value function), it could still\n    be used for all three settings (densified, embodiment transfer and\n    guiding low level controllers).\n\n-   Figure 4: while this does demonstrate improvement in the grid world\n    setting, there is no significant difference between the methods in\n    the other 2d navigation or freeway. Also, the difference in\n    asymptotic performance in gridworld is strange. Shouldn't all\n    methods eventually reach the optimal policy?\n\n-   Putting the related work at the end of the paper prevents\n    much-needed context for the proposed methods. The introduction does\n    a good job of stating what you are doing, but the question of why is\n    less clear.\n\nMinor Comments\n--------------\n\n-   Page 4 , Line 3: ammounts -\\> amounts", "summary_of_the_paper": "This paper analyzes the problem of learning from experience tuples that\nomit the action. The authors analyze tabular Q-learning and show that if\nthe action-space is a refinement, then Q-learning can learn the optimal\nvalue function. Motivated by this, an algorithm is proposed that labels\n$(s,s')$ tuples with a discrete action $\\hat a$, with Q-learning used\nwith these imputed actions. The proposed algorithm is shown to surpass\nseveral baselines on grid world, continuous control, freeway (atari),\nrobotic manipulation and rich visual navigation environments.", "summary_of_the_review": "\nThis is an interesting paper with good ideas, but is held back by an\noverall confusing experimental evaluation. The theoretical arguments\nthat motivate the proposed algorithm are well thought out. The empirical\nanalysis, however, does not give confidence that the proposed algorithm\nis an improvement in the action-less regime. I would like to see an\nenvironment that clearly shows that densified rewards with the proposed\nalgorithm is superior to densified rewards with a learned value function\nvia TD0. I also think that the correlation performance criterion needs\nto be better justified. I will currently leave my rating at below the\nacceptance threshold, but I am open to increasing my score if some of my\nconcerns are addressed.\n\nEdit: After discussion with the authors, I have increased my score to a 6.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3ezMn9C1PFm", "reviewer": "Reviewer_kJe7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Some comments on motivation, related work, structure, etc.:\n\nMotivation: I think the motivation is somewhat overstating the difficulty of the problem. Isn't it obvious that state-only often helps? I think there are enough common model examples where the action is even fully identifiable from just the current and next state (just think of a double integrator dynamics).\n\nI'm OK wuth havung the extensuve related work section at the end, but at least having a rough idea of existing work in this direction up front would be important, this is missing.\n\nI think the \"refinement\" is closely related to MDP homomorphisms and commutation of operators etc., but this seems entirely missing in the related work section.\n\nSome comments on definitions, problem formulation etc.:\n\nI'm a bit confused. There seem to be two issues: unobserved action, but \"same embodiment\", or unobserved actions and even allowing for different embodiments. It may well be that what is correctly addressed is the second case, but it would be helpful to at least discern the two things in the discussion.\n\nThe defi of \"refinement\" seems central so why put this in inline text, not as a Definition? And the inline formulation itself is very hard to parse, I think the second part of the sentence is somewhat incomplete. When continuing reading, it seems like \"refinement\" is formalized\n\nWhere is the value function V defined? This should be done explicitly in sec2.\n\nRegarding Theorem 1:\n\nIt would also help to motivate this better -- it seems that this theorem should motivate the subsequent algorithm. Personally, I would have expected a more principled result as the basic theorem in the sense of \"to what extent is the true MDP recoverable from just the given data. I think this is somehow implicated by theorem 1, but it may be worth making this more explicit and systematic.", "summary_of_the_paper": "They address undirected (i.e., the data was not necessarily generated by an optimal agent) offline RL in the case that not actions, but only state transitions plus rewards are observed. They present a theoretical result that motivates their subsequent method for finding the true Q function from just the given data. In experiments, they evaluate the method in common toy environments like gridworld.", "summary_of_the_review": "While the problem setting, i.e., state-only offline RL, is realistic and practically relevant (cheaper that classic RL), and the paper does contain some interesting ideas in terms of theory and algorithm, the paper is too premature. The writing sometimes lacks motivations and connections, and the theory, I think, can actually be extended (be made more systematic) already within this paper, this would be a deeper contribution.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2EUJ8at80GI", "reviewer": "Reviewer_cKHF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Main points:\n- I find the introduction to be well-written  and the motivation for this problem to be compelling. In particular, the idea that the same collected data could be used for different types of agents/embodiments is appealing to me, with potentially many practical use-cases.\n\n- For the theoretical result, I find that the conditions needed to satisfy theorem 3.1 may be too stringent. If I understand correctly, it amounts to assuming that the new action set $\\hat{A}$ contains all the actions of the original $A$, with possible duplicates. So, it does not allow other actions in $\\hat{A}$ that are different than those already in $A$, which is very restrictive. Essentially, it seems to say that we need to obtain exactly the same actions in $A$.\nI would find the theorem to be more meaningful if it allowed other actions in $\\hat{A}$ at least, with perhaps a different conclusion. Adding new actions to $\\hat{A}$ that are functionally different than those in $A$ may increase the optimal values in the general case, but maybe something more specific can be said under different conditions.\nAnother possible extension to the theorem would be to prove the necessity of the refinement assumption i.e. something like, if the optimal values in the two MDPs match, then one action space is a refinement of the other.\n\n- The experiments were well done with approppriate baselines being chosen and a reasonable choice of environments. The variety of different evaluations to assess the method was welcome. I appreciated the experiments which tried to directly assess the quality of the learned value functions instead of only indirectly doing so with overall performance numbers. \n- I liked the experiments with changing the embodiments of the agent. I haven't exactly this kind of experiment before and it seems to me that it could be applicable to certain practical scenarios. \n- Planning with low-level controllers was also interesting and did a good job showcasing the strengths of the proposed approach.\n\nQuestions and suggestions:\n- One aspect that confuses me is that the proposed algorithm is used to learn an action space but afterwards, this new action space is hardly used except to run Q-learning and obtain state-value functions. It seems like there should be an oppportunity to make more use of the learned action space. From an intuitive standpoint, it's not very clear to me why learning an action space is a good approach for action-less data. \n\n- Following up on the previous point, LAQ obtains a state-value function in the end, which can be used in various ways. LAQ also learns a one-step model as an intermediate step before doing Q-learning with the new actions. Have you considered trying to learn the state-value function directly as an alternate approach? This could be done using value iteration on the state-values with the learned model. \n\n- Did you try to interpret the learned actions at all? Do they seem to match the base actions or are they completely different?\n\n- Can you clarify what the clustering approaches are? I did not see any explanations about how they are used.\n\n- In Freeway, behaviour cloning seems to be performing the best. Why is that? Also, is there missing a green line extending from it for behaviour cloning + RL?\n\n- In table 1, how exactly is the Spearman correlation computed? Is it calculated using state-value functions on a batch of data? Or is it using state-action value functions? How is the batch of data selected?\n\n- Fig 4 plots are not clear. Is there learning in the actual environment? If that is the case, are batch RL methods used then? \nAre they first used to initialize the method and then continue learning?\n\n- In section 5, the term \"state-conditioned purity\" is introduced but then it isn't explained in the later sections (only in the appendix). I suggest clarifying this in the main text or moving that entire discussion to the appendix.\n\n- The (theoretical) limitation of LAQ to deterministic MDPs is quite strong. While there's a small discussion included in the text, have you done any experiments trying it out in stochastic MDPs? Perhaps it can do reasonably well anyway.\n\n- I would clarify that the word \"refinement\" in this paper is defined to be something specific and doesn't have a standard meaning. This isn't clear right now in the abstract or the introduction. I was confused since \"refinement\" is related to a set being a subset of another in mathematics.", "summary_of_the_paper": "This paper focuses on the problem of learning from an offline batch of data containing only states and rewards (no actions). A theoretical result shows when we can expect to recover the true optimal value function when using a different action space. This motivates the algorithm Latent Action Q-learning (LAQ), which first learns a discrete action space by modeling one-step transitions and then applies Q-learning with the learned action space. Experiments in a variety of settings, including reward-shaping and planning with low-level controllers, validate the effectiveness of the approach. ", "summary_of_the_review": "The paper tackles an interesting topic and proposes a simple approach which performs well empirically. I don't think the theoretical result is very strong, but the paper focuses more on the empirical aspects so it's not a major concern. Also, the paper is generally well-written although there are certain areas that require more clarification, which is easily addressed.\nOverall, given the thorough experiments and the interesting problem, I'm leaning towards acceptance.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method for learning state value functions from (s,s',r) tuples, founded on the theoretical analysis in MDP setting. The extensive evaluation in several environments shows the benefit of the algorithm.

The consensus among the reviewers, and I concur, that the paper proposes an interesting and novel method. It is cleanly presented, and well-founded. The evaluations across range of environments, including robot manipulation validate the method.

During the rebuttal, the authors provided additional evaluation, added a discussion on the latent MDPs, and made numerous clarification, addressing most / all reviewers' questions.
Decision: Accept (Poster)