Discrete Predictive Representation for Long-horizon Planning
Authors: Thanard Kurutach
Julia Peng
Yang Gao
Stuart Russell
Pieter Abbeel
Venue:  (2021)
Keywords: Discrete Representation
Learning and Planning
Model-based RL
Hierarchical RL

Abstract: Discrete representations have been key in enabling robots to plan at more abstract levels and solve temporally-extended tasks more efficiently for decades. However, they typically require expert specifications. On the other hand, deep reinforcement learning aims to learn to solve tasks end-to-end, but struggles with long-horizon tasks. In this work, we propose Discrete Object-factorized Representation Planning (DORP), which learns temporally-abstracted discrete representations from exploratory video data in an unsupervised fashion via a mutual information maximization objective. DORP plans a sequence of abstract states for a low-level model-predictive controller to follow. In our experiments, we show that DORP robustly solves unseen long-horizon tasks. Interestingly, it discovers independent representations per object and binary properties such as a key-and-door. 


Under review as a conference paper at ICLR 2021
DISCRETE PREDICTIVE REPRESENTATION FOR LONG- HORIZON PLANNING
Anonymous authors Paper under double-blind review
ABSTRACT
Discrete representations have been key in enabling robots to plan more abstract levels and solve temporally-extende tasks more efficiently for decades. However, they typically require expert specifications. On the other hand, deep reinforcement learning aims to learn to solve tasks end-to-end, but struggles with long- horizoz tasks. In this work, we propose Discrete Object-factorized Representation Plan- ning (DORP), which learns emporally-absttrcce exploratory video data in an unsupervised fashion via a mutual information max- imization objective. DORP plans a sequence of abstract states for a low-level prede--rredictve controller to follow. In our experiments, we show that DORP robustly solves unseen long- long-horizon tasks. Interestingly, it discovers independent representations per object and binary properties such as a key-and-door.
discrete representations from
INTRODUCTION
In future, we hope that robots will be able to operate in unstructured environments such as homes and hospitals, and endowed with long-horizon planning ability. Despite successes in deep reinforcement learning (RL) from raw observations, much progress relies on the availability of shaped reward to guide the learning (Ng et al., 1999; Mirza et al., 2020). On the other hand, over past decades, task and motion planning has been shown to solve much longer-horizon goal-directed tasks such as making cup of coffee from torque control (Kaelbling & Lozano-Pérez, 2011; Srivastava al., 2014: Toussaint, 2015; Wang et al., 2018). However, these methods often require pre-specified discrete abstract states, task representations and transition models, e.g., whether the robot is holding cup and what actions (or perturbations) change such abstract state. In this paper, we aim to learn discrete representations for high-level abstract planning from video interaction data, combined with learned short-horizon controller.
Learning discrete representations from unsupervised data for planning challenging for two rea- sons. First, the relationship between the optimization objective and the true task objective is not well-defined. Second, optimizing a model with discrete layer is difficult with standard deep learn- ing techniques. Recent methods approach the first problem using reconstruction constrastive objectives (Watter et al., 2015; Anand et al., 2019; Ha Schmidhuber, 2018; Kurutach et al., 2018; Hafner al., 2019; Srinivas et al., 2020); however, the learned representations in continuous latent space which is unstructured and difficult to combine with high-level abstract planning. While other methods show promise in learning discrete representations, they have been applied to temporally-extended RL tasks (Oord et al., 2018; Razavi et al., 2019; Risi & Stanley, 2019; Asai & Fukunaga, 2017; Stratos & Wiseman, 2020).
In this work, we propose Discrete Object-factorized Representations for Planning (DORP) a novel framework for visual planning and control by learning discrete representations and low-level con- troller. DORP learns discrete representations from images that change slowly overtime, such as whether or not the agent holds a key or which room the agent in, along with low-level predictive model for control. These slow features enable the agent to plan at low frequency in longer-horizon tasks. More specifically, DORP represents an abstract state as a set of one-hot vectors, and optimizes its encoder by maximizing a mutual information lower bound between the current representations to future observations (Oord et al., 2018). In order to train through the discrete layer, we apply the Numbel-SSfftmax reparametrization trick (Jang et al., 2016; Maddison et al., 2016). Using abstract states as nodes, we build an approximate feasibility graph based on observed transition data. When provided with new start and goal images, the agent plan the shortest abstract path. Using the next

---

Under review as a conference paper at ICLR 2021
01000
Goal Encoding
Encoder
10000
Current Encoding
Red Aarni
00001
Key
Yellw
(a) Unsupervisedly encode image with one-hot encodings
(b) Plan the discrete encoding space with graph search
Figure 1: Learning discrete representions for planning. We approach the long-horizon planning on high dimensional image space by learning discrete representation and planning the discrete space. (a) We learn multiple one-hot encodings for each observation fully unsupervisedly with contrastive learning. Each one-hot encoding corresponds freely moving entity in the observation, such as a room the agent is or whether a key has been picked up. (b) With the discrete encoding, we build a graph that connects the current encoding to the goal encoding and do a graph search on it to plan efficiently for long horizon tasks.
temporally abstract state of one of the
abstraction as waypoint, model-prediccive control maximizes the objective that is it reaches the target abstraction and otherwise with trained video prediction model. Unlike other subgoal plan- ning work (Savinov et al., 2018; Nasiriany et al., 2019; Laskin et al., 2020; Liu et al., 2020), by following abstract waypoints, DORP avoids unneccessary steps match exact waypoint states.
In set of experiments, we demonstrate that DORP learns temporally-consisee and object- factorized representations suitable for planning. We show that these representations enable DORP to handle unseen long-horizon tasks more successfully compared to the ofates-of-the-arts visual planning. Interestingly, we observe that latent representations show object-level factorization such as key-and-door.
2 PRELIMINARIES
We present background material on unsupervised representation learning and discrete optimization that method builds on, and problem formulation.
Contrastive Predictive Coding (CPC) (Oord et al., 2018) learns low-dimensional representations that most predictive of the future future high-dimensional sequential data. A non-linear coder go O vector representation 24- Let's define similarity score fa (2,, ot+k)
- TR! parametrized by 0, encodes the observation or E O to a latent I-dimensional (op(2
where db is a trainable (-by-l similarity matrix and Ot+ is a future observation & steps ahead of or, Given the query observation Ot, we aim classify the key observations Or4-k as positive and other sample from the dataset as negative. Formally, we optimize the loss function Lcrc
-o. log fe(2t, 0c+x) - log Σo,ex fu(z),o;) with respect to and vh. This also corre- sponds to maximizing lowerbound of the mutual information between the latent representation 2, and the future observation O++
Gumbel-Softmax (GS) (Jang al., 2016) discrete optimization technique compute the gra- dient of through samples from categorical distribution π; GS first applies reparametrization trick rewrite samples z; onehot(arg max, naaxx(a (9j logn;)) where 9j i.i.d samples from Gumbel(O, 1) (Gumbel, 1935). GS approximates arg max with softmax making the discrete stochastic layer differentiable, i.c., 2; softmax((g)
log mT)/T). When the temperature parameter 7 approaches 0, the samples converge to the true categorical distribution. Empirically, the temperature starts high and annealed closer by certain schedule.
2.1 PROBLEM STATEMENT: GOAL-DIRECTED VISUAL PLANNING AND CONTROL
We define an unknown, fully-observable, stochastic dynamical system f which maps input obser- vation Or E O and action a, E A to the next observation 0++1- Under this dynamical system, we assume simple exploration policy Trand which can collect data that characterizes the dynamics of the system. This is known as self-supervised data or play data (Agrawal et al., 2016; Lynch et al., 2020). We consider high-dimensional observation such images.
2

---

Under review as a conference paper at ICLR 2021
Zt Zi
Extractor
Enc
Ze
Same for every channel
Gumbel Softmax
One-Hot Encodings
Flattened Encading
Tied Weight
2 LZzz+
Or+k
Extractor
Enc
Z
Learnable DDDP Weights
Same for every channel
Softmax
Continuous
Encodings
Flattened
Encoding
Figure 2: Unsupervised discrete code learning. We propose to learn a set of one-hot encodings as the latent representation for each observation. For anchor observation Ot, its neighbours Oc+A for some small & are treated as positives, and random other observations are negatives. We propose a learnable weight matrix to be diagonally dominant and diagonally positive (DDDP). An object extractor architecture is applied to learn to extract objects and shared encoder is applied per chan- nel. Finally, Gumbel softmax and softmax are used to maximally allow gradient to flow back to the encoder.
Our goal is to learn discrete abstraction 2, E given observation or, an abstract feasibility model, and low-level local controller. At test time, given start o, and goal 0g observations, a sequence of abstract states 2g, 21: 2g where 2g and 2g are the representation of og and Og. and apply the low-level controller reach these abstract waypoints and finally to 0g.
plan
DISCRETE OBJECT-FACTORIZED REPRESENTATION FOR PLANNING
Our objective is to derive representation properties such fbctortfactorizzatio and temporal- consistency from an unsupervised learning objective to facilitate long-horiae planning. When object representations figurations and can quickly generalize to unseen tasks. Also, the connectivity graph can also be memory-efficiently represented and combined with more powerful planning algorithm. Another important property for representation is temporal consistency, meaning any two state observations the same abstraction should be reachable from mnothe by a short sequence of actions. When this property holds in the latent space, a high-level plan can be successfully executed by a low-level controller.
factorized, the agent escape the need of combinatorial data
We propose Discrete Object-factoriza Representation for Planning (DORP) to tackle goal-directed visual planning and control problem. DORP learns latent discrete representations, connectivity graph for abstract planning, and a low-level model-predictive controller. We describe each subcom- ponent in Section 3.3.
3.1 DISCRETE REPRESENTATIONS
Despite successes in applying its unsupervisedly learned representations for downstream tasks, CPC not endowed with inductive bias for its representations facilitate planning (Oord al., 2018). Additionally, it is unclear how a discrete representations can be extracted and optimized. Toward this goal, we propose object-factoriee architecture with asymmetric Gumbel softmax for discrete optimization, and a diagonally dominating and diagonally positive similarity matrix to encourage temporal consistency.
We implement our DORP architecture to encourage object- fpicctffattorzzation and capture discrete rep- resentations. The extractor passes an anchor observation through convolutional neural network and outputs a c-channel feature map. Each feature map is inputted to a shared encoder followed by Gumbel softmax. For positive and negative pair of images, however, we opt for continuous embedding by swapping Gumbel softmax for softmax. Empirically, without this asymmetry trick, the optimization tends to converge to poor local optima. Together these one-hots are flattened into long vectors, and their bilinear product is the similarity score between the query-key pair (see Figure 2).

---

Under review as a conference paper at ICLR 2021
We propose an inductive bias in the similarity score function to encourage tem- poral consistency. Instead of fully trainable matrix, we parameterize the simi- larity matrix v to be di- agonally dominant and di- agonally positive (DDDP). This property biases the similarity score to be high when the key embedding Žt+ close to the query embedding
Visual MPC
Current
Goal
zi
x
Repeat for every one-hot encoding z
Figure 3: DORP Planning. Since the learned representation includes multiple one-hot encodings, we plan for each one-hot encoding one time. For each of the one-hot encodings 2², we build a graph based on that subset of the representation. After finding the shortest path from current state the goal state, we use MPC to reach the next planned state. See Section 3.2 for more explanations.
and maxi- mized when 21+ = 2t- In other words, this incen- tivizes the representations of positive pairs share many one-hots as possi- ble. We reparameterize the weight matrix as ex (-cr())))) + alıxt) where lixt is an identity matrix, a is positive constant, a sigmoid function, 1/ a trainable scalar, and 1/1 a trainable 1-by-l matrix.
3.2 ABSTRACT PLANNING
critical component in planning is a connectivity graph that decides whether two discrete rep- resentations are connected. One naive solution to build such graph from data all observed representations as nodes and all observed transitions as edges. However as the number of indepen- dent entities increases, the amount of data required generate the nodes and edges order cover the state space increases exponentially. To solve this, we propose to reduce the planning problem by exploiting factorization (see Figure 3).
1. First, embed the current image o, and goal 0g 2, qo (o.) and 2y go go
2. Pick random one-hot index s.t. the current state's one-hot 2 differs from the goal's 's one-hot z/g-
Build a graph based on how this one-hot transits in the data (ignoring all other one-hots).
4. Plan path zf ..., 2i from 2 to ag. If this does not exists, then we fail the task.
Execute the low-level controller following the next target =i. h 1,..., ... while keeping other
2 the same.
6. If it doesn't it reach z remove the edge from the graph and redo the planning in step 4.
7. If it succeeds, follow the next target until it reaches z/-
8. Repeat step 2 until all the one-hots match.
Finally, execute the low-level controller to the goal image 0g.
Our proposed method assumes that each latent one-hot can be manipulated independently solve the task. This assumption can hold various settings. For example, imagine a task that requires reorganizing furniture in spacious room, and each objects is represented as separate one-hot. Here, we can manipulate each object separately, and since there is plenty of empty space in this scenario, we can easily plan around objects avoid collisions.
This assumption does not always hold; along the same scenario, when the space is more cluttered, objects more likely collide, and the proposed fully-factorized planning may fail even though a solution exists (incomplete). We extend this planning algorithm by building a graph on random set of one-hot indices at time in step 2. When can choose this set contain all the one-
implicit assumption of temporal consistency in the learned discrete representations required.
4

---

Under review as a conference paper at ICLR 2021
hots the method is equivalent to the full graph search (complete but expensive). Thus, our extended planning method can trade between completeness and efficiency.
3.3 LOW-LEVEL CONTROL
The low-level controller is the key component in order to follow abstract plans and reach the goal. For this reason, the controller must be flexible in achieving multi-objectives. Model- predictive controller (MPC) achieves this by modeling the dynamics of the world from past data and minimizing with the total predicted cost of horizon T at run-time: at,,,,, G = arg min -6-
The predicted cost is computed by applying known cost function on the predicted outcome of T -ttep action sequence from the current observation or.
E
MPC requires the world model and the costs. For the world model, we implement stochastic video prediction model (Babaeizadeh et al., 2017). In our framework, we define two cost functions for reaching abstract goals and for reaching the final goal observation. The first cost function is defined to be if the current embedding exactly matches the goal embedding (all the one-hots match) and otherwise: (0 z²) [0g) == z/1. The second cost function is defined by its L2 loss in the
observation space to the goal: Cuts 0g) = ||o ogll- This cost is versatile for reaching nearby
observations particularly to reach the goal observation once it has reached the goal code.
RELATED WORK
Object Discovery Recent work has studied unsupervised object segmentations from visual inputs (Burgess et al., 2019; Greff et al., 2019), and entity-factorized representations and models for pre- dictions (Kipf et al., 2019). However, these studies have not been demonstrated to directly solve the task. In this work, take a step further to evaluate the representations in downstream tasks. While other work (Veerapaneni et al., 2020; Watters et al., 2019) shows how the MPC agents benefit from Abject-factorize
models, they require shaped rewards for the tasks. In contrast, we consider a long- horizon task in which the agent only receives its reward when it has reached the goal. A large body of work in computer vision has studied unsupervised video object segmentation (Aloimonos al., 1988; Bajesy, 1988; Bajesy et al., 2018; Gibson, 2014; Fitzpatrick, 2003; Hausman et al., 2015; Kenney et al., 2009) and unsupervised object detection (Kwak et al., 2015; Wang et al., 2019; Xiao & Jae Lee, 2016). However, semantic segmentation and object detection might not be the correct representation for the end-task. In this work, we learn representations that are more ready to use for planning without explicit segmentation or detection.
World models Much previous work has applied generative models of the world to visual control tasks (Watter et al., 2015; Ha & Schmidhuber, 2018; Hafner et al., 2019). Other work (Ma et al., 2020; Okada & Taniguchi, 2020; Yan et al., 2020) leverages contrastive objective to learn a latent world model. These methods however require dense reward signals for most tasks. Visual foresight methods (Finn et al., 2016; Ebert et al., 2018) demonstrate impressive results on real robots using unshaped reward such as pixel distance to the goal, but it still remains limited to mhoot-o object pushing tasks. In our work, we borrow these methods for low-level controllers. Asai & Fukunaga (2017) learn discrete abstraction of the system such as 8-piece puzzle, but do not consider tempo- ral abstraction. Kansky et al. (2017) demonstrate that discrete object-factorize representations can be used learn based transitions. In combination with powerful planning, it improves gener- alization and data efficiency. However, it assumes supervised ground-truth labels for representation learning. In this work, we aim learn this in an unsupervised manner.
Hierarchial RL Recent work has tried to approach long-horizon visual planning by breaking down tasks using skills (Lynch et al., 2020; Kipf al., 2019). Our approach orthogonal and may deploy such action abstractions as a low-level controller. Other methods propose to plan subgoals and attempt to follow them either in the latent space (Kurutach et al., 2018: Nasiriany et al., 2019; Nair & Finn, 2019) or in the visual space (Savinov et al., 2018; Liu et al., 2020; Eysenbach et al., 2019). However, such methods require labor-intensive engineering to tune the threshold on when to move on to pursue the next subgoal (Savinov et al., 2018; 8; Liu et al., 2020) because the observations or the latent states can never exactly match. Imagine bringing a chair from one office to another. It would be time consuming to match all the positions and orientations of the chair along the way. Rather should care about rough area the chair has to go through in order to reach the goal. In our work, by learning the discrete codes, our method do not require such threshold as it exactly knows when
5

---

Under review as a conference paper at ICLR 2021
it has reached the target discrete representation. Additionally, it avoids aiming to match a specific waypoint observation. Instead of planning to the goal, Pertsch et al. (2020) predict intermediate images iteratively to construct subgoal tree. However in order to train such prediction model a long-sequential training data are required. Instead our work can be applied to short-horizon or long-horizon trajectories.
EXPERIMENTS
We perform experiments aimed at answering the following questions: (1) Are DORP representations comporally-consitaan for high-level planning? (3) Can DORP representations factorize objects that translate or change in appearances over time? (3) How does DORP compare to the visual planning SOTA in solving unseen long-horizon goal-directed tasks?
5.1 ENVIRONMENTS
We evaluate DORP on two main environments object-rearrangemmnt and key-room. All the ob- servations are provided color images of size 16 16 3.
--bbject-rarrrangement Multiple (k) 2-by-4 blocks of different colors can each be manipulated independently by an agent. The tasks require the agent manipulate these objects from a start configuration to a goal configuration. In each step, it can manipulate each block by one unit in one of the 4 directions. The agent collects data by randomly interacting with one of the objects at each timestep in purely exploratory manner, without any goal in mind. In this environment our trajectory has length one that the configuration randomly reset after every step.
key-room Two variations of key-room depend on the number of rooms key-corridor (6 rooms and a corridor) and key-wall (2 rooms. A key and agent are represented by pixel. A key placed in fixed location in room. If the agent steps the key door will be removed allowing the agent to enter a locked room. Each step the agent can move in one the four directions by unit. The agent collect data by long random walk 1500 steps for key-corridor and 1000 steps for key-wall. After each reset the agent will be placed randomly in one of the unlocked rooms. This environment -hoivviie--Boisvertt inspired by MiniGrid (Chevalier- Boisvert et al., 2018).
5.2 BASELINES
We choose the state-of siee-of-the aef-the-art arts visual foresight (VF) (Ebert et al., 2018) implemented using SV2P architecture (Babaeizadeh et al., 2017) our baseline for two reasons. First, VF only requires self- supervised data collection and is applicable to unseen tasks - thus aligning with goal-directed visual planning problems. Second, Visual Foresight is a low-level controller of DORP. By sharing the same video model as baseline, we show a direct improvement of abstract planning for temporally- extended tasks. Two optimization variations of VF VF with random shooting (VF-RS) and VF with cross- cross-entropy method (VF-CEM) both share the same video prediction model. Their objective is to minimize its L2 distance from the current image to the goal image. While DORP deploys VF- RS for its low-level control, VF-CEM comparision are provided for an improved baseline. VF-RS randomizes 1000 trajectories and take the full action sequence that achieves the minimum cost. VF-CEM randomizes 1000 samples per iteration. We pick the top 2% to refit distribution over sequence and repeat for iterations.
To understand DORP representation learning improvement, evaluate DORP when replacing its DDDP similarity matrix by an arbitrary weight (Oord et al., 2018) or an identity matrix (He et al., 2020) which have been used extensively in unsupervised representation learning. To understand the benefits of factorized planning, we replace it with the full graph planning with a maximum limit on the number of steps allowed.
5.3 QUANTITATIVE RESULTS
Temporal consistency We investigate DORP representations by color-coding the discrete embed- dings on the configuration map of the environments. For visualization simplicity choose 1-
With the exceptions for 5 objects in which we evaluate VF-RS on 10 tasks, VF-CEM on 11 tasks, and DORP 23 tasks due time and compute limits.

---

Under review as a conference paper at ICLR 2021
object
objects
objects
objjctt # steps
Algorithms
SR
steps
SR
# steps
SR
# steps
SR
DORP
1.00
15
1.00
49 222
1.00
75
0.87
1091
arbitary weight
0.92
26=19
0.58
52
0.26
52
identity weight
1.00
34
0,94
18
0,48
83
full graph
1.00
25
1.00
56+17
1.00
76+226
0.00
max
VF-RS
0.83
19
0.58
53 2:
0.25
93:124
0.00
max
VF-CEM
1.00
12
0.60
47 225
0.30
89
0.10
max
Table I: Comparison. We evaluate DORP in k-obiect-rerraangeenn tasks by measuring success rates across 50 sample tasks'. DORP is able to successfully solve most of the tasks as we increase the number of objects. We observe that other methods' performances degrade rapidly. We denote "max' when the agent has reached the limit on the number of steps without reaching the goal in 5-object-rearraggment this limit 1400.
object-rearanngement visualize the learned code map. In this environment have single one-hot code for representing the object. We demonstrate that DORP learns temporally consistent representations in which two states from the same discrete code are connected by short-horizon controller (see Figure 5(a)). Similarly, we observe the same property in key-wall environment (see Figure 6).
Object factorization We evaluate the behavior of the one-hot codes by changing one property of the agent at time such as positions or whether the agent has a key. Figure 4(c) we observe that by moving one object at time only at most one one-hot hot code changes its value --object-rearrangemeen In key-room, we formulate the discrete representation as follows: of the two output one-hot latents, the first is set to size zo and the other is set to size 2. Our aim is for one latent one-hot to encode the agent's ' abstract state and for the other binary latent to encode the key's 's abstract state. In Figure 6, demonstrate that the learned abstract representations are factorized as desired by observing (1) the larger one-hot changes when the agent moves with no key interactions and (2) the binary one-hot changes when removing and adding the key while maintaining the agent position.
Weight Ablation We study the effect of the proposed similarity matrix against commonly-used similarity matrices (Oord al., 2018; He al., 2020) in unsupervised representation learning. In Figure 5, we demonstrate that DORP is able to exploit more available latent code and therefore helping the latent space to be more temporal consistent.
5.4 QUANTITATIVE RESULTS
Long-horizon planning We test the agent by its ability to solve unseen goal-directed tasks. In oobject-rearrangemmnnt we can study the effect of the increase of the length and complexity of the tasks by increasing the number of objects. In Table I, we demonstrate that the number of objects increases DORP is able to succeed in most of the tasks. While other methods' performances degrade quickly. Note that when faced with 5 objects DORP employs the extended planning version by grouping the 5 one-hots into two groups of 2 and 3 one-hots. By considering more that one object time, the agent is able perform non-myopic planning and achieve 87% success rate. We also evaluate DORP on two variations of key-room. In our evaluations, we find that DORP solves key- wall with a single wall and door successfully in 9 of 10 tasks. When using ground-trutt dynamics model in place of an SV2P model, DORP solves Key-Room successfully in 37 of 40 tasks, and Key-Con cor-Corrd in 33 of 40 tasks.
CONCLUSION AND FUTURE DIRECTIONS
In this paper, we propose an unsupervised discrete representation learning method for long term planning, which can extract high level abstract states that are good for planning in an purely unsu- pervised manner. We demonstrate DORP's effectiveness over other methods on challenging long horizon tasks. We note that our method generate approximate plans more tractably and hence trades off optimality for efficiency, like other state abstraction works (Savinov al., 2018; Liu et al., 2020; Eysenbach et al., 2019). Those approximate solutions can be used to initialize a model-free policy which can be later fine-tuned to reach optimality. We leave this as our future work.

---

Under review as a conference paper at ICLR 2021
Start
(a) DORP Execution
Start
(b) VF Execution
(c) DORP One-hot Factorization
Goal
Figure 4: DORP in goal images which require temporal-extended planning. (b), when presenting the same task to VF-CEM we find that the objects are stuck in an awkward configuration where blocks (except the purple) are blocking to reach their goal positions. Finally, in (c), we visualize the representation factorization randomly moving others. We plot histogram of which one-hots have been changed per object. We find that with high probability only the one-hot that corresponds the moving object is modified.
In (a), present DORP with random unseen start and
of the five object while maintaining the positions of the
(6) DCOP matrix
260 No construnnt
idenity
Figure 5: Discrete Embedding Comparison. We visualize the color codes of different object posi- tions in The one-hot embedding has size 16 for all settings. In (a), find that DORP discrete representation temporally-consistr
per similarity matrix type. Each color shows different discrete code.
i.e., two states that map to the same embedding are connected by a short sequence of actions. In (b), when using arbitrary weight matrix the embedding is less temporally consistent. In (c), when using an identity matrix the, the embedding uses only out of 16 available codes
One hot
One hot
One hot
One hot2
Rupresertation
Chiel whihhg beddy" agent
(40 Crihhrrrurtee www.haatttttttttiaaaaaaa sidd Athooiig foong agent posttions
key
Figure 6: Key-Wall Representations. In (a) and (b), demonstrate the discrete code of the agent at different positions. Each color represents the same code. The grids black are invalid states (the wall that blocks the agent and separates the rooms). We demonstrate temporal consistency in the latent space both when the object has the key as not. In (c) and (d), we confirm that two one- hot codes are factorized by observing that only one-hot changes when removing the key while maintaining the agent position and only the other one-hot changes when the agent moves without interacting with the key.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Pulkit Agrawal, Ashvin V Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physics. In Advances in Neural Information Processing Systems, pp. 5074-5082, 2016.
John Aloimonos, Isaac Weiss, and Amit Bandyopadhyay. Active vision. International journal of computer vision, 1(4):333-335 356, 1988.
Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Marc-Alexandre Côté, and R Devon Hjelm. Unsupervised state representation learning in atari. In Advances in Neural Information Processing Systems, pp. 8769-8782, 2019.
Masataro Asai and Alex Fukunaga. Classical planning in deep latent space: Bridging the subsymbolic-symbolic boundary. arXiv preprint arXiv: 54,15,100 2017.
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy Campbell, and Sergey Levine. Stochastic variational video prediction. arXiv preprint 1 /252, 2017.
Ruzena Bajesy, Active perception. Proceedings of the IEEE, 76(8):966-1055, 1988.
Ruzena Bajesy, Yiannis Aloimonos, and John K Tsotsos. Revisiting active perception. Autonomous Robots, 42(2):177-196, 2018
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and represen- tation. arXiv preprint 1901 2019.
Maxime Chevalier- Chevalier-Boisven,, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. hitps /ttpttgttttt/tt///////////tttght com/ www.mecbwiiiaitcc/mmmmmcccccccet gym- minigrid, 2018.
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual fore- sight: Model-based deep reinforcement learning for vision- based robotic control. arXiv preprint arXiv: 00568, 2018.
Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement learning. Advances in Neural Information Processing Systems, pp. 15246-15251 2019.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In Advances neural information processing systems, pp. 64-72, 2016.
Paul Fitzpatrick. First contact: active vision approach segmentation. In Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003)(Cat. No. 03CH37453), volume 3, pp. 2161-2166. IEEE, 2003.
James J Gibson. The ecological approach visual perception: classic edition. Psychology Press, 2014.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. arXiv preprint
2019.
Emil Julius Gumbel. Les valeurs extrêmes des distributions statistiques. Annales de l'institul Henri Poincaré, volume 5, pp. 115-158, 1935.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint 1803.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pp. 2555 2565. PMLR, 2019.
Karol Hausman, Dejan Pangercic, Coltán-Csaber Márton, Ferenc Sehint-Benczédi, Christian Bersch, Megha Gupta, Gaurav Sukhatme, and Michael Beetz. Interactive segmentation of textured and textureless objects. In Handling Uncertainty and Networked Structure in Robot Control, pp. 237- 262. Springer, 2015.

---

Under review as a conference paper at ICLR 2021
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameteriization with gumbel-softmax. arXiv preprint arXiv:1611.01 144, 2016.
Leslic Pack Kaclbling and Tomás Lozano-Pérez Hicrarchical task and motion planning in the now. In 2011 IEEE International Conference on Robotics and Automation, pp. 1470-1477. IEEE, 2011 1.
Ken Kansky, Tom Silver, David A Mély, Mohamed Eldawy, Miguel
Xinghua Lou, Nimrod Dorfman, Szymon Sidor, Scott Phoenix, and Dilcep George. Schema networks: Zero-shot transfer with a generative causal model of intuitive physics. arXiv preprint 1rXiv:1706.04317,, 2017.
Jacqueline Kenney, Thomas Buckley, and Oliver Brock. Interactive segmentation for manipulation in unstructured environments. In 2009 IEEE International Conference on Robotics and Automa- fion, pp. 1377-1382. IEEE, 2009.
Thomas Kipf, Elise van der Pol, and Max Welling. Contrastive learning of structured world models. arXiv preprini arXiv: 19l1. 2019.
Thanard Kurutach, Aviv Tamar, Ge Yang, Stuart J Russell, and Pieter Abbeel. Learning plannable representations with causal infogan. In Advances in Neural Information Processing Systems, pp- 8733-8744, 2018
Suha Kwak, Minsu Cho, Ivan Laptev, Jean Ponce, and Cordelia Schmid. Unsupervised object discovery and tracking in video collections. In Proceedings of the IEFE international conference on computer vision, pp. 3173-3181, 2015.
Michael Laskin, Scott Emmons, Ajay Jain, Thanard Kurutach, Pieter Abbeel, and Deepak Pathak. Sparse graphical memory for robust planning. arXiv preprint arXiv:2003.0644 2020.
Kara Liu, Thanard Kurutach, Christine Tung, Pieter Abbeel, and Aviv Tamar. Hallucinative topo- logical memory for zero-shot visual planning. arXiv preprint arXiv:2002.12336, 2020.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on Robot Learning, pp. 1113- 1132, 2020.
Xiao Ma, Siwei Chen, David Hsu, and Wee Sun Lee. Contrastive variational model-based reinforce- ment learning for complex observations. arXiv preprint arXiv:2008.024300 2020.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00772 2016.
Mehdi Mirza, Andrew Jaegle, Jonathan J Hunt, Arthur Guez, Saran Tunyasuvunakool, Alistair Muldal, Théophane Weber, Peter Karkus, Sébastien Racanière, Lars Buesing, et al. Physi- cally embedded planning problems: New challenges for reinforcement learning. arXiv preprint 20X00:2009.05524, 2020.
Suraj Nair and Chelsea Finn. Hierarchical foresight: Self-supervised learning of long-horizon tasks via visual subgoal generation. arXiv preprint arXiv:1909.0582. 2019.
Soroush Nasiriany, Vitchyr Pong, Steven Lin, and Sergey Levine. Planning with goal-conditioned policies. In Advances in Neural Information Processing Systems, pp. 14843-14854, 2019.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application reward shaping. In ICML, volume 99, pp. 278-287, 1999.
Masashi Okada and Tadahiro Taniguchi. Dreaming: Model-based reinforcement learning by latent imagination without reconstruction. arXiv preprint 1wXxv:2007.14535, 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.0077 2018.
10

---

Under review as a conference paper at ICLR 2021
Karl Pertsch, Oleh Rybkin, Frederik Ebert, Chelsea Finn, Dinesh Jayaraman, and Sergey Levine. Long-horizon visual planning with goal-conditioned hierarchical predictors. arXiv preprint 1rXiv:2006.13225.. 2020.
Ali Razavi, Aaron den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In Advances in Neural Information Processing Systems, pp. 14866-14876, 2019.
Sebastian Risi and Kenneth O Stanley. Deep neuroevolution of recurrent and discrete world models. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 456-462, 2019.
Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi-parametric topological memory for navigation. arXiv preprint arXiv: 1803.000000065 2018.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning, arXiv preprint arXiv:2004.04136, 2020.
Siddharth Srivastava, Eugene Fang. Lorenzo Riano, Rohan Chitnis, Stuart Russell, and Pieter Abbeel. Combined task and motion planning through an extensible planner-independen interface layer. In 2014 IEEE international conference on robotics and automation (ICRA), pp. 639-646. IEFE, 2014.
Karl Stratos and Sam Wiseman. Learning discrete structured representations by adversarially max- imizing mutual information. arXiv preprint arXiv:2004.03991, 2020.
Mare Toussaint. Logic-geometric programming: An optimization-basee approach combined task and motion planning. In ICAA,, pp. 1930-1936, 2015.
Rishi Veerapaneni, John D Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua Tenenbaum, and Sergey Levine. Entity abstraction in visual model-based reinforcement learning. In Conference on Robot Learning, pp. 1439-1456. PMLR, 2020.
Ning Wang, Yibing Song, Chao Ma, Wengang Zhou, Wei Liu, and Houqiang Li. Unsupervised deep tracking. Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 1308 1317, 2019.
Zi Wang, Caelan Reed Garrett, Leslie Pack Kaelbling, and Tomas Lozano-Perez. Active model learning and diverse action sampling for task and motion planning. In International Conference on Intelligent Robots and Systems (IROS), 2018. URL http: http:///iss csail mit. edu/ edu/pubs/
wang-irosls pdf.
Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In Advances neural information processing systems, pp. 2746-2754, 2015.
Nicholas Watters, Loic Mathey, Matko Bosnjak, Christopher P Burgess, and Alexander Lerchner. Cobra: Data-efficient model-based rl through unsupervised object discovery and curiosity-driven exploration. arXiv preprint 1rXiv:1905.09277 2019.
Fanyi Xiao and Yong Jae Lee. Track and segment: An iterative unsupervised approach for video object proposals. In Proceedings of the IEEE conference on computer vision and pultern recog- nition, pp. 933-942, 2016.
Wilson Yan, Ashwin Vangipuram, Pieter Abbeel, and Lerrel Pinto. Learning predictive represen- tations for deformable objects using contrastive estimation. arXiv preprint 2rXiv:2003.05436 2020.
11


Reviews:
[{"review_id": "dh5xg3bq8RK", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method that combines learning discrete representations together with planning using graph search to solve long horizon tasks from vision. The approach works by generating data via random exploration and trains a representation encoder on this data. This network extracts objects from the observations and passes them through a CNN and shared encoder to generate one-hot encodings; these encodings are concatenated to generate the discrete representation. The encoder is trained via a contrastive learning objective with a similarity matrix that encourages nearby states to share similar encodings, thereby encouraging spatial and temporal abstractions. Next, these representations are combined together with an abstract planner to generate a sequence of waypoints to the goal. This is done by creating a graph of transitions from the collected exploratory data and search within this graph — this is executed for each encoding at a time with the assumption that the task can be solved by moving each object independently of the others. Finally, a low-level controller is used for reaching the waypoints and final goal, this is done via MPC on an action-conditional predictive model that generates future observations given the current state and goal. The approach is tested on two simple planar planning tasks where the agent has to solve k-object arrangement and open a room with a key respectively.\n\nPros:\n1. This paper tackles an important problem of representation learning for long-horizon planning tasks. It proposes a standard bi-level architecture of a planner and low level controller but it nicely brings together different ideas from prior work such as the visual foresight based low level controller and graph search based high level planner.\n2. A few novel points include the use of the Gumbel Softmax for learning discrete one-hot representations, the use of a DDDP matrix to encourage spatial and temporal similarity of learned encodings, and the factorised planning schema that takes advantage of the one-hot encodings to decompose the planning problem into tractable sub problems.\n3. The presented results show that the approach scales well as the number of objects in the scene are increased. Even for rearrangement of 5 objects the method is able to get a performance close to 90% compared to baseline methods.\n\nCons:\nWhile the proposed approach is well motivated there a few design choices that severely limit its applicability and the evaluations, baselines and results are a bit weak.\n1. The approach is only tested on two visually simple 2D scenes where the task of learning a discrete object-centric representation is quite straightforward. This makes it hard to evaluate the strength and generalisation of the proposed approach — how well would the approach generalise to scenes that are more complex? It would be useful to have a discussion in the paper in this regard.\n2. Looking at Fig 2., the encoder uses an “Extractor” that seems to generate bounding boxes and produce features of these boxes. What is this extractor and how is it trained? No details are provided regarding this network. If this is a network that is already trained to crop objects in the scene then the job for the encoder is quite straightforward — it just has to learn an object classifier which should be trivial to do in the current setting where the scenes are quite simple visually. If this is the case then it is quite obvious that the network would learn object-centric representations.\n3. The presented baselines use the low-level controller to reach the final goal with a cost that is the error in image space. This is a very weak baseline (especially due to the cost function) that is bound to get stuck in local optima for the presented problems. An alternative baseline could be a method that uses a latent space error term similar to the proposed approach but with a dense latent that is not a one-hot vector. Such a latent could also be trained via a contrastive loss and has a better chance at succeeding on the proposed task — thereby forming a stronger baseline. It can also better elucidate the strength of a discrete representation for abstract planning. \n4. The success of the proposed approach is highly dependent on the availability of good exploration data. It is not clear how this data is collected currently (a random walk is mentioned but more details would be useful). Also, how much data is collected? Is there an analysis on how well the approach does with varying amounts of data use? Similarly, how are the different networks trained? It would be great if more details are provided.\n5. As mentioned in the paper, the method assumes that the planning problem can be factored into sub-problems where the task is to just move individual objects in the scene. While an alternative formulation of randomly sampling “k” one-hots and planning in this graph is provided this is still a fundamental assumption that limits the applicability of the proposed approach.\n\nOther comments:\n1. In Fig 5.a) there seems to be some wraparound between the different discrete codes. Why does this occur?\n2. The planning algorithms use a very large number of samples/trajectories for these tasks. Is this needed?\n\nOverall, the approach combines several interesting ideas but needs better baselines and evaluations. More details also need to be provided regarding the training, network architectures etc. I would suggest a weak reject.", "extra_scores": {}, "extra_text": {}}, {"review_id": "YuMZm28qARt", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work presents Discrete Object-factorized Representation Planning (DORP), which learns a discrete representation from videos with an enforced temporal consistency. This representation can then be planned over through a sequence of small alterations to the discrete embedding, which are then executed via MPC. DORB is demonstrated to solve long-horizon tasks and learn representations that consider objects and their properties. \n\nThe paper is clear, though there are several typos on the paper, including the two words of the intro (“In the future”). The paper should be proofread carefully. The work is interesting, particularly the efficient search and graphical representation of a state space to plan high-dimensional and long-horizon problems. Such approaches are quite promising and the approach is well founded. The figures and demonstrations qualitatively demonstrate the learned representation’s properties nicely.\n\nThe planning process proceeds by altering the goal state one-hot’s a single index at a time until the initial state is reached. A few aspects and consequences of this planning are not clear:\n- How does one know if a path does not exist at step 4? \n- How often are one-hot’s not represented in the data? Can density of the latent discrete embeddings be encouraged in any way or does this naturally arise?\n- “We extend this planning algorithm by building a graph on a random set of one-hot indices at a time in step 2.” This should be written in the algorithm description for clarity. \n\nMy main issue is the experiments, in terms of (1) motivating the approach, (2) baseline comparison, and (3) difficulty. \n1) The primary contribution is the framework for learning this discrete embedding and planning over it, but the need for a discrete embedding is never fully considered. This should be ablated over with a continuous latent space, as has been used previously e.g, in https://arxiv.org/abs/1807.10366, Watter 2015, Hafner 2019 and https://arxiv.org/pdf/1912.01603.pdf and related works.\n2) The results should compared to methods such as Eysenbach 2019 and Savinov 2018 as these use similar representations, which are discrete via their use of actual datapoints. \n2) The examples are nice to specifically show properties of the learned space and visualize them, but the problems are not particularly realistic. How would DORP work with a more continuous task like a robot pick and place with multiple objects for example?\n\nMinor notes:\n- Why is the 1 object result for VF-CEM not bolded in Table 1? It seems most efficient in steps and has a 1.00 SR.\n- A video would be helpful of both learning the space and planning.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "XF_kH80aGX", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the problem of long horizon visual planning, with the aim of of being able to plan actions to reach distant goals. This is a well studied problem, and like prior work this method considers the setting where the agent is given an offline dataset of interaction, which it learns from to be able to reach new goals (specified by a goal image). The method first does unsupervised representation learning, where it learns a discrete representation of images (using contrastive predictive coding (CPC) with a discrete latent variable using Gumbel Softmax). Using a set of discrete latent variable, it then builds a set of graphs which connects these discrete states based on the collected experience, and derives a planning procedure to find a path in the graph which reaches the goal. \n\nPros:\nThe problem of long horizon visual planning to goal images is important, and the paper does a good job of motivating it and describing the prior work in the area. The idea the paper presents is also intuitive and well motivated - while several recent works have explored building graphs and searching in these graphs to find subgoals, doing so over all experience can be costly, and discrete representations can aid in more compact graphs. \n\nFurthermore, the proposed technique for learning this discrete representation, CPC but using a set of discrete latents trained with Gumbel Softmax is intuitive and to the best of my knowledge novel. And given a set of these discrete one hot encodings, the proposed inductive bias on the similarity function to encourage matching as many one-hots as possible seems to help performance. \n\nCons:\nMy main criticisms are on (1) the generality of the proposed planning procedure/representation learning, and (2) the thoroughness of the experimental comparisons.\n\n(1): \nFirst, while the proposed representation learning procedure seems to work quite well on the block world setting used in this paper, I have doubts about its ability to scale to more realistic environments. Since the proposed architecture learns a discrete latent variable for each channel feature map, it seems natural that this would easily learn to separate different multicolored blocks on a static background into different one hot encodings representing their position. But in more challenging domains like robotic manipulation or ego-centric visual navigation (which have been studied in the prior work), learning this discrete representation seems substantially more challenging, and its not clear that objects would be so cleanly separated.\n\nFurthermore, these objects being cleanly factorized seems critical to the subsequent planning procedure. The idea of building a separate graph for each of the i one-hot encodings, and planning along a single one hot encoding at a time seems very specific to the domains used for evaluation. In general given a discrete representation of state, shouldn't the planning graph consider connections between this entire discrete state? The current planning algorithm (as the paper mentions) is dependent on the assumption that each latent one-hot can be manipulated independently. Not only does this in general not hold, it also implicitly assumes that the learned representation successfully factorizes the state space, which as mentioned above can be non trivial in harder environments.\n\n(2):\nThe experimental evaluation is limited, in both domains and comparisons. First, there are many relevant works which the paper cites, but does not compare to. For the types of problems the paper looks at visual foresight is not the state of the art baseline. Methods like [1,2,3,4,5,6,7] (all of which the paper cites) would make for better points of comparison. Additionally, since a big part of the papers contribution is the method for learning the discrete representation, the paper should also compare to other techniques for learning such a discrete representation, such as [8,9].\n\nAlso as mentioned above, the experimental domains are visually simple, and designed in a way which is very particular to this method. I think showing results on more general problems like robotic manipulation or egocentric navigation are necessary to understanding if this approach really works. \n\nOverall, this paper provides some interesting and well motivated ideas, but has some limitations in the generality of the proposed method and the thoroughness of the experimental evaluation. \n\n[1] Kurutach et al. Learning plannable representations with causal infogan. 2018.\n[2] Nasiriany et al. Planning with goal-conditioned policies. 2019. \n[3] Nair et al. Hierarchical foresight: Self-supervised learning of long-horizon tasks via visual subgoal generation. 2019.\n[4] Savinov et al. Semi-parametric topological memory for navigation. 2018.\n[5] Liu et al. Hallucinative topological memory for zero-shot visual planning. 2020.\n[6] Eysenbach et al. Search on the replay buffer: Bridging planning and reinforcement learning. 2019.\n[7] Pertsch et al. Long-horizon visual planning with goal-conditioned hierarchical predictors. 2020.\n[8] Caron et al. Deep Clustering for Unsupervised Learning of Visual Features. 2019.\n[9] Van den Oord et al. Neural Discrete Representation Learning. 2018.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5RQDNq4_78", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents learning discrete encoding (as one-hot vectors) given a scene image to represent various semantics about the scene with an aim to perform graph search over these representations in performing high level tasks.\n\nThe research direction is quite relevant and the motivation make a good case for it. Related work is adequately discussed. This is a promising line of work but in its current state a bit premature due to two main related reasons: a mismatch between the motivation/goal discussed early on and the validation performed; and the experiments are conducted on very small and simplistic game like setups.\n\n- The two main environments studied are game like and too simplistic. While these can help in building intuition without more sufficiently complex experiments it is hard to understand the boundaries within which this approach can be used. Input images are 16x16 which is quite small. Several new environments like AI2THOR are around to study task planning, would the proposed method be able to tackle any such scenarios? More task planning baselines (particularly non-learning based) should be considered. What if a goal image is not available as the target but the task is provided by other means?\n\n- Building the initial representation by pure exploration seems prohibitively expensive for complex setting, would any efficient alternative work?\n\n- Learning 'low level control' is referenced several time in the paper but the experiments have no mention of it or are designed to explore/study this.\n\nOther comments:\n\n- It wasn't too clear in the initial read until much later in the paper, what 'object-factorized' really meant\n\n- Problem statement in 2.1 is quite vague and grounding it in an example would aid in understanding.\n\n- The underlying approach is a bit hard to fully understand. An explanation of 3.2 along with a simple example in parallel would help (maybe walkthrough a full iteration of the approach).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores a foundational problem in AI around learning abstractions that allow for easier planning.  The work proposes a specific procedure for learning temporally abstract, discrete representations in which it becomes tractable to perform graph-based search.  Evaluation is performed on two 2D tasks where a goal is specified visually and the system must produce the actions to achieve this target state/observation. 

The reviewers were in a uncommonly tight consensus as to their evaluation of the paper (all 4).  All reviewers essentially expressed that the motivation of the paper was solid but that the domains considered were too simplistic for validation of this class of approach, especially in light of substantial previous work in the area that was not adequately captured in the baseline comparisons.  The authors did not respond to the reviewers.

My decision is to reject the paper.
Decision: Reject