Year: 2020

Abstract:
Existing model-based reinforcement learning methods often study perception modeling and decision making separately. We introduce joint Perception and Control as Inference (PCI), a general framework to combine perception and control for partially observable environments through Bayesian inference. Based on the fact that object-level inductive biases are critical in human perceptual learning and reasoning, we propose Object-based Perception Control (OPC), an instantiation of PCI which manages to facilitate control using automatic discovered object-based representations. We develop an unsupervised end-to-end solution and analyze the convergence of the perception model update. Experiments in a high-dimensional pixel environment demonstrate the learning effectiveness of our object-based perception control approach. Specifically, we show that OPC achieves good perceptual grouping quality and outperforms several strong baselines in accumulated rewards.

heading: 1 INTRODUCTION
text: Human-like computing, which aims at endowing machines with human-like perceptual, reasoning and learning abilities, has recently drawn considerable attention (Lake, 2014; Lake et al., 2015; Baker et al., 2017). In order to operate within a dynamic environment while preserving homeostasis (Kauffman, 1993), humans maintain an internal model to learn new concepts efficiently from a few examples (Friston, 2005). The idea has since inspired many model-based reinforcement learning (MBRL) approaches to learn a concise perception model of the world (Kaelbling et al., 1998). MBRL agents then use the perceptual model to choose effective actions. However, most existing MBRL methods separate perception modeling and decision making, leaving the potential connection between the objectives of these processes unexplored. A notable work by Hafner et al. (2020) provides a unified framework for perception and control. Built upon a general principle this framework covers a wide range of objectives in the fields of representation learning and reinforcement learning. However, they omit the discussion on combining perception and control for partially observable Markov decision processes (POMDPs), which formalizes many real-world decision-making problems. In this paper, therefore, we focus on the joint perception and control as inference for POMDPs and provide a specialized joint objective as well as a practical implementation.
Many prior MBRL methods fail to facilitate common-sense physical reasoning (Battaglia et al., 2013), which is typically achieved by utilizing object-level inductive biases, e.g., the prior over observed objects’ properties, such as the type, amount, and locations. In contrast, humans can obtain these inductive biases through interacting with the environment and receiving feedback throughout their lifetimes (Spelke et al., 1992), leading to a unified hierarchical and behavioral-correlated perception model to perceive events and objects from the environment (Lee and Mumford, 2003). Before taking actions, a human agent can use this model to decompose a complex visual scene into distinct parts, understand relations between them, reason about their dynamics and predict the consequences of its actions (Battaglia et al., 2013). Therefore, equipping MBRL with object-level inductive biases is essential to create agents capable of emulating human perceptual learning and reasoning and thus complex decision making (Lake et al., 2015). We propose to train an agent in a similar way to gain inductive biases by learning the structured properties of the environment. This can enable the agent to plan like a human using its ability to think ahead, see what would happen for a range of possible choices, and make rapid decisions while learning a policy with the help of the inductive bias (Lake et al., 2017). Moreover, in order to mimic a human’s spontaneous acquisition of inductive biases
o1
s1
a1 o2 a2
……θ1 s2 θ 2
x1 x2
q(s) p (s |x, o) p (x, o |s) p (s)
throughout its life, we propose to build a model able to acquire new knowledge online, rather than a one which merely generates static information from offline training (Dehaene et al., 2017).
In this paper, we introduce joint Perception and Control as Inference (PCI) as shown in Fig. (1), a unified framework for decision making and perception modeling to facilitate understanding of the environment while providing a joint objective for both the perception and the action choice. As we argue that inductive bias gained in object-based perception is beneficial for control tasks, we then propose Object-based Perception Control (OPC), an instantiation of PCI which facilitates control with the help of automatically discovered representations of objects from raw pixels. We consider a setting inspired by real-world scenarios; we consider a partially observable environment in which agents’ observations consist of a visual scene with compositional structure. The perception optimization of OPC is typically achieved by inference in a spatial mixture model through generalized expectation maximization (Dempster et al., 1977), while the policy optimization is derived from conventional temporal-difference (TD) learning (Sutton, 1988). Proof of convergence for the perception model update is provided in Appendix A. We test OPC on the Pixel Waterworld environment. Our results show that OPC achieves good quality and consistent perceptual grouping and outperforms several strong baselines in terms of accumulated rewards.

heading: 2 RELATED WORK
text: Connecting Perception and Control Formulating RL as Bayesian inference over inputs and actions has been explored by recent works (Todorov, 2008; Kappen et al., 2009; Rawlik et al., 2010; Ortega and Braun, 2011; Levine, 2018; Tschiatschek et al., 2018; Lee et al., 2019b;a; Ortega et al., 2019; Xin et al., 2020; O’Donoghue et al., 2020). The generalized free energy principle (Parr and Friston, 2019) studies a unified objective by heuristically defining entropy terms. A unified framework for perception and control from a general principle is proposed by Hafner et al. (2020). Their framework provides a common foundation from which a wide range of objectives can be derived such as representation learning, information gain, empowerment, and skill discovery. However, one trade-off for the generality of their framework is the loss in precision. Environments in many real-world decision-making problems are only partially observable, which signifies the importance of MBRL methods to solving POMDPs. However, relevant and integrated discussion is omitted in Hafner et al. (2020). In contrast, we focus on the joint perception and control as inference for POMDPs and provide a specialized joint-objective as well as a practical implementation.
Model-based Deep Reinforcement Learning MBRL algorithms have been shown to be effective in various tasks (Gu et al., 2016), including operating in environments with high-dimensional raw pixel observations (Igl et al., 2018; Shani et al., 2005; Watter et al., 2015; Levine et al., 2016; Finn and Levine, 2017). Existing methods have considered incorporating reward structure into modellearning (Farahmand et al., 2017; Oh et al., 2017), while our proposed PCI takes one step forward by incorporating the perception model into the control-as-inference derivation to yield a single unified objective for multiple components in a pipeline. One of the methods closely related to OPC is the World Model (Ha and Schmidhuber, 2018), which consists of offline and separately trained models for vision, memory, and control. These methods typically produce entangled latent representations for pixel observations whereas, for real world tasks such as reasoning and physical interaction, it is often necessary to identify and manipulate multiple entities and their relationships for optimal performance. Although Zambaldi et al. (2018) has used the relational mechanism to discover and reason about entities, their model needs additional supervision of location information.
Object-based Reinforcement Learning The object-based approach, which recognizes decomposed objects from the environment observations, has attracted considerable attention in RL as well (Schmidhuber, 1992). However, most models often use pre-trained object-based representations rather than learning them from high-dimensional observations (Diuk et al., 2008; Kansky et al., 2017). When objects are extracted through learning methods, these models usually require supervised modeling of the object property, by either comparing the activation spectrum generated from neural network filters with existing types (Garnelo et al., 2016) or leveraging the bounding boxes generated by standard object detection algorithms (Keramati et al., 2018). MOREL (Goel et al., 2018) applies optical flow in video sequences to learn the position and velocity information as input for model-free RL frameworks.
A distinguishing feature of our work in relation to previous works in MBRL and the object-based RL is that we provide the decision-making process with object-based abstractions of high-dimensional observations in an unsupervised manner, which contribute to faster learning.
Unsupervised Object Segmentation Unsupervised object segmentation and representation learning have seen several recent breakthroughs, such as IODINE (Greff et al., 2019), MONet (Burgess et al., 2019), and GENESIS (Engelcke et al., 2020). Several recent works have investigated the unsupervised object extraction for reinforcement learning as well (Zhu et al., 2018; Asai and Fukunaga, 2017; Kulkarni et al., 2019; Watters et al., 2019; Veerapaneni et al., 2020). Although OPC is built upon a previous unsupervised object segmentation back-end (Greff et al., 2017; van Steenkiste et al., 2018), we explore one step forward by proposing a joint framework for perceptual grouping and decision-making. This could help an agent to discover structured objects from raw pixels so that it could better tackle its decision problems. Our framework also adheres to the Bayesian brain hypothesis by maintaining and updating a compact perception model towards the cause of particular observations (Friston, 2010).

heading: 3 METHODS
text: We start by introducing the environment as a partially observable Markov Decision Process (POMDP) with an object-based observation distribution in Sect. 3.11. We then introduce PCI, a general framework for joint perception and control as inference in Sect. 3.2 and arrive at a joint objective for perception and control models. In the remainder of this section we propose OPC, a practical method to optimize the joint objective in the context of an object-based environment, which requires the model to exploit the compositional structure of a visual scene.

heading: 3.1 ENVIRONMENT SETTING
text: We define the environment as a POMDP represented by the tuple = hS,P,A,X ,U ,Ri, where S,A,X are the state space, the action space, and the observation space, respectively. At time step t, we consider an agent’s observation xt 2 X ⌘ RD as a visual image (a matrix of pixels) composited of K objects, where each pixel xi is determined by exactly one object. The agent receives xt following the conditional observation distribution U(xt|st) : S ! X , where the hidden state st is defined by the tuple (zt,✓t1, . . . ,✓tK). Concretely, we denote as z
t 2 Z ⌘ [0, 1]D⇥K the latent variable which encodes the unknown true pixel assignments, such that zti,k = 1 iff pixel z t i was generated by component k. Each pixel xti is then rendered by its corresponding object representations ✓tk 2 RM through a pixel-wise distribution U ti,k(x t i|zti,k = 1) 2, where ti,k = f (✓tk)i is generated by feeding ✓tk into a differentiable non-linear function f . When the environment receives an action a t 2 A, it moves to a new state st+1 following the transition function P(st+1|st, at) : S⇥A ! S . We assume the transition function could be parameterized and we integrate its parameter into . To embed the control problem into the graphical model, we also introduce an additional binary random variable ot to represent the optimality at time step t, i.e., ot = 1 denotes that time step t is optimal, and ot = 0 denotes that it is not optimal. We choose the distribution over ot to be p(ot = 1|st, at) / exp(rt),
1Note that the PCI framework is designed for general POMDPs. We extend Sect. 3.1 to object-based POMDPs for the purpose of introducing the environment setting for OPC.
2We consider U as Gaussian, i.e., U ti,k (x t i|zti,k = 1) ⇠ N (xti;µ = ti,k, 2) for some fixed 2.
where rt 2 R is the observed reward provided by the environment according to the reward function R(rt|st, at) : S ⇥A ! R. We denote the distribution over initial state as p(s1) : S ! [0, 1].

heading: 3.2 JOINT PERCEPTION AND CONTROL AS INFERENCE
text: To formalize the belief about the unobserved hidden cause of the history observation xt, the agent maintains a perception model qw(s) to approximate the distribution over latent states as illustrated in Fig. (1). An agent’s inferred belief about the latent state could serve as a sufficient statistic of the history and be used as input to its policy ⇡, which guides the agent to act in the environment. The goal of the agent is to maximize the future optimality o t while learning a perception model by inferring unobserved temporal hidden states given the past observations xt and actions at, which is achieved by maximizing the following objective:
log p(o t,xt|a<t) = log X
s 1,a t,x>t
p(o t, s 1,x 1, a t|a<t)
= log
2 4 X
st
qw Qt j=1 p(x j |sj)p(s1) Qt 1 m=1 p(s
m+1|sm, am) qw [ 1 ]
3
5 , (1)
where we denote qw .= q(st|xt, a<t) and use 1 to represent the term related to control as
1 = X
s>t,a t,x>t
Y
n=t
p(on|sn, an) Y
k=t
p(xk+1|sk+1)p(sk+1|sk, ak)p(ak|a<t).
The full derivation is presented in Appendix C.1. We denote qw .= q(st|xt, a<t) and assume qw = p(s1) Qt g=2 q(s
g|x<g, a<g), where we slightly abuse notation for qw by ignoring the fact that we sample from the model p(s1) for t = 1. We then apply Jensen’s inequality to Eq.(1) and get
log p(o t,xt|a<t) Est⇠qw
" log Qt j=1 p(x j |sj)p(s1) Qt 1 m=1 p(s m+1|sm, am)
p(s1) Qt
g=2 q(s g|x<g, a<g)
# + Est⇠qw [log 1 ]
=Est⇠qw
" log tY
j=1
p(xj |sj) # DKL " t 1Y
g=1
q(sg+1|xg, ag)kp(sg+1|sg, ag) #
| {z } Lw(qw, )
+Est⇠qw [log 1 ] ,
where DKL represents the Kullback–Leibler divergence (Kullback and Leibler, 1951). As introduced in Sect. 3.1, we parameterize the transition distribution p (st+1|st, at) and the observation distribution p (xt|st) by . An instantiation to optimize the above evidence lower bound (ELBO) Lw(qw, ) in an environment with explicit physical properties and high-dimensional pixel-level observations will be discussed in Sect. 3.3.
We now derive the control objective by extending 1 as
log X
s>t,a t,x>t
q(s>t, a t,x>t)
Q n=t p(o n|sn, an) Q k=t p (x k+1|sk+1)p (sk+1|sk, ak)p(ak|a<t)
q(s>t, a t,x>t)
where we assume q(s>t, a t,x>t) = Q
h=t p (x h+1|sh+1)p (sh+1|sh, ah)⇡(ah|sh) and denote
qc . = q(s>t, a t,x>t). We then apply Jensen’s inequality to Est⇠qw [log 1 ] and get
Est⇠qw [log 1 ] Est⇠qw
2 4 X
s>t,a t,x>t
qc log
Q n=t p(o n|sn, an) Q
k=t p (x k+1|sk+1)p (sk+1|sk, ak)p(ak|a<t)Q
h=t p (x h+1|sh+1)p (sh+1|sh, ah)⇡(ah|sh)
3
5
=Est⇠qw " X
t
Est+1,at,xt+1⇠⇢ ⇡ ⇥ R(st, at) DKL(⇡(at|st)||p(at|a<t)) ⇤ # ,
(2) where we denote ⇢ as p (xh+1|sh+1)p (sh+1|sh, ah). Note that as this control objective is an expectation under ⇢ , reward maximization also bias the learning of our perception model. We will propose an option to optimize the control objective Eq.(2) in Sect. 3.4.

heading: 3.3 OBJECT-BASED PERCEPTION MODEL UPDATE
text: We now introduce OPC, an instantiation of PCI in the context of an object-based environment. Following the generalized expectation maximization (Dempster et al., 1977), we optimize the ELBO Lw(qw, ) by improving the perception model qw about the true posterior p (st+1|xt+1, st, at) with respect to a set of object representations ✓t = [✓t1, . . . ,✓tK ] 2 ⌦ ⇢ RM⇥K .
E-step to compute a new estimate qw of the posterior. Assume the ELBO is already maximized with respect to qw at time step t, i.e., qw = p (st+1i |x t+1 i , s t, at) = p (z t+1 i |x t+1 i , t i , a
t), we can generate a soft-assignment of each pixel to one of the K objects as
⌘ti,k . = p (z t+1 i,k = 1|x t+1 i , t i , a t). (3)
M-step to update the model parameter . We then find ✓t+1 to maximize the ELBO as ✓t+1 = argmax
✓t+1 E zt+1⇠⌘t
⇥ log p (x t+1, zt+1, t+1|st, at) ⇤ . = argmax
✓t+1 ⇤✓t(✓
t+1). (4)
See derivation in Appendix C.2. Note that Eq. (4) returns a set of points that maximize ⇤✓t(✓t+1), and we choose ✓t+1 to be any value within this set. To update the perception model by maximizing the evidence lower bound with respect to q(st) and ✓t+1, we compute Eq. (3) and Eq. (4) iteratively. However, an analytical solution to Eq. (4) is not available because we use a differentiable non-linear function f to map from object representations ✓tk into t i,k = f (✓ t k)i. Therefore, we get ✓ t+1 k by
✓t+1k = ✓ t k + ↵
@⇤✓t+1(✓ t+2)
@✓t+1k
✓t+1k =✓ t k
= ✓tk + ↵ DX
i=1
⌘ti,k · ti,k x t+1 i 2 · @ ti,k @✓tk , (5)
where ↵ is the learning rate (see details of derivation in Appendix C.3).
We regard the iterative process as a tro-step rollout of K copies of a recurrent neural network with hidden states ✓tk receiving ⌘ t k ( tk xt+1) as input (see the inner loop of Algorithm 1). Each copy generates a new t+1k , which is then used to re-estimate the soft-assignments ⌘t+1k . We parameterize the Jacobian @ t k/@✓ t k and the differentiable non-linear function f k using a convolutional encoder-decoder architecture with a recurrent neural network bottleneck,
which linearly combines the output of the encoder with ✓tk from the previous time step. To fit the statistical model f to capture the regularities corresponding to the observation and the transition distribution for given POMDPs, we back-propagate the gradient of E⌧⇠⇡ ⇥ ⇤✓t(✓t+1) ⇤ through “time” (also known as the BPTT Werbos (1988)) into the weights . We demonstrate the convergence of a sequence of {Lw✓t(qw, )} generated by the perception update in Appendix A. The proof is presented by showing that the learning process follows the Global Convergence Theorem (Zangwill, 1969).
Our implementation of the perceptual model is based on the unsupervised perceptual grouping method proposed by (Greff et al.,
2017; van Steenkiste et al., 2018). However, using other advanced unsupervised object representations learning methods as the perceptual model (such as IODINE (Greff et al., 2019) and MONet (Burgess et al., 2019)) is a straightforward extension.
Algorithm 1 Learning the Object-based Perception Control Initialize ✓,⌘, , ⇣, ⇣v, Tepi, tro, K Create Ne environments that will execute in parallel while training not finished do
Initialize the history Da,Dx,D⌘ with environment rollouts for tro +1 time-steps under current policy ⇡⇣ for T = 1 to Tepi tro do
d 0 Get ⌘T 1 from D⌘ for t = T to T + tro do
Get at,xt from Da,Dx respectively Feed ⌘t 1k ( t 1 k x
t) into each of the K RNN copy to get ✓tk and forward-output tk Compute ⌘tk by Eq. 3 d d + @ ⇤✓t(✓t+1) /@ by Eq. (4)
Perform aT+tro according to policy ⇡⇣(aT+tro |✓T+tro) Receive reward rT+tro and new observation xT+tro+1 Store aT+tro ,xT+tro+1,⌘T in Da,Dx,D⌘ respectively Feed ⌘T+trok ( T+tro k x
T+tro+1) into each of the K RNN copy to get ✓T+tro+1k y rT+tro + V⇣v (✓T+tro+1) d⇣ r⇣ log ⇡⇣(aT+tro |✓T+tro)(y V⇣v (✓T+tro)) d⇣v @ y V⇣v (✓T+tro) 2 /@⇣v d d + @ y V⇣v (✓T+tro) 2 /@
Perform synchronous update of using d , of ⇣ using d⇣, and of ⇣v using d⇣v

heading: 3.4 DECISION-MAKING MODULE UPDATE
text: Recall that the control objective in Eq.(2) is the sum of the expected total reward along trajectory ⇢✓⇡ and the negative KL divergence between the policy and our prior about the future actions. The prior can reflect a preference over policies. For example, a uniform prior in SAC (Haarnoja et al., 2018) corresponds to the preference for a simple policy with minimal assumptions. This could prevent early convergence to sub-optimal policies. However, as we focus on studying the benefit of combing perception with control, we do not pre-impose a preference for policies. Therefore we set the prior equal to our training policy, resulting in a zero KL divergence term in the objective.
To maximize the total reward along trajectory ⇢✓⇡, we follow the conventional temporal-difference (TD) learning approach (Sutton, 1988) by feeding the object abstractions into a small multilayer perceptron (MLP) (Rumelhart et al., 1986) to produce a (dimR(A) + 1)-dimensional vector, which is split into a dimR(A)-dimensional vector of ⇡⇣’s (the ‘actor’) logits, and a baseline scalar V⇣v (the ‘critic’). The ⇡⇣ logits are normalized using a softmax function, and used as the multinomial distribution from which an action is sampled. The V⇣v is an estimate of the state-value function at the current state, which is given by the last hidden state ✓ of the tro-step RNN rollout. On training the decision-making module, the V⇣v is used to compute the temporal-difference error given by
LTD = (yt+1 V⇣v (✓ t+1))2, yt+1 = rt+1 + V⇣v (✓ t+2), (6)
where 2 [0, 1) is a discount factor. LTD is used both to optimize ⇡⇣ to generate actions with larger total rewards than V⇣v predicts by updating ⇣ with respect to the policy gradient
r⇣ log ⇡⇣(at+1|✓t+1)(y V⇣v (✓ t+1)),
and to optimize V⇣v to more accurately estimate state values by updating ⇣v. Also, differentiating LTD with respect to enables the gradient-based optimizers to update the perception model. We provide the pseudo-code for one-step TD-learning of the proposed model in Algorithm 1. By grouping objects concerning the reward, our model distinguishes objects with visual similarities but different semantics, thus helping the agent to better understand the environment.

heading: 4 EXPERIMENTS

heading: 4.1 PIXEL WATERWORLD
text: We demonstrate the advantage of unifying decision making and perception modeling by applying OPC on an environment similar to the one used in COBRA (Watters et al., 2019), a modified
Waterworld environment (Karpathy, 2015), where the observations are 84⇥ 84 grayscale raw pixel images composited of an agent and two types of randomly moving targets: the poison and the food, as illustrated in Fig. (2). The agent can control its velocity by choosing from four available actions: to apply the thruster to the left, right, up and down. A negative cost is given to the agent if it touches any poison target, while a positive reward for making contact with any food target. The optimal strategy depends on the number, speed, and size of objects, thus requiring the agent to infer the underlying dynamics of the environment within a given amount of observations.
The intuition of this environment is to test whether the agent can quickly learn dynamics of a new environment online without any prior knowledge, i.e., the execution-time optimization throughout the agent’s life-cycle to mimic humans’ spontaneous process of obtaining inductive biases. We choose Advantage Actor-Critic (A2C) (Mnih et al., 2016) as the decision-making module of OPC without loss of generality, although other online learning methods are also applicable. For OPC, we use K = 4 and tro = 20 except for Sect. 4.5, where we analyze the effect of the hyper-parameter setting.

heading: 4.2 ACCUMULATED REWARD COMPARISONS
text: To verify object-level inductive biases facilitating decision making, we compare OPC against a set of baseline algorithms, including: 1) the standard A2C, which uses convolutional layers to transform the raw pixel observations to low dimensional vectors as input for the same MLP described in Sect. 3.4, 2) the World Model (Ha and Schmidhuber, 2018) with A2C for control (WM-A2C), a state-of-the-art modelbased approach, which separately learns a visual and a memorization model to provide the in-
put for a standard A2C, and 3) the random policy. For both the baseline A2C, WM-A2C and the decision-making module of OPC, we follow the convention of Mnih et al. (2016) by running the algorithm in the forward view and using the same mix of n-step returns to update both ⇡⇣ and V⇣v . Details of the model and of hyperparameter setting can be found in Appendix B. We build the environment with two poison objects and one food object, and set the size of the agent 1.5 times smaller than the target. Results are reported with the average value among separate runs of three random seeds. Note that the training procedure of WM-A2C includes independent off-line training for each component, thus requiring many more observation samples than OPC. Following its original early-stopping criteria, we report that WM-A2C requires 300 times more observation samples for separately training a perception model before learning a policy to give the results presented in Fig. (3).
Fig. (3a) shows the result of accumulated rewards after each agent has experienced the same amount of observations. Note that OPC uses the plotted number of experienced observations for training the entire model, while WM-A2C uses the amount of observations only for training the control module. It is clear that the agent with OPC achieves the greatest performance, outperforming all other agents regardless of model in terms of accumulated reward. We believe this advantage is due to the object-level inductive biases obtained by the perception model (compared to entangled representations extracted by the CNN used in standard A2C), and the unified learning of perception and control of OPC (as opposed to the separate learning of different modules in WM-A2C).
To illustrate each agent’s learning process through time, we also present the period reward, which is the accumulated reward during a given period of environment interactions (2e4 of experienced
observations). As illustrated in Fig. (3b), OPC significantly improves the sample efficiency of A2C, which enables the agent acting in the environment to find an optimal strategy more quickly than agents with baseline models. We also find that the standard version of A2C with four parallel threads gives roughly the same result as the single-threaded version of A2C (the same as the decision-making module of OPC), eliminating the potential drawback of single-thread learning.

heading: 4.3 PERCEPTUAL GROUPING RESULTS
text: To demonstrate the performance of the perception model, we provide an example rollout at the early stage of training in Fig. (4). We assign a different color to each k for better visualization
and show through the soft-assignment ⌘ that all objects are grouped semantically: the agent in blue, the food in green, and both poisons in red. During the joint training of perception and policy, the soft-assignment gradually constitutes a semantic segmentation as the TD signal improves the recognition when grouping pixels into different types based on the interaction with the environment. Consequently, the learned object representations ✓ becomes a semantic interpretation of raw pixels grouped into perceptual objects.

heading: 4.4 BENEFITS OF JOINT INFERENCE
text: To gain insights into the benefit of the control objective for joint inference, we further compare OPC against OP, a perception model with the same architecture as OPC but no guidance of the TD signal from RL, i.e., using the same unsupervised object segmentation back-end as
in Greff et al. (2017). Fig. (5) shows the result of accumulated rewards and the soft-assignment ⌘ produced by both perception models. Results are produced by models running in environments with the same random seed.
As illustrated in Fig. (5a), OPC outperforms OP in terms of the accumulated rewards through time with a given amount of observations. This performance difference owes to the fact shown in Fig. (5b), where the ⌘ of OP are overlaid, and the shapes are not delineated in bold colors but are a mixture, showing that OP has not learned to segment the objects clearly. The pixel assignment (coloring) does not appear to converge even after an extremely large number of iterations, suggesting that OP alone cannot properly show signs of any semantic adaptation to the task at hand.
0.0 0.5 1.0 1.5 # of experienced observations (*1e5)
−6
−4
−2
0
C os
ts (*
1e 3)
23C 23
(a) OPC OP OPC OP
# = 14000 # = 14500
OPC OP OPC OP
# = 14000 # = 14500
x
(b)
environment setting of the high moving speed, larger target sizes and more poison objects, we believe that agents are given strong indications to stay away from all stimuli. Thus, the result shows an understanding of visual reasoning that the agent can separate itself from the rest of the objects.

heading: 4.5 ABLATION STUDY ON HYPER-PARAMETERS
text: We also investigate the effect of hyper-parameters to the learning of OPC, by changing: 1) the number of recurrent copies K, 2) the rollout steps for recurrent iteration tro, and 3) the use of relational mechanism across object representations ✓ (see Sect. 2.2 of van Steenkiste et al. (2018) for more details). Fig. (3c) and Fig. (3d) show period reward results across different hyper-parameter settings.
As illustrated in Fig. (3c), the number of recurrent copies affects the stability of OPC learning, as OPC with K = 3 has experienced larger variance during the agent’s life-cycle. We believe the difference comes from the environment dynamics as we have visually four objects in the environment. During the earlier stage of interacting with the environment, OPC tries to group each object into a distinct class; thus, a different number of K against the number of objects in the environment confuse the perception model and lead to unstable learning. Although different K settings might affect the learning stability and slow down the convergence, OPC can still find an optimal strategy within a given amount of observations.
In Fig. (3d), we compare OPC with different steps of recurrent rollout tro. A smaller tro means fewer rounds of perception updates and therefore slower convergence in terms of the number of experienced observations. We believe that the choice of tro depends on the difficulty of the environment, e.g., a smaller tro can help to find the optimal strategy more quickly for simpler environments in terms of wall training time.
Meanwhile, results in Fig. (3c) and Fig. (3d) show that the use of a relational mechanism has limited impact on OPC, possibly because the objects can be well distinguished and perceived by their object-level inductive biases, i.e. shapes in our experiment. We believe that investigating whether the relational mechanism will have impact on environments where entities with similar object-level inductive bias have other different internal properties is an interesting direction for future work.

heading: 5 CONCLUSIONS
text: In this paper, we propose joint Perception and Control as Inference (PCI), a general framework to combine perception and control for POMDPs through Bayesian inference. We then extend PCI to the context of a typical pixel-level environment with compositional structure and propose Object-based Perception Control (OPC), an instantiation of PCI which manages to facilitate control with the help of automatically discovered object-based representations. We provide the convergence proof of OPC perception model update and demonstrate the execution-time optimization ability of OPC in a high-dimensional pixel environment. Notably, our experiments show that OPC achieves high quality and consistent perceptual grouping and outperforms several strong baselines in terms of accumulated rewards within the agent’s life-cycle. OPC agent can quickly learn the dynamics of a new environment without any prior knowledge, imitating the inductive bias acquisition process of humans. For future work, we would like to investigate OPC with more types of inductive biases and test the model performance in a wider variety of environments.

heading: A CONVERGENCE OF THE OBJECT-BASED PERCEPTION MODEL UPDATE
text: Under main assumptions and lemmas as introduced below, we demonstrate the convergence of a sequence of {Lw✓t(qw, )} generated by the perception update. The proof is presented by showing that the learning process follows the Global Convergence Theorem (Zangwill, 1969). Assumption 1. ⌦✓0 = ✓ 2 ⌦ : Lw✓ (qw, )  Lw✓0(qw, ) is compact for any Lw✓0(qw, ) < 1. Assumption 2. L is continuous in ⌦ and differentiable in the interior of ⌦.
The above assumptions lead to the fact that {Lw✓t(qw, )} is bounded for any ✓0 2 ⌦. Lemma 1. Let ⌦S be the set of stationary points in the interior of ⌦, then the mapping argmax✓t+1 ⇤✓t(✓ t+1) from Eq. 4 is closed over ⌦\⌦S (the complement of ⌦S).
Proof. See Wu (1983). A sufficient condition is that ⇤✓t(✓t+1) is continuous in both ✓t+1 and ✓t.
Proposition 1. Let ⌦S be the set of stationary points in the interior of ⌦, then (i.) 8✓t 2 ⌦S ,Lw✓t+1(q w, )  Lw✓t(qw, ) and (ii.) 8✓t 2 ⌦\⌦S ,Lw✓t+1(q w, ) < Lw✓t(qw, ).
Proof. Note that (i.) holds true given the condition. To prove (ii.), consider any ✓t 2 ⌦\⌦S , we have
@Lw✓t(qw, ) @✓t+1
✓t+1=✓t
= @⇤✓t(✓t+1)
@✓t+1
✓t+1=✓t
6= 0.
Hence ⇤✓t(✓t+1) is not maximized at ✓t+1 = ✓t. Given the perception update described by Eq. (5), we therefore have ⇤✓t(✓t+1) > ⇤✓t(✓t), which implies Lw✓t+1(q w, ) < Lw✓t(qw, ).
Theorem 1. Let {✓t} be a sequence generated by the mapping from Eq. 4, ⌦S be the set of stationary points in the interior of ⌦. If Assumptions 1 & 2, Lemma 1, and Proposition 1 are met, then all the limit points of {✓t} are stationary points (local minima) and Lw✓t(qw, ) converges monotonically to J⇤ = Lw✓⇤(qw, ) for some stationary point ✓⇤ 2 ⌦S .
Proof. Suppose that ✓⇤ is a limit point of the sequence {✓t}. Given Assumptions 1 & 2 and Proposition 1.i), we have that the sequence {✓t} are contained in a compact set ⌦K ⇢ ⌦. Thus, there is a subsequence {✓l}l2L of {✓t} such that ✓l ! ✓⇤ as l ! 1 and l 2 L. We first show that Lw✓t(qw, ) ! Lw✓⇤(qw, ) as t ! 1. Given L is continuous in ⌦ (Assumption 2), we have Lw✓l(q w, ) ! Lw✓⇤(qw, ) as l ! 1 and l 2 L, which means
8✏ > 0, 9l(✏) 2 L s.t. 8l l(✏), l 2 L,Lw✓l(q w, ) Lw✓⇤(qw, ) < ✏. (7)
Given Proposition 1 and Eq. (4), L is therefore monotonically decreasing on the sequence {✓t}1t=0, which gives 8t,Lw✓t(qw, ) Lw✓⇤(qw, ) 0. (8) Given Eq. (7), for any t l(✏), we have
Lw✓t(q w, ) Lw✓⇤(qw, ) = Lw✓t(q w, ) Lw✓l(✏)(q w, )
| {z } 0
+Lw✓l(✏)(q w, ) Lw✓⇤(qw, )| {z }
<✏
< ✏. (9)
Given Eq. (8) and Eq. (9), we therefore have Lw✓t(qw, ) ! Lw✓⇤(qw, ) as t ! 1. We then prove that the limit point ✓⇤ is a stationary point. Suppose ✓⇤ is not a stationary point, i.e., ✓⇤ 2 ⌦\⌦S , we consider the sub-sequence {✓l+1}l2L, which are also contained in the compact set ⌦K . Thus, there is a subsequence {✓l0+1}l02L0 of {✓l+1}l2L such that ✓l
0+1 ! ✓⇤0 as l0 ! 1 and l0 2 L0, yielding Lw ✓l0+1 (qw, ) ! Lw ✓⇤0 (qw, ) as l0 ! 1 and l0 2 L0, which gives
Lw ✓⇤0 (qw, ) = lim l0!1 l02L0 Lw ✓l0+1 (qw, ) = lim t!1 t2N Lw✓t(qw, ) = Lw✓⇤(qw, ). (10)
On the other hand, since the mapping from Eq. (4) is closed over ⌦\⌦S (Lemma 1), and ✓⇤ 2 ⌦\⌦S , we therefore have ✓⇤
0 2 argmax✓t+1 ⇤✓⇤(✓t+1), yielding Lw✓⇤0 (q w, ) < Lw✓⇤(qw, ) (Proposition
1.ii), which contradicts Eq. (10).

heading: B EXPERIMENT DETAILS
text: OPC In all experiments we trained the perception model using ADAM Kingma and Ba (2014) with default parameters and a batch size of 32. Each input consists of a sequence of binary 84⇥ 84 images containing two poison objects (two circles) and one food object (a rectangle) that start in random positions and move within the image for tro steps. These frames were thresholded at 0.0001 to obtain binary images and added with bit-flip noise (p = 0.2). We used a convolutional encoder-decoder architecture inspired by recent GANs Chen et al. (2016) with a recurrent neural network as bottleneck, where the encoder used the same network architecture from Mnih et al. (2013) as
1. 8⇥ 8 conv. 16 ELU. stride 4. layer norm 2. 4⇥ 4 conv. 32 ELU. stride 2. layer norm 3. fully connected. 256 ELU. layer norm 4. recurrent. 250 Sigmoid. layer norm on the output 5. fully connected. 256 RELU. layer norm 6. fully connected. 10⇥ 10⇥ 32 RELU. layer norm 7. 4⇥ 4 reshape 2 nearest-neighbour, conv. 16 RELU. layer norm 8. 8⇥ 8 reshape 4 nearest-neighbour, conv. 1 Sigmoid
We used the Advantage Actor-Critic (A2C) Mnih et al. (2016) with an MLP policy as the decision making module of OPC. The MLP policy added a 512-unit fully connected layer with rectifier nonlinearity after layer 4 of the perception model. The decision making module had two set of outputs: 1) a softmax output with one entry per action representing the probability of selecting the action, and 2) a single linear output representing the value function. The decision making module was trained using RMSProp Tieleman and Hinton (2012) with a learning rate of 7e 4, a reward discount factor = 0.99, an RMSProp decay factor of 0.99, and performed updates after every 5 actions.
A2C We used the same convolutional architecture as the encoder of the perception model of OPC (layer 1 to 3), followed by a fully connected layer with 512 hidden units followed by a rectifier nonlinearity. The A2C was trained using the same setting as the decision making module of OPC.
WM-A2C We used the same setting as Ha and Schmidhuber (2018) to separately train the V model and the M model. The experience was generated off-line by a random policy operating in the Pixel Waterworld environment. We concatenated the output of the V model and the M model as the A2C input, and trained A2C using the same setting as introduced above.

heading: C DETAILS OF DERIVATION
text: C.1 DERIVATION OF EQ. (1)
logp(o t,xt|a<t)
= log X
s 1,a t,x>t
p(o t, s 1,x 1, a t|a<t)
= log X
s 1,a t,x>t
p(o t|s 1,x 1, a 1)p(s 1,x 1, a 1|a<t)
= log X
s 1,a t,x>t
Y
n=t
p(on|sn, an)p(xt|s 1,x>t, a 1)p(s 1,x>t, a 1|a<t)
= log X
s 1,a t,x>t
Y
n=t
p(on|sn, an) tY
j=1
p(xj |sj)p(x>t|s 1, a 1)p(s 1, a 1|a<t)
= log X
s 1,a t,x>t
Y
n=t
p(on|sn, an) tY
j=1
p(xj |sj) Y
k=t+1
p(xk|sk)p(s>t|st, a 1)p(st, a 1|a<t)
= log X
s 1,a t,x>t
Y
n=t
p(on|sn, an) tY
j=1
p(xj |sj) Y
k=t+1
p(xk|sk) Y
l=t
p(sl+1|sl, al)p(s1) t 1Y
m=1
p(sm+1|sm, am) Y
h=t
p(ah|a<t)
= log
2
6666664 X st tY j=1 p(xj |sj)p(s1) t 1Y m=1 p(sm+1|sm, am)
2
6666664 X s>t,a t,x>t Y n=t p(on|sn, an) Y k=t p(xk+1|sk+1)p(sk+1|sk, ak)p(ak|a<t)
| {z } 1
3
7777775
3
7777775
= log
2 4 X
st
q(st|xt, a<t) Qt j=1 p(x j |sj)p(s1) Qt 1 m=1 p(s
m+1|sm, am) q(st|xt, a<t) [ 1 ]
3
5
C.2 DERIVATION OF EQ. (4)
✓t+1 =argmax ✓t+1 qw log
Qt j=1 p(x j |sj)p(s1) Qt 1 m=1 p(s m+1|sm, am)
p(s1) Qt 1
g=1 q(s g+1|xg, ag)
= argmax ✓t+1
qw log p(s1)p(x1|s1)
Qt 1 j=1 p(x
j+1, sj+1|sj , aj) p(s1) Qt 1 g=1 q(s g+1|xg, ag)
= argmax ✓t+1
⌘t log p(x1|s1)
Qt 1 j=1 p(x
j+1, sj+1|sj , aj) Qt 1
g=1 ⌘ g+1
drop terms which are constant with respect to ✓t+1
=argmax ✓t+1 E zt+1⇠⌘t
⇥ log p (x t+1, zt+1, t+1|st, at) ⇤
. =argmax
✓t+1 ⇤✓t(✓
t+1)
C.3 DERIVATION OF EQ. (5)
@⇤✓t+1(✓ t+2)
@✓t+1k = @⇤✓t+1(✓
t+2) @ t+1k · @{ t+1k }T @{✓t+1k }T
=
" @⇤✓t+1(✓ t+2)
@ t+11,k . . .
@⇤✓t+1(✓ t+2)
@ t+1D,k
# ·
2
6664
@ t+11,k @✓t+1k
... @ t+1D,k @✓t+1k
3
7775
= DX
i=1
@⇤✓t+1(✓ t+2) @ t+1i,k · @ t+1i,k @✓t+1k
= DX
i=1
@
@ t+1i,k
2 4 X
zt+1i
p (z t+1 i |x t+1 i , t i , a t) log p t+1i (xt+1i , z t+1 i |s t, at)
3 5 · @ t+1i,k @✓t+1k
= DX
i=1
@
@ t+1i,k
2 4 X
zt+1i
p (z t+1 i |x t+1 i , t i , a t) log p t+1i (xt+1i |z t+1 i )p(z t+1 i |s t, at)
3 5 · @ t+1i,k @✓t+1k
= DX
i=1
@
@ t+1i,k
" KX
k=1
p (z t+1 i,k = 1|x t+1 i , t i,k, a t) log p t+1i,k (xt+1i |z t+1 i )p(z t+1 i |s t, at) # · @ t+1i,k @✓t+1k
= DX
i=1
p (z t+1 i,k = 1|x t+1 i , t i , a
t) @
@ t+1i,k
h log p t+1i,k (xt+1i |z t+1 i )p(z t+1 i |s t, at) i · @ t+1i,k @✓t+1k
= DX
i=1
p (z t+1 i,k = 1|x t+1 i , t i , a
t) @
@ t+1i,k
h log p t+1i,k (xt+1i |z t+1 i ) + log p(z t+1 i |s t, at) i · @ t+1i,k @✓t+1k
= DX
i=1
p (z t+1 i,k = 1|x t+1 i , t i , a
t) @
@ t+1i,k
h log p t+1i,k (xt+1i |z t+1 i ) i · @ t+1i,k @✓t+1k
= DX
i=1
p (z t+1 i,k = 1|x t+1 i , t i , a
t) @
@ t+1i,k
" log " 1p 2⇡ 2 exp
1 2
(xt+1i t+1 i,k ) 2
2
!## · @ t+1i,k @✓t+1k
= DX
i=1
p (z t+1 i,k = 1|x t+1 i , t i , a
t) @
@ t+1i,k
" log log p 2⇡ 1
2
(xt+1i t+1 i,k ) 2
2
# · @ t+1i,k @✓t+1k
= DX
i=1
p (z t+1 i,k = 1|x t+1 i , t i , a t) · 1 2 2(xt+1i t+1 i,k ) · ( 1) 2 · @ t+1i,k @✓t+1k
Eq. (3) =
DX
i=1
⌘ti,k · t+1i,k x t+1 i 2 · @ t+1i,k @✓t+1k .

Review summaries:
This paper proposes an extension of the RL as Inference framework, and demonstrates how to use it to express an object-centric RL model and train it on simple environments. It appears to be a combination of NEM [1] with a simple TD-learning objective on top. Results are a bit hard to interpret but seem promising.

The authors propose a framework for joint perception and control as inference (PCI) to combine perception and control for the case of POMDPs. The authors particularly focus on the case of hidden perceptual states linked to small image observations, which are composed of pixels belonging to up to exactly one object each. Their main proposal is denoted as OPC, which stands for object based perception and control, which serves the purpose of automatically discovering objects from pixels while controlling the system.