Contrastive Active Inference
Authors: Pietro Mazzaglia
Tim Verbelen
Bart Dhoedt
Venue: NeurIPS 2021 Poster (2021)
Keywords: active
inference
contrastive
learning
free
energy
control

Abstract: Active inference is a unifying theory for perception and action resting upon the idea that the brain maintains an internal model of the world by minimizing free energy. From a behavioral perspective, active inference agents can be seen as self-evidencing beings that act to fulfill their optimistic predictions, namely preferred outcomes or goals. In contrast, reinforcement learning requires human-designed rewards to accomplish any desired outcome. Although active inference could provide a more natural self-supervised objective for control, its applicability has been limited because of the shortcomings in scaling the approach to complex environments. In this work, we propose a contrastive objective for active inference that strongly reduces the computational burden in learning the agent's generative model and planning future actions. Our method performs notably better than likelihood-based active inference in image-based tasks, while also being computationally cheaper and easier to train. We compare to reinforcement learning agents that have access to human-designed reward functions, showing that our approach closely matches their performance. Finally, we also show that contrastive methods perform significantly better in the case of distractors in the environment and that our method is able to generalize goals to variations in the background.

Contrastive Active Inference
Pietro Mazzaglia
Tim Verbelen IDLaa Ghent University
Bart Dhoedt IDLab Ghent University bart. dhoed @ugent.
IDL.ab
Ghent University pietro www.mmzlllagagnntttomm
tim. www.ereeeeeeeet..eeebbb
Abstract
Active inference is unifying theory for perception and action resting upon the idea that the brain maintains an internal model of the world by minimizing free energy. From behavioral perspective, active inference agents can be seen self- evidencing beings that act to fulfill their optimistic predictions, namely preferred outcomes or goals. In contrast, reinforcement learning requires human-desse -deinndd re- wards to accomplish any desired outcome. Although active inference could provide a more natural self-supervised objective for control, its applicability has been lim- ited because of the shortcomings in scaling the approach to complex environments. In this work, we propose contrastive objective for active inference that strongly reduces the computational burden in learning the agent's generative model and planning future actions. Our method performs notably better than likelihood-basoe active inference in image-based tasks, while also being computationally cheaper and easier to train. We compare to reinforcement learning agents that have access to human designed reward functions, showing that our approach closely matches their performance. Finally. we also show that contrastive methods perform significantly better in the case of distractors in the environment and that method is able generalize goals to variations in the background.
Introduction
Deep Reinforcement Learning (RL) has led to successful results in several domains, such robotics,
video games and board games 42 36 2. From neuroscience perspective, the reward prediction error signal that drives learning in deep RI. closely relates to the neural activity of dopamine neurons for reward-bart based learning [44 3). However, the reward functions used in deep RL typically require domain and task-specific design from humans, spoiling the generalization capabilities of RL agents.
Furthermore, the possibility faulty reward functions makes the application of deep RL risky in
real-world contexts, given the possible unexpected behaviors that may derive from it 10 29 38
Active Inference (AIF) has recently emerged as a unifying framework for learning perception and action. In AIF, agents operate according to one absolute imperative: minimize their free energy [15]. With respect to past experience, this encourages to update an internal model of the world to maximize evidence with respect to sensory data. With regard to future actions, the inference process becomes "active" and agents select behaviors that fulfill optimistic predictions of their model, which are represented as preferred outcomes or goals [17]. Compared to RI., the AIF framework provides more natural way of encoding objectives for control. However, its applicability has been limited because of the shortcomings in scaling the approach to complex environments, and current implementations have focused on tasks with either low-dimensional sensory inputs and/or small sets discrete actions [[2). Moreover, several experiments in the literature have replaced the agent's preferred outcomes with RL-like rewards from the environment, downplaying the AIF potential to provide superviee
objectives B 34 49
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

One of the major shortcomings in scaling AIF to environments with sigh--imensional, e.g. image- based, environments comes from the necessity of building accurate models of the world, which try to reconstruct every detail in the sensory data. This complexity is also reflected in the control stage, when AIF agents compare future imaginary outcomes of potential actions with their goals, to select the most convenient behaviors. In particular, we advocate that fulfilling goals in image space can be poorly informative to build an objective for control.
In this work, we propose Contrastive Active Inference, a framework for AIF that aims to both reduce the complexity of the agent's internal model and propose a more suitable objective to fulfill preferred outcomes, by exploiting contrastive learning. Our method provides self-supervised objective that constantly informs the agent about the distance from its goal, without needing to reconstruct the outputs of potential actions in nigh-dimensional image space.
The contributions of our work be summarised follows: (i) propose a framework for AIF that drastically reduces the computational power required both for learning the model and planning future actions, (ii) we combine our method with value iteration methods for planning, inspired by the RL literature, to amortize the cost of planning in AIF, (iii) we compare our framework to state-of-the-arl RL techniques and a non-contrastive AIF formulation, showing that our method compares well with reward-based systems and outperforms non-contrastive AIF, (iv) we show that contrastive methods work better than reconstruction- based methods presence of distractors in the environment, (v) we found that our contrastive objective for control allows matching desired goals, despite differences in the backgrounds. The latter finding could have important consequences for deploying AIF in real-world settings, such robotics, where perfectly reconstructing observations from the environment and matching them with high-dimensional preferences is practically unfeasible.
2 Background
The control setting can be formalized as a Partially Observable Markov Decision Process (POMDP), which is denoted with the tuple M = A, T,T,T,S. T, S,O,,,,,,, 0,7}, where S is the set of un- observed states, the set of actions, 7' is the state transition function, also referred to as the dynamics of the environment, Ω the set observations, O a of conditional observation probabilities, and by is discount factor (Figure TT. We use the terms observations and outcomes interchangeably throughout the work. In RL, the agent has also access to a reward function R. mapping state-action pairs rewards.
Figure 1: POMDP Graphical Model
Active Inference. In AIF, the goal of the agent is to minimize (a variational bound on) the surprisal observations, log p(o). With respect to past observations, the upper bound leads to the variational free energy F, which for timestep f is:
F Feg(s)) [log (*::) log p(o,, 8t)] 2 - logp(or)
where ((x)) represents an approximate posterior.
The agent hence builds a generative model over states, actions and observations, by defining a state transition function pl p(se|se-1, and a likelihood mapping p(or]st), while the posterior distribution over states is approximated by the variational distribution (()))o)). The free energy can then be decomposed as:
FAIF
DKL. [g(s:/o)\llss/ss-1.... a11] Poggslo)
complexity
accuracy
This implics that minimizing variational free energy, on the one hand, maximizes the likelihood of observations under the likelihood mapping (i.e. maximizing accuracy), whilst minimizing the KL divergence between the approximate posterior and prior (i.e. complexity). Note that for the past we assume that outcomes and actions observed, hence only inferences are made about the state st- Also note that the variational free energy is defined as the negative evidence lower bound as known from the variational autoencoder framework (39 27
For future timesteps, the agent has to make inferences about both future states and actions g(se, a:) = q(a, (s) (s))),, while taking into account expectations over future observations. Crucially, in active

---

the expected free inference the agent has prior distribution p(oz) on preferred outcomes it expects to obtain. Action selection then cast an inference problem, i.c. inferring actions that will yield preferred outcomes, or more formally that minimize the expected free energy G:
G ,. [log (( ax) log plor, S1, az)],
(3)
where p(o,, St, a4) p(a:) p( ((o:)) the agent's 's biased generative model, and the expectation is over predicted observations, states and actions q(0,, $1, at) = ( oz).
If assume the variational posterior states is good approximation of the true posterior, i.e.
(0x) Rs p( (e), and we also consider uniform prior p(a+) over actions 35
energy can be formulated as:
GAUF
Fo(o (DRL [+(5)1o)))((ss)]]
Pogo:)loogpooi ((o)) E Ea(s.) ((tg)))((((())s)))
(4)
intrinsic value
extrinsic value
action entropy
Intuitively, this means that the agent will infer actions for which observations have a high information gain about the states (i.e. intrinsic value), which will yield preferred outcomes (i.e. extrinsic value), while also keeping its possible actions as varied as possible (i.e. action entropy).
Full derivations of the equations in this section provided in the Appendix.
Reinforcement Learning. In RL, the objective of the agent is to maximize the discounted sum of rewards, or return, over time Le 7'r,- Deep RL can also be cast as probabilistic inference, by introducing an optimality variable O, which denotes whether the time step optimal (30].
The distribution over the optimality variable is defined in terms of rewards as p(O, = 1 1 ax)
exp(r(st,a)) a:). Inference then obtained by optimizing the following variational lower bound
(ogp(O,) S Ex() [log ((s,, az) log p(,, St, az)] - E. a:)] gat..) Egt..) (H([([)ss))))
(5)
where the reward- maximizing RL objective is augmented with an action entropy term, in maximum entropy control [20]. As also highlighted (35.. if we assume plo (sy) log ((O))s)) we can see that RL works alike AIF, but encoding optimality value in the likelihood rather than in the prior.
In order to improve sample-effiienncy of RL., model-based approaches (MBRL), where the agent relies on an internal model of the environment to plan high-rewarding actions, have been studied.
Contrastive Learning. Contrastive representations, which aim to organize the data distinguishing similar and dissimilar pairs, can be learned through Noise Contrastive Estimation (NCE) [19]. Following (37], NCE loss be defined lower bound on the mutual information between two variables. Given two random variables X and Y, the NCE lower bound is:
K log
T(X;Y) JNCE A E
Σ
(6)
K
E (
1=1
K
where the expectation is over K independent samples from the joint distribution: II, p(zj, y5) and
f(x,, y) is a function, called critic, that approximates the log density ratio log p(a y)
piz) Crucially, the
critic can be unbounded, as in (50), where the authors showed that an inner product of transformated
samples from X and Y, namely f(x y) h(x)) g(y), with and g functions, works well as a critic.
Contrastive Active Inference
In this section, present the Contrastive Active Inference framework, which reformulates the problem of optimizing the free energy of the past F and the expected free energy of the future G as contrastive learning problems.
3.1 Contrastive Free Energy of the Past
In order to learn a generative model the environment following AIF, an agent could minimize the variational free energy FAIF from Equation 2 For high-dimensional signals, such as pixel-based

---

images, the model works similarly to Variational AutoEncoder (VAE) [27], with the informa- tion encoded in the latent state s, being used to produce reconstructions of the lighdimmnssionll observations through the likelihood model.
However, reconstructing images pixel level has several shortfalls: (a) it requires models with high capacity, (b) it can be quite computationally expensive, and (c) there is the risk that most of the representation capacity is wasted on complex details of the images that irrelevant for the task.
We can avoid predicting observations, by using an NCE loss. Optimizing the mutual information between states and observations, it becomes possible to infer s; from o,, without having to compute reconstruction. In order to turn the variational free energy loss into contrastive loss, we add the constant marginal log probability of the data log p(o)) to F, obtaining:
F DRL [q( [{(sclo))llp<ss) (o.) [lo p(o:(s)) log p(o:)
DxL [q(s [[)p((s))) T(S,; O,).
As for Equation 6 we can apply lower bound on the mutual information [(5;; Oz). We can define the contrastive free energy of the past as:
FNCE DKL (a(s:lo=)]lp(s.) INCE(S,; O+)
Dxi. (<(xs))
Be(srlo.)por) (or, st) + Egger oo) &े Ex= (o)))
where the dynamics p($²) is modelled as St-1, 0--), and the K samples from the distribution p(o') represent observations that do not match with the state st, catalyzing the contrastive mechanism. Given the inequality INCE S L, this contrastive utility provides an upper bound on the variational free energy. F S FACE, and thus on suprisal.
p(s)
3.2 Contrastive Free Energy of the Future
Performing active inference for action selection means inferring actions that realize preferred out- comes, by minimizing the expected free energy G. In order to assess how likely expected future outcomes are to fulfill the agent's preferences, in Equation 4, the agent uses its generative model to predict future observations.
Reconstructing imaginary observations in the future can be computationally expensive. Furthermore, matching imagined outcomes with the agent's preferences in pixel space can be poorly informative, as pixels are not supposed to capture any semanties about observations. Also, observations that are "far" in pixel space aren't necessarily far in transition space. For example, when the goal is behind door, standing before the door "far" pixel space but only one action away (i.e. opening the door).
When the agent learns a contrastive model of the world, following Equation g] it can exploit its ability match observations with states without reconstructions, in order search for the states that correspond with its preferences. Hence, we formulate the expectation in the expected free energy G in terms of the preferred outcomes, so that we can add the constant marginal ((o,), obtaining:
G - F((or)((er.a)) [log g( a.) log p(o,, 8t, aq) log ((o))) DxL (g(s)))p((s) - I(St Ô,) - Fe(s.)) [H(g
With abuse of notation, the mutual information between S, and Ô, quantifies the amount of informa- tion shared between future imaginary states and preferred outcomes.
We further assume DRL. (())))(((s))) preventing it to change the dynamics of the world to accomplish its goal, as pointed out 30 This leads to the following objective for the contrastive free energy of the future:
which constrains the agent only modify its actions,
ONCE = INcE Ô.) F
(10)
-Bg(,)))( (f st)] + Ex(e))(o flog से Σソ [f(os+)))) Eq(a.) [H(g(ar(s)))
Similar as in the JNCF, the K samples from p(o') foster the contrastive mechanism, ensuring that the state corresponds to the preferred outcomes, while also being as distinguishable possible from other observations. This component implies a similar process as to the ambiguity minimization aspect typically associated with the AIF framework [16].

---

Model and Algorithm
The AIF framework entails perception and action, in unified view. In practice, this is translated into learning a world model, to capture the underlying dynamics of the environment, minimizing the free energy of the past, and learning behavior model, which proposes actions to accomplish the agent's 's preferences, minimizing the free energy of the future. In this work, we exploit the high expressiveness of deep neural networks to learn the world and the behavior models.
The world model is composed by the following components:
Prior network:
ac-1) (8 a2-
Posterior network:
Representation model:
fo(o,
For the prior network, we use a GRU [9] while the posterior network combines a GRU with a CNN process observations. Both the prior and the posterior outputs are used to parameterize Gaussian multivariate distributions, which represent a stochastic state, from which we sample using the reparameterization trick (77. This setup is inspired upon the models presented in [21 54 4. For the representation model, we utilize a network that first processes and s, with MLPs and then computes the dot-product between the outputs, obtaining fo(o, = ho(o)" go(s), analogously to 50) We indicate the unified world model loss with: JΦ L, FNCE(SE, 0z).
In order to amortize the cost of long-term planning for behavior learning, we use an expected utility function ((s)) to estimate the expected free energy in the future for the state st, similarly to [34). The behavior model is then composed by the following components:
Action network:
go va(ae)8
Expected utility network:
go (st)
where the action and expected utility networks are both MLPs that are concurrently trained in actor-critic architectures for RL [28 20]. The action model aims to minimize the expected utility, which is an estimate of the expected free energy of the future over potentially infinite horizon, while the utility network aims to predict a good estimate of the expected free energy of the future that is obtainable by following the actions of the action network. We indicate the action network loss with Jø = E. (NCE(s)) and the utility network loss with Ju = E, (ge (s:) Σk=r (NCE($))) where the sum from the current time step to an infinite horizon obtained by using TD(A) exponentially-weigheed estimator that trades off bias and variance 43 (details in Appendix).
The training routine, which alternates updates to the models with data collection, is shown in Algorithm[] | At each training iteration of the model, we sample B trajectories of length L from the replay buffer D. Negative samples for the contrastive functionals are selected, for each state, by taking L - 1 intra-episode negatives, corresponding to temporally different observations, and (B - 1) L extra-episode negatives, corresponding to observations from different episodes.
Most of the above choices, along with the training routine itself, are deliberately inspired to current state-of-the-ee aat--artaa approaches for MBRL 23 22 II]. The motivation behind this is twofold: on the one hand, we want to show that approaches that have been used to scale RL for complex planning can also straightforwardly be applied for scaling AIF. On the other hand, in the next section, we offer direct comparison to current state-of-the -fet-------tttttt techniques for RL that, being unbiased with respect to the models' architecture and the training routine, can focus on the relevant contributions of this paper, which concerns the contrastive functionals for perception and action.
Relevant parameterization for the experiments can be found in the next section, while hyperparameters and detailed description of each network are left to the Appendix.
5 Experiments
In this section, we compare the contrastive AIF method to tikelihood-based AIF and MBRL in high-dimensional image-based settings. As the experiments are based in environments originally designed for RL, we defined ad-hoc preferred outcomes for AIF. Our experimentation aims answer the following questions: (i) possible to achieve high-dimensional goals with AlF-based methods? (ii) what is the difference in performance between RL-based and AlF-based methods? (iii) does
5

---

Algorithm 1: Training and Data Collection
I: Initialize world model parameters & and behavior model parameters and t/.
2: Initialize dataset 'D with R random-action episodes.
3: while not done do
4: for update step u=1..U do
5: Sample trajectories of length L from D.
6:
Infer states & using the world model. Update the world model parameters 6 on the trajectories, minimizing Ce. Imagine I trajectories of length H from each 8t.
7:
8:
9:
Update the action network parameters on the trajectories, minimizing fio- Update the expected utility network parameters ib the trajectories, minimizing Lyp. end
10:
11:
12:
Reset the environment.
13:
Init state 8t = So and set t= Init new trajectory with the first observation T }
14:
15:
while environment not done do
16:
Infer action Gr using the action network qo (ar)ss)))
17:
Act the environment with at, and receive observation O++11 Add transition to the buffer T = 'TU {at, 0t+1 and set t= Infer state 8, using the world model
18:
19:
20:
end
21:
end
contrastive AIF perform better than likelihood- based AIF? (iv) in what contexts contrastive methods are more desirable than likelihood-baat based methods? (v) are AlF-based methods resilient to variations in the environment background?
We compare the following four flavors of MBRL and AIF, sharing similar model architectures and all trained according to Algorithm[]
Dreamer: the agents build a world model able to reconstruct both observations and rewards from the state. Reconstructed rewards for imagined trajectories are then used to optimize the behavior model in an MBRL fashion [3 22].
Contrastive Dreamer: this method is analog to its reconstruction-bunn based counterpart, apart from that it uses contrastive representation model, like our approach. Similar methods
have been studied in [23 32).
Likelihood-AIf: the agent minimizes the AIF functionals, using observation reconstructions. The representation model from the previous section is replaced with an observation like- lihood model (o (8,), which we model as transposed CNN. Similar approaches have been presented in 13
Contrastive-AIF (ours): the agent minimizes the contrastive free energy functionals.
In Table I] we compare the number of parameters and of multiply-accumulate (MAC) operations required for the two flavors of the representation model in our implementation: likelihood- based and contrastive (ours). Using contrastive representation makes the model 13.8 times more efficient in terms of MAC operations and reduces the number of parameters by factor 3.5.
In Table 2 we compare the computation speed in our experiments, measuring wall-clock time and average 16% faster, while Likelihood-AlF:
using Dreamer reference. Contrastive methods which in addition to Dreamer reconstructs observations for behavior learning, is 224% slower.
Table 1: Computational Efficiency
Table 2: Computation Time
MMACs
Params
w.r.t. Dreamer
Likelihood
212.2
4485.7k
Contrastive Dreamer/AlF
0.84
Ours
15.4
1266.7k 7k
Likelihood-Ale
3.24

---

6x6 Empty
Empty
1.0
0.8
Dreamer
Rewards
0.6
Contrastive Dreamer
Likelihood-AIF
0.4
Contrestive-AIF (ours)
0.2
0.0
100
200
300
400
200
400
600
800
1003
Trajectories
Figure 2: MiniGrid Experiments. (left) Empty task goal image. (right) Results: shaded areas
indicate standard deviation across several runs.
5.1 MiniGrid Navigation
We performed experiments on the Empty 6x6 and the Empty 8x8 environments from the MiniGrid suite [8] these tasks, the agent, represented as a arrow, should reach the goal green square navigating black grid (see Figure Ga). The agent only sees part of the environment, corresponding 7x7 grid centered on the agent (in the bottom center tile). We render observations as 64 64 pixels. For RL, a positive reward between 0 and I is provided the agent soon the agent reaches the goal tile: the faster the agent reaches the goal, the higher the reward. For AIF agents, we defined the preferred outcome as the agent seeing itself on the goal green tile, as shown in Figure 2 (left).
For the 6x6 task, the world model is trained by sampling B 50 trajectories length while the behavior model is trained by imagining H = 6 steps long trajectories. For the 8x8 task, we increased the length collected R 50 random episodes, to populate the replay buffer, and train for U 100 steps after collecting new trajectory. Being the action set discrete, we optimized the action network employing REINFORCE gradients 122 with respect to the expected utility network's s estimates.
11 and the imagination horizon H 10. For both tasks, we first
We assess performance terms the rewards achieved along one trajectory, stressing that AIF methods did not have access to the reward function but only to the goal observation, during training. The results, displayed in Figure 2 (right), show the average sum of rewards obtained along training, over the number of trajectories collected. We chose to compare over the number of trajectories as the trajectories' length depends on whether the agent completed the task or
In this benchmark, we see that MBRL algorithms rapidly converge to highly rewarding trajectories, in both the 6x6 and the 8x8 tasks. Likelihood-AIF struggles converge trajectories that reach the goal consistently and fast, mostly achieving a reward mean lower than 0..4 In contrast, our method performs comparably to the MBRL methods in the 6x6 grid and reaches the goal twice more consistently than Likelihood-AlF in the x grid, leaning towards Dreamer and Contrastive Dreamer's results.
Utility Function Analysis. In order to understand the differences between the utility functions we experimented with, analyze the values assigned to each tile in the 8x8 task by every method.
(a) Grid Task
(b) Rewards
(c) Model
(d) NCE Model (ours)
Figure 3: Utility Values MiniGrid. (b-c-d) Darker tiles correspond to higher utility values.

---

(a) Reacher Easy Goal
(b) Reacher Hard Goal
(c) Distracting Reacher Easy
Figure 4: Continuous tasks setup. Note that the Reacher Easy Goal is also used for the Distracting Reacher Easy task, without changing the goal's background.
For the AIF methods, we collected all possible transitions in the environment and used the model to compute utility values for each tile. The results are shown in Figure 3
The reward signal for the Empty environment is very sparse and informative only once the agent reaches the goal. In contrast, AIF methods provide denser utility values. particular, we noticed that the Likelihood-AlP model provides a very strong signal for the goal position, whereas other values are less informative of the goal. Instead, the Contrastive- AIF model seems to capture some semantic information about the environment: it assigns high values to all corners, which are conceptually closer outcomes to the goal, while also providing the steepest signal for the green corner and its neighbor tiles. As also supported by the results obtained in terms of rewards, our method provides denser and more informative signal to reach the goal in this task.
5.2 Reacher Task
We performed continuous-contro experiments on the Reacher Easy and Hard tasks from the Deep- Mind Control (DMC) Suite [48] and on Reacher Easy from the Distracting Control Suite 47]. In this task, a two-link arm should penetrate goal sphere with its tip in order to obtain rewards, with the sphere being bigger the Easy task and smaller in the Hard one. The Distracting Suite adds an extra layer of complexity to the environment, altering the camera angle, the arm and the goal colors, and the background. In particular, we used the "casy" version of this benchmark, corresponding to smaller changes in the camera angles and in the colors, and choosing the background from one of four vidcos (example in Figure 4c).
In order to provide consistent goals for the AIF agents, we fixed the goal sphere position as shown in Figure 4 and 4a As there is no fixed background in the Distracting Suite task, we could not use a goal image with the correct background, as that would have meant changing it at every trajectory. To not introduce 'external' interventions into the AIF experiments, we decided to use goal image with the original blue background from the DMC Suite to test out the AIF capability to generalize goals to environments having the same dynamics but different backgrounds.
For both tasks, the world model is trained by sampling B = 30 trajectories of length L = 30, while the behavior model is trained by imagining H 10 steps long trajectories. We first collect R = 50 random episodes, to populate the replay buffer, and train for U = 100 steps after every new trajectory. Being the action set continuous, we optimized the action network backpropagating the expected utility value through the dynamics, by using the reparameterization trick for sampling actions
[33
The results are presented in Figure 5 evaluating agents in term of the rewards obtained per trajectory. The length of a trajectory is fixed to 1-103 steps.
Reacher Easy/Hard. The results on the Reacher Easy and Hard tasks show that our method was the fastest to converge to stable high rewards, with Contrastive Dreamer and Dreamer following. In particular, Dreamer's delay to convergence should be associated with the more complex model, that took more epochs training than the contrastive ones to provide good imagined trajectories for planning, especially for the Hard task. The Likelihood-Ar AIF failed to converge in all runs, because of the difficulty of matching the goal state in pixel space, which only differs a small number of pixels from any other environment observation.
Distracting Reacher Easy. On the Distracting task, we found that Dreamer failed to succeed. As we show in Appendix, the reconstruction model's capacity was entirely spent reconstructing the complex backgrounds, failing to capture relevant information for the task. Conversely, Contrastive Dreamer able to ignore the complexity of the observations and the distractions present in the
8

---

Easy
Distracting Easy
1000
800
820
600
600
Contresiive Dreamer
Likalihond-AIF
400
400
400
Contrastive AlF (ours)
100
Trajectonies
Figure 5: Reacher Results. Shaded areas indicate standard deviation across several runs.
environment, eventually succeeding in the task. Surprisingly, also our Contrastive-AAe method was able to succeed, showing generalization capabilities that are not shared by the likelihood counterpart.
We believe this result is important for two better capture semantic information the environment, potentially ignoring complex irrelevant details, (2) contrastive objectives for planning be invariant to changes in the background, when the underlying dynamics of the task stays the same.
it provides evidence that contrastive features
Utility Function Analysis. To collect further insights the different methods' objectíves, analyze the utility values assigned observations with different poses in the Reacher Hard task. In Figure show comparison where all the values normalized in the range [0, 1], considering the maximum and minimum values achievable by each method.
The reward signal is sparse and provided only when the arm is penetrating the goal sphere with his orange tip. In particular, a reward of +1 is obtained only when the tip is entirely contained in the sphere. The Likelihood-AIP utility looks very flat due to the static background, which causes any observation to very similar the preferred outcome in pixel space. Even a pose that is very different from the goal, such as the top left one, is separated only by a relatively small number of pixels from the goal one, in the bottom right corner, and this translates into very minor differences in utility values (i.e. 0.98 vs 1.00). For Contrastive-AlI: we see that the model provides higher utility values for observations that look perceptually similar to the goal and lower values for more distant states, providing denser signal optimize for reaching the goal. This was certainly crucial in achieving the task this experiment, though overly-shaper utility functions can be more difficult to optimize [I] and future work should analyze the consequences of such dense shaping.
84
08
68
66
06
1.0
84
88
48
66
06
Rewards
AIF model
NCE model (ours)
Figure 6: Utility Values Reacher. Normalized utility values for multiple poses in Reacher Hard.

---

Related Work
Contrastive Learning. Contrastive learning methods have recently led to important breakthroughs
in the unsupervised learning landscape. Techniques like MoCO 7 24 and SimCLR [5] 6 have progressively improved performance in image recognition, by using only a few supervised labels. Contrastive learning representations have also shown successful when employed for natural language processing (5) and model-free RI. [46].
Model-based Control. Improvements in the dynamics generative model 21 have recently allowed model-based RL methods to reach state-of-the-art performance, both in control tasks [23] and on video games [22 26). An important line of research focuses on correctly balancing real-world
experience with data generated from the internal model of the agent [25
Outcome-Driven Control. The idea of using desired outcomes to generate control objectives has been explored in RL as well [1] 18 40). In [I], the authors propose a system that, given a desired goal, can sample plans of action from latent space and decode them act on the environment. DISCERN [51] maximizes mutual information to the goal, using cosine similarity between the goal and given observation, in the feature space of CNN model.
Active Inference. In our work, we used active inference to derive actions, which is just one possibility to perform AIF, discussed in [14 335. In other works, the expected free energy is passively used as the utility function to select the best behavior among potential sequences of actions [5] [G. Methods that combine the expressiveness of neural networks with AIF have been raising in popularity in the last years [53] In [3], the authors propose an amortized version of Monte Carlo Tree Search, through an habit network, for planning. In [49], AIF is seen performing better than RL algorithms in terms of reward maximization and exploration, on scall-scale tasks. In [34, they propose an objective amortize planning value iteration fashion.
7 Discussion
We presented the Contrastive Active Inference framework, contrastive learning approach for active inference, that casts the free energy minimization imperatives of AIF as contrastive learning problems. We derived the contrastive objective functionals and we corroborated their applicability through empirical experimentation, in both continuous and discrete action settings, with high-dimensional observations. Combining our method with models and learning routines inspired from the model- based RL scene, we found that our approach can perform comparably to models that have access to human-designed rewards. Our results show that contrastive features better capture relevant information about the dynamics of the task, which can be exploited both to find conceptually similar states preferred outcomes and make the agent's preferences invariant irrelevant changes in the environment (e.g. background, colors, camera angle).
While the possibility to match states outcomes in terms of similar features is rather convenient in image-based tasks, the risk is that, if the agent never saw the desired outcome, it would converge to the semantically closest state in the environment that it knows. This raises important concerns about the necessity to provide good exploratory data about the environment, in order to prevent the agent from hanging in local minima. For this reason, we aim to look into combining our agent with exploration-driven data collection, for zero-shot goal achievement [33 45). Another complementrry line of research would be equipping our method with better experience replay mechanisms, such as HER [T]. to improve the generalization capabilities of the system.
Broader impact
Active inference is teloogccalll-llussbbe unifying theory for perception and action. Implementa- tions of active inference that are both tractable and computationally cheap are important to foster further research towards potentially better theories of the human brain. By strongly reducing the computational requirements of our system, compared to other deep active inference inpleementaiinss, we aim to make the study of this framework more accessible. Furthermore, our successful results on the robotic manipulator task with varying realistic backgrounds show that contrastive methods are promising for real- world applications with complex observations and distracting elements.
10

---

Acknowledgments and Disclosure of Funding
This research received funding from the Flemish Government (Al Research Program).
References
[1] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin,
O. Pieter Abbeel, and W. Zaremba. Hindsight experience replay. In I. Guyon, U. V. Luxburg. S. Ben- gio, H. Wallach. R. Fergus. S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[2] A. P. Badia, B. Piot, S. Kapturowski, P. Sprechmann, A. Vitvitskyi, D. Guo, and C. Blundell. Agent57: Outperforming the atari human benchmark, 2020.
[3] R. Bogacz. Dopamine role in learning and action inference. eliff,, 9se53262, jul 2020.
[4] I.. Buesing, T. Weher, S. Racamiere, S. M. A. Eslami, D. Rezende, D. P. Reichert, F. Viola, F. Besse, K. Gregor, D. Hassabis, and D. Wierstra. Learning and querying fast generative models for reinforcement learning. 2018.
[5] T. Chen, S. Komblithh, M. Norouzi, and G. Hinton, A simple framework for contrastive learning visual representations. arXiv preprint 2001::2002.05009, 2020.
[6] T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. Hinton. Big self-supervixed models are strong semi-supervised learners. arXiv preprint arXiv:2006. 10029, 2020.
[7] X. Chen, H. Fan, Girshick, and K. Hc. Improved baselines with momentum contrastive learning, 2020.
[8] M. Chevalier-Boisvert, L. Willems, and S. Pal. Minimalistic gridworld environment for openai gym. httpps///gtthb...ooox/xxxxmymyy wwtp:///////e/nm////e///mea wwwpeee/b/ymmmminiiidi... 2018.
Chung, Gulcehre, K. Cho, and Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. In NIPS 2014 Workshop on Deep Learning, December 2014, 2014.
[10] J. Clark and D. Amodei. Faulty reward functions in the wild, 2016.
[11] Clavera, V. Fu, and P. Abbeel. Model-augmented actor-critic: Backpropagating through paths, 2020.
[12] L. Da Costa, T. Parr, N. Sajid, Veselic, V. Neacsu, and Friston. Active inference on discrete state-spaces: synthesis, Journal of Mathematical Psychology, 99:102447, 2020.
[13] Z. Fountas, N. Sajid, P. Mediano, and K. Friston. Deep active inference agents using monte-carlo methods. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Processing Systems, volume 33, pages 11662-11675. Curran Associates, Inc., 2020.
in Neural Information
[14] K. Friston, L. Da Costa, D. Hafner, C. Hesp, and T. Parr. Sophisticated Inference. Neural Computation. 33(3):713-76 17--731113677,.. 03 2021.
[15] K. Friston, T. FitzGerald, F. Rigoli, P. Schwartenbeck, J. O'Doherty, and G. Pezzulo. Active inference and learning. Neuroscience Biobehavioral Reviews, 68:862 879, 2016.
[16] K. Friston, F. Rigoli, D. Ognibene, C. Mathys, T. Fitzgerald, and G. Pezzulo. Active inference and epistemic value. Cogn Neurosci, 6(4):187-211 2015.
[17] K. J. Friston, J. Daunizeau, J. Kilner, and Kiebel. Action and behavior: free-energy formulation. Biological Cybernetics, 102(3):227-266 Mar 2010.
[18] Y. Ganin, T. Kulkarni, I. Babuschkin, S. M. A. Eslami, and O. Vinyals. Synthesizing programs for images using reinforced adversarial learning. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research. PMLR, 2018
[19] Gutmann and A. Hyvärinen. Noise-contrastive estimation: new estimation principle unnormalized statistical models. In Y. W. Tch and M. Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume of Proceedings of Machine Learning Research, pages 297-304, Chia Laguna Resort, Sardinia. Italy, 13 15 May 2010. PMLR.
11

---

[20] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Dy and A. Krause, editors. Proceedings of the 35th International Conference on Machine Learning, volume 80 Proceedings of Machine Learning Research. pages 1861-1870, PMLR, 10-15 Jul 2018.
[21] D. Hafner, T. Lillicrap. I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson, Learning latent dynamics for planning from pixels. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning. volume 97 of Proceedings of Machine Learning Research, pages 3555-2565. PMLR, 09-15 Jun 2019.
[22] D. Hafner, T. Lillierap, M. Norouzi, and J. Ba. Mastering atari with discrete world models, 2021.
[23] D. Hafner, T. P. Lillicrap. J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. In ICLR, 2020.
[24] K. He, H. Fan. Y. Wu. S. Xie, and B. Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. pages 9726-9735. IEEE, 2020.
[25] Janner, Fu, Zhang. and S. Levine. When to trust your model: Model-based policy optimization. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[26] L. Kaiser, M. Babacizadeh, P. Milos, B. Osinski, R. H. Camphell, K. Czechowski, D. Erhan, C. Finn, P. Kozakowski, Levine, A. Mohiuddin, Sepassi, G. Tucker, and H. Michalewski. Model-based reinforcement learning for atari, 2020.
[27] D. Kingma and M. Welling. Auto-cncoding variational bayes, 2014.
[28] V. R. Konda and J. N. Tsitsiklis. Actor-critic algorithms. In Advances in neural information processing systems, pages 1008-1014. Citeseer, 2000.
[29] V. Krakovna et al. Specification gaming: the flip side of ai ingenuity, 2020.
[30] S. Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review, 2018.
[31] C. Lynch, Khansari, Xiao, V. Kumar. J. Tompson, Levine, and P. Sermanet. Learning latent plans from play, 2019.
[32] X. Ma, S. Chen, D. Hsu, and W. S. Lee. Contrastive variational model-based reinforcement learning for complex observations. In Proceedings of the 4th Conference on Robot Learning, 2020.
[33] P. Mazzaglia, O. Catal, T. Verbelen, and B. Dhoedt. Self-supervised exploration via latent bayesian surprise, 2021.
[34] B. Millidge. Deep active inference as variational policy gradients, 2019.
[35] B. Millidge, A. Tschantz, A. K. Seth, and C. Buckley. On relationship between active inference and control as inference, 2020.
[36] OpenAI, I. Akkaya, M. Andrychowicz, M. Chociej. M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plap- pert, G. Powell, R. Ribas, J. Schneider, N. Tezak, J. Tworek, P. Welinder, I.. Weng, Q. Yuan, W. Zaremba, and L.. Zhang. Solving rubik's cube with robot hand, 2019.
[37] B. Poole, Ozair, A. Den Oord, A. Alemi, and G. Tucker. variational bounds of mutual information. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages PMLR, 09 15 Jun 2019.
[38] 1. Popov al. Data-efficient deep reinforcement learning for dexterous manipulation. 2017.
[39] D. Rezende, S. Mohamed, and Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31 International Conference on Machine Learning (ICML), volume 32, pages 1278-1286, 2014.
[40] T. G. J. Rudner, V. H. Pong. R. McAllister, Y. Gal, and S. Levine. cattomee-diver reinforcement learning via variational inference, 2021.
12

---

[41] T. Schaul, D. Horgan, K. Gregor, and D. Silver. Universal value function approximators. In F. Bach and D. Blei, editors, Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1312-1320, Lille, France, 07-09 Jul 2015. PMLR.
[42] J. Schrittwieser. 1. Antonoglou, T. Hubert, K. Simonyan, L. Sifre. S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, I. Graepel, and al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839))604-608, Dec 2020.
[43] Schulman, Moritz, Levine, Jordan, and Abbeel. High-dimensional continuous control using generalizeed advantage estimation, 2018.
[44]
Schultz. Predictive reward signal of dopamine neurons. Neurophysiol, 80(1):1-27, Jul 1998.
[45] R. Sekar, O. Rybkin, K. Daniilidis, Abbeel, D. Hafner, and D. Pathak. Planning explore via self- supervised world models. ICML, 2020.
[46] A. Srinivas, M. Laskin, and P. Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning, 2020.
[47] A. Stone, O. Ramirez, K. Konolige, and R. Jonschkowski. The distracting control suite challenging benchmark for reinforcement learning from pixels, 2021.
[48] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. de Las Casas, Budden, A. Abdolmaleki, J. Merel,
A. Lefraneq, Lillicrap, and M. Riedmiller. Deepmind control suite, 2018.
[49] A. Tschantz, Millidge, A. Seth, and Buckley. Reinforcement learning through active inference, 2020.
[50] A. den Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding, 2019.
[51] D. Warde-Farley, T. V. de Wiele, T. D. Kulkarni, C. Ionescu, S. Hansen, and V. Mnih. Unsupervised control through non-parann non-parametric discriminative rewards. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.nee 2019.
[52] R. J. Williams. Simple statistical folliint algorithms for connectionist reinoorcmmet learning. Mach. Learn., 8(3-4):229-2555) May 1992.
[53] O. Çatal, T. Verbelen, Nauta, C. D. Boom, and B. Dhoedt. Learning perception and planning with deep active inference. In ICASSP 2020 2020 IEEE International Conference on Acoustics. Speech and Signal Processing (ICASSP), pages 3952-3956, 2020.
[54] O. Çatal, S. Wauthier, C. De Boom, T. Verbelen, and B. Dhoedt. Learning generative state space models for active inference. Frontiers in Computational Neuroscience, 14:103, 2020.
13


Reviews:
[{"review_id": "xBxXue_vZOa", "reviewer": "Reviewer_FFRc", "summary": "The authors work in the area of active inference, using a contrastive function in optimization to enhance scalability.  They show that their method significantly outperforms non-contrastive active inference on their tasks, and are competitive with some other RL approaches.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The modifications made in this paper to standard active inference are not complicated, but they seem novel and effective.  This could be a significant result in making active inference more practicable, and it seems to merit acceptance.\n\nLine 92: \"good approximate\" -> \"good approximation\"\nLine 109: \"that aim\" -> \"which aim\"\nLine 146: \"as pixel\" -> \"as pixels\"\nLine 198: don't capitalize \"Section\"\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors describe limitations of their work in Section 7.\n\nThey state in the Checklist that they do not see potential negative impacts, but do not address it in the main body of the manuscript.\n", "review_text": "", "extra_scores": {"ethical_concerns": "I don't see any ethical concerns in this work.\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "eTHXbh5d7-s", "reviewer": "Reviewer_WaHE", "summary": "This paper introduces a novel variation of the “Active Inference Model” that by incorporating contrastive learning instead of likelihood-based learning, makes the learning more computationally tractable and not reliant on high-dimensional observation reconstriction.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Finding alternative methods to reward learning is an important research direction that could be fruitful in having more generalizable and biologically plausible models. In addition, recently there have been very promising results in using contrastive learning in a wide array of domains. And this work takes advantage of those recent advances in contrastive learning to push forward the active inference model. The paper is well written and it is a pleasure to read. And the authors do discuss some of the drawbacks of their work.\nThe main shortcoming of the paper is its experiments section, which is limited to only two simple tasks. Would be interesting to see the performance in a wider array of tasks and more in-depth analysis. \n\n\nSpecifying desired outcomes might pose its own challenges similar to those of reward design. Could you please elaborate more on that?\n\nCould you please describe more in detail how $\\tilde{p}(o_t)$ formulated for each task?\n\nDo you have any results or intuition on how the agent might behave when the goal is not fixed in the reacher task?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "PpTBS2kC2wg", "reviewer": "Reviewer_SRcQ", "summary": "This paper proposes a framework for contractive active inference, leveraging contrastive learning to improve active inference in a self-supervised manner. Experiments in goal-conditioned tasks show the proposed algorithm works comparably with reward-based reinforcement learning methods and outperforms the non-contrastive variant of its own. Experiments in an image-based learning task show the proposed method can reach the desired goal despite visual differences in background.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nPros: This paper builds on the idea of active inference, which is a very interesting topic, and proposes to incorporate contrastive learning for improving the efficiency of active inference. Active inference is an important research topic/idea that can potentially address some of reinforcement learning’s shortcomings. The proposed method makes active inference more efficient through leveraging contrastive learning. This paper is well-written with clarity. The experiments demonstrate various advantages of the proposed algorithm.\n\nCons: The experiments are conducted in two simple domains, gridworlds and a 2D control task. Many goal-conditioned RL algorithms have been successfully applied to high-dimensional control tasks. It is important to also compare the proposed algorithm with RL baselines in these more complex settings and evaluate how it scales.  \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed limitations and potential societal impact.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "KHPsJCqYqGg", "reviewer": "Reviewer_kRCS", "summary": "This work proposes an active inference implementation that scales in high dimensional observations and continuous actions spaces. To achieve this the authors propose to replace the reconstruction required to optimize the free energy objective with the contrastive loss which aims to maximize the mutual information between the state and the observation at each timestep and is more robust under distractors and low capacity models. They show comparable performance to state of the art model-based RL baselines (dreamer, CVRL) and they demonstrate robustness under distractors setting. Also, the provided code is great and that would make a very nice contribution as well (the existing implementations of AIF are too difficult to parse I think)", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "However, my main concern is regarding the experimental setup. \n\nQuestions to the authors:\n\nDid you compare with the original implementation of Dreamer and CVRL? If so, which versions did you use? Dreamer v1 or v2? Comparing between dreamer / cvrl / aif / contrastive aif would also introduce confounding effects by the difference in architectures / code if any. \n\nfigure 6: what is the x axis? does 100 represent 100*10^5 steps?\n\nAlso in the CVRL paper and reacher easy / natural environment CVRL achieves around 990 and dreamer around 133. Is the distracting environment \n\nIn active inference, the intrinsic motivation module will provide gradients when the prior and posterior have different values. However, in mujoco tasks, the POMDPness is coming from the fact that you'd need several frames to estimate the true state which is already provided to you via the RNN. As a result, observations won't reduce the uncertainty of the posterior belief significantly and the KL would be mostly constant (I might be totally wrong but my intuition is that mujoco tasks are fairly observable - only velocity/acceleration information is missing which can be inferred with 2-3 frames).\n\nSo, the main question is the following: For the selected tasks, the optimal policy doesn't seem to require an information-seeking behaviour and as such the reward would end up being dominated by the extrinsic value, in which case it should provide any additional benefit over dreamer.\n\nI like this work a lot but i think the experimental section doesn't sell it. You'd need experiments that show how AIF is better than dreamer (exploration) and how CAIF is better than AIF (distractors/capacity). But the way is currently designed, the comparison is only between CAIF and {Dreamer / AIF} (that's because the intrinsic value would be mostly zero on those environments).\n\nMisc comments:\n\nConsider referring to the agent as neutral \"it\" instead of \"he\" (e.g. line 149)\n\nline 30: \"... fulfil optimistic predictions ...\" - how's optimistic meant here? consider omitting optimistic for clarity\n\nline 35: ... have replaced the agents' preferred outcomes with RL like rewards ... - How is it different to the extrinsic value of your proposal? In fact, in [13] the authors follow the AIF formulation which is similar to this work. Additionally, they assume the active learning objective which is closer to the original formulation of AIF (they have the KL between prior (eq 7b) and posterior plus the active learning objective (eq 7b))\n\nContrastive examples: How did you select negative examples? Please state this explicitly in the main text.\n\nBesides those concerns, I really like the code quality and I see great value at releasing it.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "the authors discuss the broader impact", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method that bypasses the need for reconstruction in MBRL using a contrastive objective. The proposed method outperforms Dreamer on the domains studied in the paper. All reviewers unanimously vote to accept the paper, which I agree with. I would encourage the authors to scale up with their method and present results on at least a few ATARI environments in the camera-ready version. 


Decision: Accept (Poster)