Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a novel variation of the “Active Inference Model” that by incorporating contrastive learning instead of likelihood-based learning, makes the learning more computationally tractable and not reliant on high-dimensional observation reconstriction.
main_review: Finding alternative methods to reward learning is an important research direction that could be fruitful in having more generalizable and biologically plausible models. In addition, recently there have been very promising results in using contrastive learning in a wide array of domains. And this work takes advantage of those recent advances in contrastive learning to push forward the active inference model. The paper is well written and it is a pleasure to read. And the authors do discuss some of the drawbacks of their work.
The main shortcoming of the paper is its experiments section, which is limited to only two simple tasks. Would be interesting to see the performance in a wider array of tasks and more in-depth analysis. 


Specifying desired outcomes might pose its own challenges similar to those of reward design. Could you please elaborate more on that?

Could you please describe more in detail how $\tilde{p}(o_t)$ formulated for each task?

Do you have any results or intuition on how the agent might behave when the goal is not fixed in the reacher task?
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work proposes an active inference implementation that scales in high dimensional observations and continuous actions spaces. To achieve this the authors propose to replace the reconstruction required to optimize the free energy objective with the contrastive loss which aims to maximize the mutual information between the state and the observation at each timestep and is more robust under distractors and low capacity models. They show comparable performance to state of the art model-based RL baselines (dreamer, CVRL) and they demonstrate robustness under distractors setting. Also, the provided code is great and that would make a very nice contribution as well (the existing implementations of AIF are too difficult to parse I think)
main_review: However, my main concern is regarding the experimental setup. 

Questions to the authors:

Did you compare with the original implementation of Dreamer and CVRL? If so, which versions did you use? Dreamer v1 or v2? Comparing between dreamer / cvrl / aif / contrastive aif would also introduce confounding effects by the difference in architectures / code if any. 

figure 6: what is the x axis? does 100 represent 100*10^5 steps?

Also in the CVRL paper and reacher easy / natural environment CVRL achieves around 990 and dreamer around 133. Is the distracting environment 

In active inference, the intrinsic motivation module will provide gradients when the prior and posterior have different values. However, in mujoco tasks, the POMDPness is coming from the fact that you'd need several frames to estimate the true state which is already provided to you via the RNN. As a result, observations won't reduce the uncertainty of the posterior belief significantly and the KL would be mostly constant (I might be totally wrong but my intuition is that mujoco tasks are fairly observable - only velocity/acceleration information is missing which can be inferred with 2-3 frames).

So, the main question is the following: For the selected tasks, the optimal policy doesn't seem to require an information-seeking behaviour and as such the reward would end up being dominated by the extrinsic value, in which case it should provide any additional benefit over dreamer.

I like this work a lot but i think the experimental section doesn't sell it. You'd need experiments that show how AIF is better than dreamer (exploration) and how CAIF is better than AIF (distractors/capacity). But the way is currently designed, the comparison is only between CAIF and {Dreamer / AIF} (that's because the intrinsic value would be mostly zero on those environments).

Misc comments:

Consider referring to the agent as neutral "it" instead of "he" (e.g. line 149)

line 30: "... fulfil optimistic predictions ..." - how's optimistic meant here? consider omitting optimistic for clarity

line 35: ... have replaced the agents' preferred outcomes with RL like rewards ... - How is it different to the extrinsic value of your proposal? In fact, in [13] the authors follow the AIF formulation which is similar to this work. Additionally, they assume the active learning objective which is closer to the original formulation of AIF (they have the KL between prior (eq 7b) and posterior plus the active learning objective (eq 7b))

Contrastive examples: How did you select negative examples? Please state this explicitly in the main text.

Besides those concerns, I really like the code quality and I see great value at releasing it.

limitations_and_societal_impact: the authors discuss the broader impact
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a framework for contractive active inference, leveraging contrastive learning to improve active inference in a self-supervised manner. Experiments in goal-conditioned tasks show the proposed algorithm works comparably with reward-based reinforcement learning methods and outperforms the non-contrastive variant of its own. Experiments in an image-based learning task show the proposed method can reach the desired goal despite visual differences in background.
main_review: 
Pros: This paper builds on the idea of active inference, which is a very interesting topic, and proposes to incorporate contrastive learning for improving the efficiency of active inference. Active inference is an important research topic/idea that can potentially address some of reinforcement learning’s shortcomings. The proposed method makes active inference more efficient through leveraging contrastive learning. This paper is well-written with clarity. The experiments demonstrate various advantages of the proposed algorithm.

Cons: The experiments are conducted in two simple domains, gridworlds and a 2D control task. Many goal-conditioned RL algorithms have been successfully applied to high-dimensional control tasks. It is important to also compare the proposed algorithm with RL baselines in these more complex settings and evaluate how it scales.  


limitations_and_societal_impact: The authors have adequately addressed limitations and potential societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors work in the area of active inference, using a contrastive function in optimization to enhance scalability.  They show that their method significantly outperforms non-contrastive active inference on their tasks, and are competitive with some other RL approaches.

main_review: The modifications made in this paper to standard active inference are not complicated, but they seem novel and effective.  This could be a significant result in making active inference more practicable, and it seems to merit acceptance.

Line 92: "good approximate" -> "good approximation"
Line 109: "that aim" -> "which aim"
Line 146: "as pixel" -> "as pixels"
Line 198: don't capitalize "Section"

limitations_and_societal_impact: The authors describe limitations of their work in Section 7.

They state in the Checklist that they do not see potential negative impacts, but do not address it in the main body of the manuscript.
review_title: null

Meta-review: The paper proposes a method that bypasses the need for reconstruction in MBRL using a contrastive objective. The proposed method outperforms Dreamer on the domains studied in the paper. All reviewers unanimously vote to accept the paper, which I agree with. I would encourage the authors to scale up with their method and present results on at least a few ATARI environments in the camera-ready version. 


Decision: Accept (Poster)