Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper provides a theoretical connection between active inference and reinforcement learning. The authors show that the concept of expected free energy (EFE) can be extended to a stochastic setting and propose an action-conditioned EFE that can be interpreted as the well-known RL Q-function. They also propose a prior preference learning approach to learn from expert demonstrations.

The paper sheds light on a novel interpretation of active inference from the point of view of RL and demonstrates a theoretical connection between the two. 

However, the concepts of active inference should be more clearly introduced and some intuition should be provided. It is quite hard for a reader not truly familiar with the field to follow.
Also, the experiment section is lacking comparison with traditional RL algorithms.

A few comments:

In the derivations in page 4, some approximations are used. It would help to explain why these can be made.

In the experiment section, it is not clear which algorithms are compared. We can assume that PPL refers to Algorithm 1, although the latter is never referred to in the text. Also, it is not clear what "conventional global preference" refers to. Also,  it would help put things in perspective to compare the authors' approach to classic RL/IRL algorithms.

Minor typo: in page 2, section 2, paragraph 1, line 5: space missing before “The agent”


review_title: This paper provides a interesting theoretical connection between active inference and reinforcement learning. Impact could be made stronger by extending the experiments.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper provides a theoretical connection between active inference and reinforcement learning and develops a method that can find a prior preference from experts. The new theory is derived from the concept of expected free energy (EFE) based on the free-energy principle.  Simulation experiments were conducted, and the effect of the prior preference learning was demonstrated.

The theoretical contribution of the paper is to find the relationship between EFE and negative value function and proposed a prior preference learning method. The theoretical connection is insightful and interesting. 

However, the originality of the proposed method itself is not clear from the theoretical and practical viewpoints.
In the experiment, they compared their method with a baseline method, i.e., global preference. 
There is no comparison between the pre-existing baseline method.
Though the EFE-based approach is very interesting, the authors did not succeed in providing evidence of the advantage of the proposed method.
It is questionable if this experiment is suitable for evaluating the main argument of this paper.

Also, from the viewpoint of the information-theoretic approach to RL and the relation to the free energy principle, studies related to "control as inference" is worth mentioning.

- Levine, Sergey. "Reinforcement learning and control as probabilistic inference: Tutorial and review." arXiv preprint arXiv:1805.00909 (2018).
- Okada, Masashi, and Tadahiro Taniguchi. "Variational inference mpc for bayesian model-based reinforcement learning." Conference on Robot Learning. 2020.
- Hafner, Danijar, et al. "Action and perception as divergence minimization." arXiv preprint arXiv:2009.01791 (2020).

<Minor comments>

Capitalized Q is used for representing a variational density function. Q is often used in action-value function in the context of RL. If this is not equivalent to Q-function, it cannot be very clear. I think using q is a better choice.

In 5.1.1, "We did not run setting 2 in this study, because Acrobat is ambiguous in defining the global
preference of the environment."
-> This may be "setting 4."


The definition of "global preference" is not given. To my understanding, the term is not so well-known in the community of imitation and reinforcement learning. That should be defined. Because of this, what the experiment showed is unclear to potential readers.


In conclusion, they describe, "We also show that active inference can provide insights to solve the inverse RL problems." However, they did not provide any explicit discussion over "inverse RL." This is actually the second time they mention "inverse RL." The first one is just at the end of the introduction.
This should be explicitly mentioned if the authors put this statement in conclusion.
review_title: The EFE-based approach is interesting, but validation is not sufficient.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The work in this paper draws connections between the active inference literature and reinforcement learning frameworks. The paper proposes a connection between these two methods more formally so that you can convert the active inference learning problem into a reinforcement learning problem. The paper also shows some success in being able to solve some simple control problems by providing some expert demonstrations it seems. It's not clear if this work is significantly different than the deep active inference paper which does something very similar and also runs experiments on the mountain car problem.

https://arxiv.org/abs/1709.02341

- The equations are not numbered in the paper, but the first equation in section 3 is a little unclear given the paragraph before it on how it would be obvious that this follows. It will help the reader to add more description about this.
- Q appears to be overloaded many times in the mathematics of the paper and makes it a bit difficult to follow the theory and section 2.
- The author's note and the experiment section that some type of expert preferences is greater than some type of global preference. This terminology is confusing and it's not clear where the "global preference" comes from.
- Similarly, "Expert Batch" is used to help learning but there does not appear to be a definition for the expert batch. Is it related to the 5000 tuples collected early? Where does this expert data come from?

---- Post Discussion ----
The discussion with the authors improved my understanding of how the paper fits with recent work.
review_title: Interesting start but the message got lost.

Meta-review: The meta-reviewer agrees with the reviewers that this is a marginal case. Conditioned on the quality of content and comparisons to other works:
Constrained Reinforcement Learning With Learned Constraints (https://openreview.net/forum?id=akgiLNAkC7P)
Parrot: Data-Driven Behavioral Priors for Reinforcement Learning (https://openreview.net/forum?id=Ysuv-WOFeKR)
PERIL: Probabilistic Embeddings for hybrid Meta-Reinforcement and Imitation Learning (https://openreview.net/forum?id=BIIwfP55pp)

We believe that the paper is not ready for publication yet. We would strongly encourage the authors to use the reviewers' feedback to improve the paper and resubmit to one of the upcoming conferences.

Decision: Reject