Prior Preference Learning From Experts: Designing A Reward with Active Inference
Authors: Jin Young Shin
Cheolhyeong Kim
Hyung Ju Hwang
Venue:  (2021)
Keywords: Active Inference
Free Energy Principle
Reinforcement Learning
Reward Design

Abstract: Active inference may be defined as Bayesian modeling of a brain with a biologically plausible model of the agent. Its primary idea relies on the free energy principle and the prior preference of the agent. An agent will choose an action that leads to its prior preference for a future observation. In this paper, we claim that active inference can be interpreted using reinforcement learning (RL) algorithms and find a theoretical connection between them. We extend the concept of expected free energy (EFE), which is a core quantity in active inference, and claim that EFE can be treated as a negative value function. Motivated by the concept of prior preference and a theoretical connection, we propose a simple but novel method for learning a prior preference from experts. This illustrates that the problem with RL can be approached with a new perspective of active inference. Experimental results of prior preference learning show the possibility of active inference with EFE-based rewards and its application to an inverse RL problem.

Under review as a conference paper at ICLR 2021
PRIOR PREFERENCE LEARNING FROM EXPERTS: DESIGNING A REWARD WITH ACTIVE INFERENCE
Anonymous authors Paper under double-blind review
ABSTRACT
Active inference may be defined Bayesian modeling of brain with biolog- ically plausible model of the agent. Its primary idea relies on the free energy principle and the prior preference of the agent. An agent will choose an action that leads to its prior preference for a future observation. In this paper, we claim that active inference can be interpreted using reinforcement learning (RL) algo- rithms and find a theoretical connection between them. We extend the concept of expected free energy (EFE), which core quantity in active inference, and claim that EFE can be treated as a negative value function. Motivated by the con- cept of prior preference and a theoretical connection, we propose a simple but novel method for learning prior preference from experts. This illustrates that the problem with inverse RL can be approached with new perspective of active inference. Experimental results of prior preference learning show the possibility active inference with EFE- based rewards and its application to an inverse RL problem.
INTRODUCTION
Active inference Friston et al.
2009) is a theory emerging from cognitive science using a Bayesian
modeling of the brain function (Friston et al. 2006 Friston 2010 Friston et al. 2015 2013). predic- tive coding (Friston et al.) [201]
Lopez-Persent et al. 2016). and the free energy principle (Friston) 2012 Parr& & Friston| 2019 Friston 2019). It states that the agents choose actions to minimize an expected future surprise (Friston et al. 2012 201 70 b), which is measurement of the difference be- tween an agent's prior preference and expected future. Minimization an expected future surprise can be achieved by minimizing the expected free energy (EFE), which is core quantity of active inference. Although active inference and EFE have been inspired and derived from cognitive science using a biologically plausible brain function model, its usage in RL tasks is still limited owing its computational issues and prior-prefere preferrreeeeece design. (Millidge 2020 Fountas et al. 2020)
First, EFE requires heavy computational cost. A precise computation of an EFE theoretically erages all possible policies, which is clearly intractable as an action space A and a time horizon T increase in size. Several attempts have been made to calculate the EFE in tractable manner, such as limiting the future time horizon from t to t H (Tschantz et al. 2019). and applying Monte-Carlo based sampling methods (Fountas et 2020 Çatal al. 2020) for the search policies.
Second, it is unclear how the prior preferences should be set. This is the same question as how to design the rewards in the RL algorithm. In recent studies (Fountas al. 2020 Çatal et al. 2020 Ueltzhöffer 2018) the agent's 's prior preference is simply set as the final goal of a given environment for every time step. There are some environments in which the prior preference can be set as time independent. However, most prior preferences in RL problems are neither simple nor easy to design because prior preferences of short and long-sighted futures should generally be treated in different ways.
In this paper, we first claim that there a theoretical connection between active inference and RL al- gorithms. We then propose prior preference learning (PPL), simple and novel method for learning a prior-preference of an active inference from an expert simulation. In Section 2, we briefly intro- duce the concept of an active inference. From the previous definition of the EFE of a deterministic policy, in Section 3, we extend the previous concepts of active inference and theoretically demon- strate that it can be analyzed in view of the RL algorithm. We extend this quantity to stochastic

---

Under review as a conference paper at ICLR 2021
EFE for a given action and a given policy network. policy network and define an bootstrapping argument, we show that the optimal distribution Following Millidge (2020) using a
over the first-step action induced from active inference can be interpreted using Q-Learning. Con- sequently, we show that EFE can be treated as a negative value function from an RI. perspective. From this connection, in Section we propose novel inverse RI algorithm for designing EFE- based rewards, by learning a prior preference from expert demonstrations. Through such expert demonstrations, an agent learns its prior preference given the observation achieve final goal, which can effectively handle the difference between local and global preferences. It will extend the scope of active inference to inverse RL problem. Our experiments in Section 6 show the applicability of active inference based rewards using EFE to an inverse RL problem.
2 ACTIVE INFERENCE
The active inference environment rests on the partially observed Markov decision process settings with an observation that comes from sensory input or and hidden state which is encoded in the agent's latent space, We will discuss a continuous observation/hidden state space, discrete time step, and a discrete action space A: At current time < T with given time horizon T, the agent receives an observation o₁. The agent encodes this observation to a hidden state & in its internal generative model, (i.e., generative model for the given environment in an agent) and then searches for the action sequence that minimizes the expected future surprise based on the agent's prior preference p(o,) of a future observation o, with > f. (i.e. The agent avoids action which leads to unexpected and undesired future observations, which makes the agent surprised.)
In detail, we can formally illustrate the active inference agent's process as follows: 8t and or a hidden state and an observation at time 4, respectively. In addition, ा = (01, (01,42,--.. 42,.-.. @r) is a se- quence of actions. Let p(01:T, $1:T) be a generative model of the agent with its transition model p(st+1|St, az), and o(01.T,$1:77 T) be variational density. A distribution over policies q(T) will be determined later. From here, we can simplify the parameterized densities as trainable neural net- works with p(o,|s: (s:) as a decoder, (s)))o)) as an encoder, and in our generative model.
(St, 02) as transition network
First, we minimize the current surprise of the agent, which is defined as - llggp(o)) Its upper bound can be interpreted as the well-known negative ELBO term, which frequently referred as the variational free energy F; at time / in studies on active inference.
-logp(o)) S E,(+: (ss))o;)
s:)]
Minimizing F3 provides an upper bound on the current surprise, and makes our networks in the generative model well-fitted with our known observations of the environment and its encoded states. For the future action selection, the total EFE ((x)) over all possible policies at the current state s, at time should be minimized.
g(s,) Regto [log ((5--1:T, πT)
(2)
p(s 1.+1
Focusing on the distribution q(T). it is known that the total EFE ((s)) ) is minimized when the distribution over policies (=x) follows o (-Gz(se)), where (.)) softmax over the policies and G=(s)) is the EFE under given state s, for fixed sequence of actions x at time t. (Millidge et al. 2020
4(s TT)
(st) Σ Gx( T>t
E T>t
(3)
((0))((5
This means that a lower EFE is obtained for a particular action sequence क.; a lower future surprise will be expected and a desired behavior p(o) will be obtained. Several active inference studies introduce temperature parameter 7 > 0 such that (z) = a->GG>>s)) to control the agent's behavior between exploration and exploitation. Because know that the optimal distribution over the policies is q (--G, (s:)). the action selection problem boils down to a calculation of the expected free energy G (St) of given action sequence
The learning process of active inference contains two parts: (1) learning an agent's 's generative model with its trainable neural networks p(o.)s:). q(s, (o)), and p( az) that explains the current observations and (2) learning to select an action that minimizes a future expected surprise of the agent by calculating the EFE of given action sequence.
2

---

Under review as a conference paper at ICLR 2021
3 EFE AS A NEGATIVE VALUE: BETWEEN RL AND ACTIVE INFERENCE
In this section, we first extend the definition of an EFE to a stochastic policy. We then, pro- pose an action-conditioned EFE that has similar role negative action-value function in RL. Based on these extensions, we will repeat the arguments in the active inference and claim that the RL algorithm with EFE as negative value is equivalent to controlling the policy network to- ward an ideal distribution in the active inference. Calculating the expected free energy (s)) for all possible deterministic policies is intractable even in a toy-example task because the num- ber of policies rapidly increases as the time horizon T and the number of actions |LA] increases. Instead of searching for a number of deterministic policies, we extend the concept of EFE stochastic policy based on a policy network Ø = (s)) In this case, we also extend g(s,, (T) to ( := p( a--)),,0--1)aa---1- 1 which distribution over the states and ac- tions, where each action d--1 is only dependent state 5--1 with 6(a--1ls--1). Plugging in deterministic policy o = x yields q(s,, a,-1 = (s--11) a') with ((,--1) = a', which the active inference. Suppose we choose an action d, based on the current state 8; and a given action network Φ, its corresponding expected free energy term can then be written as follows. This can be interpreted by substituting our extensions for the probabilities in
same equation used in previous studies
EFE of sequence of policies (Ø, "......).) Φ)
g(St, al 110) log p(o,)q(s, (4(())(s000 l0,)
Go(s;)= E Irse al8, 160)
(4)
Note that the agent uses the same action network ((a-1s,) for all T, and we can therefore rewrite this equation in a recursive form.
log (60, 110))
Go($t) = E.,,, 0(
(lo))((87007
TSt
(1844,
log g( 11-)
= =E (1 flg
p p(0441)((5++1, ar 0(111)
plo,)g(8,10, )0,)
tst+1
(ef 000)) log (81+1,0010)
log g(s,, 1
E Trser 111
p(0++1)((c+110..
plo,)g 04)
T>1+1
- llg (((+1,
(10+119((44100111 Go(s++1)
(5) Replacing and fixing the first action a; an action-conditionedd EFE Go(s)la) of a given action a and a policy network @ then defined as follows:
log a1 = + Go(sstz)
Go(sela)= ==
p(0(+1,
(6)
(s441) log p( a)
p(o++))((s44100411 (0++1) + Go(
Taking an expectation over ((als)), we obtain the relationship between Gs())) and Go(sslac).
Ea(al [G)(s:aa)] (s + H(b(a]s:)
We may consider separating the distribution over the first-step action from & to find an alternative distribution that minimizes the EFE. Substituting the distribution over the first-step action ((a)) instead of p(a)), we obtain its one-step substituted EFE as indicated below. This can be interpreted as the EFE of sequence of policies (g(az); o,..)
(8) = (eg(a))((s4) Tr l0) log ((a)) S01 ( 0 110)
1( 1141:T)
-agla)oleeni on lll o(s.. (() + log g(sc+1las) + log
a7)))
II
p(0(+1,84+1)
plor)
T>441
-Bgars) logg(as) Go(dela))
(8)
3

---

Under review as a conference paper at ICLR 2021
Under given 6, the value above depends only the distribution q(a)). Thus, minimizing the quantity above will naturally introduce the distribution q (a(a) = σal (s (a))), which is known to be similar in terms of active inference the ? case.
(i-ste((5t) = By(ar)
logg(ar) + Go(stlaa))
KI(g(aa)lla" (a:)) log Σ exp( (s)(a)
d.EA
Therefore, 9 9i-step(8)) (st) 2 -log En.e. E.A exp(-Ga(sala))) and the equality holds if and only if ((a:) oa( (G Let us further consider the temperature hyperparameter > 0 such that Similarly the above, we obtain the following:
(,,),) = -aa oall Gs(- at)).
gp ((-step(0)) B(ce)) y (logg(a)) GGG(sa))))
7 Recar) (1 -)--(((-))) log ((a))) -))(-log((a)))
+ logg(a)) log o7 (ax))
1
log Σ exx((7Go((tlaa))
- (-1))(((aa))) 1 [K(alae)l((((a))) ? arEd
1 log E exp(=>Go(sslat))
7
(10)
From the arguments above, we can conclude that (1) 9 st-step(s)) is minimized when g" = 0al va(-G(ssla.)). a 'natural case' in which n = 1, which was heuristically set in the experiments described in Millidge 2020). (2) Plugging (a,)) aal oa(-oG((sloo)) to q(a) in the equation above, we obtain
9 st-sppp(1) (3t) = (1 - 7) 7) (a) G (s)()))))))) - log D,y
(11)
= (1 - 7) (1-7)BBy
(s((ea)) + min (Go(sela))
a.EA
=E Ea. exp(->G (a,)) and the last comes from the smooth approximation to the where D. =
maximum function of log-sum-exp. This approximation becomes accurate when the maximum is much larger than the others.
log Dy log Σ exp(->Goossla))
max ( = min Ga(sela:) (12) atEd
a.EA
When = 1, the quantity 9f (5t) ) is minimized with of (ar) ) (=(s_l())) and its minimum value can be approximated as - log D1 se minored a.) When y + co, 9 (ax) can be considered deterministic policy seeking the smallest EFE, which leads to the following:
- (8t) _ 1)H(()((a.))
1 log Σ 7 0.EA
(xp(->Go(s((a:)) min Go(stlat) (13)
a.EA
Note that Q-learning with negative EFE can be interpreted as 9f with the case of xo. When Y O, (az) converges to a uniform distribution over the action space A, and the temperature hyperparameter yy motivates the agent to explore other actions with a greater EFE. Its weighted sum also converges to an average of the action-conditions EFE.
Considering the optimal policy 6" that seeks an action with minimum EFE, we obtain the follow- ing:
Go-(s1) min G (s)la)
(m.42) log p( + Go- (s4t1)
(14)
min E
((++))(((=+1004))
This equation extremely similar to the Bellman optimality equation. Here, Gs(syla) and (s141) correspond to the action-value function and the state value function, respectively. Based on this sim- p(s
ilarity, we can consider the first term log plo--1 ( (o++1) as one-step negative reward (because active inference aims to minimize the expected free energy of the future) and EFE as a negative value function.
4

---

Under review as a conference paper at ICLR 2021
4 PPL: PRIOR PREFERENCE LEARNING FROM EXPERTS
p(su)))).d)) From the previous section, we verified that using log plor+s (0(s++ [or-1] as a negative reward can handle EFE with traditional RL methods. Through a simple calculation, the given term can be decomposed as follows:
Rg llg ((8 01) log p(0++1) (olog P($++1|5t, ai) Ri, Ri,e (15)
7(01-1)9((441
((t--10041)
The first term measures the similarity between preferred future and predicted future, which is intuitive value. The second term is called the epistemic value, which encourages the exploration of an agent. (Friston et al., 2015) The epistemic value can be computed either with prior knowledge on the environment of an agent or with various algorithms to learn generative model. (Ig Iglee et al. 2018
Kaiser et al. 2019)
The core key to calculating the one-step reward and learning EFE is the prior preference p(oz). which includes information about the agent's preferred observation and goal. Setting prior preference for an agent is a challenging point for active inference. A simple method that has been used in recent studies (Ueltzhöffer 2018 Çatal et al., 2020) is set a prior preference as Gaussian distribution with mean of the goal position. However, it would be inefficient to use the same prior preference for all observations o. We called this type preference global preference. Taking a mountain- car environment as example, to reach the goal position, the car must move away from the goal position in an early time step. Therefore p(or) must contain information about local preference. is also difficult directly design the prior preference for some observation space, such in image.
To solve the problems arising from active inference, we introduce prior preference learning from experts (PPL). Suppose that agent access expert simulations S {or,1, ..,,, } i= where N is the number simulations. From the expert simulations S, an agent can learn an expert's prior p(o=+1 Jox) based on the current observation Oz, Model p(o-- (0:)) captures the expert's 's local preference, which is more effective than the global preference. We can state that this method can be applied only with expert simulations, regardless of the knowledge on the forward dynamics of the environment.
Once learning the prior preference p(or-1l0;) lo:). we can calculate R₃ given in (15) We can use any RL alogorithm to approximate EFE, G (5t) in (14) In Algorithm we provide our pseudo-code of PPL with Q-learning.
RELATED WORKS
Active inference on RL. Our works are based on active inference and reinforcement learning. Active inference was first introduced in Friston et al. (2006), inspired from neuroscience and free energy principle. It explains how a biological system is maintained with brain model. Furthermore, Friston et al. (2009) treated the relation between active inference and reinforcement learning. Early studies (Friston] 2010 Friston al. 2011 2015 2017a) dealt with tabular or model based problem due to computational cost.
Recently, (ullzhööfer (2018) introduced deep active inference which utilizes deep neural network to approximate the observation and transition models on MountainCar environment. This work used EFE objective function and its gradient propagates through the environment dynamics. To handle this problem, Ueltzhöffer (2018) used stochastic weights to apply an evolutionary strategy (Salimans al. 2017). which method for gradient approximation. For stable gradient approx- imation, about 10y environments were used in parallel.
Çatal et al. (2020); Millidge 2020)) introduced end-to-end differentiable models by including envi- ronment transition models. Both require much less interaction with environments than before. Çatal ctt al. 22200) used the Monte Carlo sampling to approximate EFE and used global prior preference. On the other hand Millidge (2020) used bootstrapping methods and used common RL rewards with model driven values derived from EFE. Millidee (2020) verified that the model driven values induce faster and better results MountainCar environment. Furthermore, TTschnnt al. (2020); Millldgel let al. (2020) introduced a new KL objective called free energy of expected future (FEEF) which is
related to probabilistic RI. (Levine) 2018b Rawlik 2013 Kappen al. 2012 [ee et al. 2019).
5

---

Under review as a conference paper at ICLR 2021
Algorithm 1: Inverse Q-Learning with Prior Preference Learning
Learning prior preference from expert simulations; Input: Expert simultaions S 1, ..,.)......)))) ..,T)))))=11 Initialize prior preference network po()); while not converge do Compute loss ppp (f(())) Update - a Leppi
end
Output: (o)
Learning EFE and forward dynamic of an environment;
Input: Prior preference (o)
Initialize the forward dynamic (ol(ss), (8 az), (slo). and EFE network Gg(se, 0z).
while not converge do
Reset environment;
for t +- 0 do
Select action a; arg max, G(st, a) with e-greedy;
Observe new observation ot+1i
Compute Rg = log plo--1 (9(8e41 (ot+1]
Compute the environment model loss Linode ((p o q))o),
Compute the EFE network loss Lefe (5,, az), Rr + max, Ge(st+1,a))
Update 7 a La and
L.fe.
end
end
Our work extends the scope of active inference to inverse RL by learning a preferred observation from experts. common limitation previous studies on active inference is the ambiguity of prior preference distribution. Previous works have done in environments where its prior preference can naturally expressed, which is clearly not true general. PPL. an active inference based approach for the invser RL problem setting, which allows us to learn prior preference from expert simulations. This broadens the scope of active inference and provides a new perspective about connection between active inference and RL.
Control as inference. Levine (0188) proposed control as inference framework, which interprets control problem probabilistic inference with additional binary variable O, that indicates whether given action is optimal or not. Several studies on the formulation of RL as an inference prob- lem (Todorov] 2008 Kappen et al. 2009) have been proposed. Control as inference measures prob- ability that given action is optimal based on given reward with p(O, = 1|s,, ar) = exp(r(s,, a,))- That is, from the given reward and the chosen probability model, control as inference calculates the probability of the given action a, with the state & to be optimal. In contrast, active inference mea- sures this probability in (13) with prior preference distribution P and constructs reward, view- ing EFE as a negative value function. Although control as inference and active inference RL in Section have theoretical similarity based on a duality proposed PPL can interpret the reward r(st, at) and the message function 3(st) in Levine (201 Sa) as EFE-related quantitics. Therefore, learning the expert as prior preference p immeciately con- structs its active inference based reward function and thereby applicable to reward construction problem and several inverse RL problems.
control problem and an inference, our
EXPERIMENTS
In this section, we discuss and compare the experimental results of our proposed algorithm PPL and other inverse RL algorithms on the several classical control environments. We evaluate our approach with classic control environment implemented in Open AI Gym (Brockman al. 2016). First, we aim to compare the conventional global preference method to the PPL. We expect the PPL to be effective in environments where the local and global preferences different. Second, claim

---

Mountain Car is an environment with a car on the valley. The goal is to make the car reach the goal position, the right peak of the mountain. The state space is of two dimensions (z, .) where is the position of the car. The action space consists three actions: accelerating the car to the left or right, Under review as a conference paper at ICLR 2021
Table Experment setting for PPL and global preference and its variants
Preference
Batch sampling
Reward
Setting
PPI.
Replay memory Experts
Re, + Ri,
Setting 2
PPL.
Replay memory
Re + Ri,
Setting
PPL.
Replay memory Experts
Re
Setting 4
Global preference
Replay memory Experts
Re + Re,
that our active inference based approach can achieve a compatible results with current inverse RL algorithms. Expert simulations were obtained from Open Al RL baseline zoo (Rallin 2018).
PPL AND GLOBAL PREFERENCE
First, we compare our proposed algorithm (setting in T and its variants. Table[[containg four experimental settings, where the setting 1 is our proposed method and the others are experimental groups to compare the effects of an expert batch, the epistemic value Re,ss and PPL.
Expert Batch. We use expert simulations for batch sampling when learning EFE and forward dynamic model. This allows an EFE network and a dynamic model be trained even in states that do not reach the early stage of learning. We use the Q-learning algorithm to learn the EFE network. Commonly used techniques in RL, such as replay memory and target network MMih et al. 2015)
are used.
Reward. We observe how an epistemic value Ri, in 15 influences the learning process and its performance.
Global preference. Global preference is distribution over a state which can naturally induced from the agent's goal, whereas our PPL is learned prior preference from the expert's 's simulations. Roughly, global preference can be understood as a hard-coded prior preference based on the prior knowledge of the environment, coded global preferences in our experiments can be found in the below sub-subsection.
'goal directed behavior' in Ueltzhöffer (2018). Detailed hard-
6.1.1 ENVIRONMENTS AND EXPERIMENT DETAILS
Acrobot, Cartpole, and MountainCar. We used 5000 pairs of (ot, 0++1) to train the prior preference p(0++1). We used the same neural network architecture for all environments, except for the number of input and output dimensions. During the training process, we clip the epistemic value to prevent a gradient explosion while using the epistemic value in settings
We tested three classical control environments:
and 4. Acrobot an environment with a two-link pendulum, where the second link is actuated. The goal of this environment is swing the end effector up the horizontal line. The state space has six dimensions: (cos 01, sin 01, cos Sin sin 022 02). where 01 and 0g the two rotational joint angles. The action space consists of three actions: accelerating +1, O, or -1 torque on the joint between the two links. We did not run setting in this study, because Acrobot is ambiguous defining the global preference of the environment.
Cartpole an environment with cart and a pole on the cart. The goal is balance the pole by controlling the cart. The state space of 4 dimensions: (,, 0), where the position of the cart and ( is the angle of the pole from the vertical line. The action space consists of two actions: accelerating the cart to the left or right. The global preference is given by a Gaussian distribution with mean (0, 0, 0, 0) with fixed standard deviation 0.1 on each dimension.
or leaving it. The global preference is given by Gaussian distribution with mean (0.5, 0) with fixed standard deviation 0.1 on each dimension. Note that (0.5, 0) is the state of the goal position with zero velocity.

---

Under review as a conference paper at ICLR 2021
MountainCar-
CartPolev
500
-120
.00
Mostl roNand
-200
reward
-140
300
330
-10
290
400
100
-200
100 Number eovodes
350
800
N.mber
Number epsindes
Setting 1
Setting
Setting 3
Secting 4
Expert
Figure 1: Experiment results on three classical control environments: MountainCar-vo, Acrobot-vl, and CartPole-vl. The curves the figure were averaged over 50 runs, and the standard deviation of 50 runs is given shaded area. Each policy was averaged out of 5 trials. All rewards of the environment follow the default settings of Open AI Gym.
6.1.2 RESULTS AND DISCUSSIONS
Figure D shows the experimental results on MountainCar, Acrobot, and Cartpole. Their perfor- mances are compared and benchmarked with the default reward of the environments.
PPL and global preference. Comparing setting (blue line) and setting 2 (red line), it can be that PPL is more efficient than the conventional global preference expected. In particular, in MountainCar, we can see that little learning is achieved. This seems to be because the difference between global and expert preferences is greater in the MountainCar environment. the Cartpole, setting 2 learned more slowly than setting 1.
Expert Batch. Comparing setting 1 (blue line) and setting 3 (orange line), it can be seen that using the expert batch is helpful for certain tasks. With Acrobot and MountainCar, the use of an expert batch performs better than the case without an expert batch. However, the results without expert batch are marginally better than those of setting for Cartpole. This is because an agent of Cartpole only moves near the initial position, and thus there no need for an expert batch to discover the dynamics of the generative model.
Epistemic Value. We found that the epistemic value in the EFE term does not significantly impact the training process. Comparing setting (blue line) and setting 4 (green line), the results were similar regardless whether the epistemic value was used. In Acrobot and MountainCar, standard deviations were marginally smaller, but there were significant differences between them. In the result of CartPole-vl, we found that setting 4 with no epistemic value term learned faster than our proposed setting at the beginning of the training process. We deduce that this initial performance drop is due to the instability of the epistemic term. At the beginning of the training process, the generative model is not learned, and thus the related epistemic term becomes unstable. We leave this issue to a future study.
6.2 PPL AND INVERSE RL ALGORITHMMS
Second, we check that our proposed PPL is compatible with traditional inverse RL algorithms. We compared PPL with behavioral cloning (BC, Pomerleau (189)) and maximum entropy inverse RL (MaxEnt, Ziebart et al. (2008) as benchmark models. We use setting in Table as our proposed PPL here, and we test on MountainCar-vi and CartPole-vI. Note that BC does need interact with the environment and the state space was discretized to use original MaxEnt algorithm. We also tried to run the experiment on Acrobot-vl for PPI. and other benchmarks, but we failed to make the agent Iearn with MaxEnt. A discretized state space for MaxEnt becomes larger exponentially to its state dimension. We think it is due to a larger dimension of its state space compared to the others. Therefore, only report that PPL and BC give similar results to Acrobot-vl.
We verified that our method PPL gives compatible results on the MountainCar-v0 and CartPole- v1. Compared MaxEnt, PPL shows better results than MaxEnt on both environments. Note that MaxEnt needs much more episodes to converge. Also, PPL obtained almost similar mean rewards to BC MountainCar-v00 whereas BC gives better results than PPL CartPole-vI.
8

---

Under review as a conference paper at ICLR 2021
CartPotevvl
6250
12500 16750
25000 31250 37300 45750
50000
2000
4000
6900
8000 10000
-100
-120
|ااا المع و الام
144
Hean reward
-000
Aort A Arii الا
1160
200
-180
100
-200
kumber apiiedes (PPL]
Number opiendes (PPL)
Expert
Figure 2: Inverse RL Experiment results on MountainCar-v0 (left) and CartPole-vl (right). The curves in the figure were averaged over 50 runs, and the standard deviation of 50 runs is given as shaded area. Note that black and green dashed line on the right are overlapped. All rewards of the environment follow the default settings of Open Al Gym. (BC Behavioral Cloning, MaxEnt Maximum Entropy)
CONCLUSION
In this paper, we introduced the use of active inference from the perspective of RL. Although active inference emerged from the Bayesian model of cognitive process, we show that the concepts of active inference, especially for EFE, are highly related to RL using the bootstrapping method. The only difference is that, the value function of RL is based on reward, while active inference is based on the prior preference. We also show that active inference can provide insights to solve the inverse RI. problems. Using expert simulations, an agent can learn a local prior preference, which is more effective than the global preference. Furthermore, our proposed active inference based reward with a prior preference and generative model makes the previous invser RL problems free from an ill-posed state. Our work active inference is complementary to RL because it can be applied to model-based RL for the design reward and model-free RL for learning of generative models.
REFERENCES
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Zafeirios Fountas, Noor Sajid, Pedro M. Mediano, and Karl J. Friston. Deep active inference agents using monte-carlo methods. CoRR, abs/2006.04176, 2020. URL htttt://////
brg/abs/200s 04176
K. Friston, J. Kilner, and I.. Harrison. A free energy principle for the brain. Journal of Physiology- Paris, 100:70-87, 2006.
Karl Friston. The free-energy principle: a unified brain theory? Nature reviews. Nearoxcience, 127-38, 02 2010. doi: 10.1038/nm2787.
Karl Friston, Spyridon Samothrakis, and Read Montague. Active inference and agency: Optimal control without cost functions. Biological cybernetics, 106:523-41, 08 2012. doi: 10.1007/ 500422-012-0512-8.
Karl Friston, Philipp Schwartenbeck, Thomas Fitzgerald, Michael Moutoussis, Tim Behrens, and Raymond Dolan. The anatomy of choice: active inference and agency. Frontiers in Human Neuroscience, 7:598, 2013. ISSN 1662-5161.
doi: 10.3389/fnhum 2013 33339//fhuum22003000500....
URL htLhst:s:
7/www. front ienatteerrin
3389/ fnhum. 2013. 00598

---

Under review as a conference paper at ICLR 2021
Karl Friston, Francesco Rigoli, Dimitri Ognibene, Christoph Mathys, Thomas Fitzgerald, and Gio- vanni Pezzulo. Active inference and epistemic value. Cognitive Neuroscience, 6(4):187-214,
2015. doi: 10.1080/1 00000001777892222111100000000000
Karl Friston, Thomas FitzGerald, Francesco Rigoli, Philipp Schwartenbeck, and Giovanni Pez- zulo. Active inference: process theory. Neural Computationn 29(1):1-49, 2017a. doi: 10.1 10.1162/NECO) - URL httph:ttt///e/oo /tte:://doo.oooo http://:://ooo.o//111111113 1ttp://1..111/2BEE. a PMID:
00912.
27870614.
Karl J. Friston. A free energy principle for biological systems. Entropy, 14(11):2100-2121, 2012. doi: 10.3390/e14112100
Kadl J. Friston. A free energy principle for particular physics, 2019. URL https://arxiv
org/abs/1906. 10184
Karl Friston, Jean Daunizeau, and Stefan Kiebel. Reinforcement learning or active inference?
PloS one, 4(7)xe6421, 2009.
Karl J. Friston, Jérémie Mattout, and James Kilner. Action understanding and active inference.
Biol. Cybern., 104(1-2):137-160, 2011. doi: 10.1077/50422--01-04244- URL https:
doi.org/ 10. 100000000000022200000 012-0
Karl J. Friston, Marco Lin, Christopher D. Frith, Giovanni Pezzulo. J. Allan Hobson, and Sasha On- dobaka. Active inference, curiosity and insight. Neural Computation, 29(10):2633-26)) 2017b. doi: 10.1166//11 10.1112/nec))
URL htttps:///o...oooo
Maximilian Igl, Luisa Zintgraf, Tuan Anh Le, Frank Wood, and Shimon Whiteson. Deep variational reinforcement learning for pomdps. arXiv preprint (eXivv/806.02444 02442, 2018.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based reinforcement learning for atari. arXiv preprint
2019.
Bert Kappen, Vicenç Gómez, and Manfred Opper. Optimal control as a graphical model inference problem. CoRR, abs/0901.0633, 2009. URL http:
abs/0901.0633
Hilbert J Kappen, Vicenç Gómez, and Manfred Opper. Optimal control as a graphical model infer- ence problem. Machine learning. 87(2):15--182, 2012.
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdi- nov. Efficient exploration via state marginal matching. arXiv preprint arXiv:1906.05221 2019,
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. 2018a. URL URL/ittpp://aaxiioor///////
CoRR,
00909
Sergey Levine. Reinforcement learning and control probabilistic inference: Tutorial and review. arXiv preprini 1 00909, 2018b.
Alizée Lopez-Persem, Philippe Domenech, and Mathias Pessiglione. How prior preferences de- termine decision-making frames and biases in the human brain. Neuroscience, 5:c20317, 2016. ISSN 2050-084X. doi: 10.7554/eL.ife.2331
20317
Beren Millidge. Deep active inference variational policy gradients. Journal of Mathemat- ical Psychology, 96:102348, 2020. ISSN 0022-2496.
doi: http://dbl.orr//0.0166///j URL http:://ww..t http://wwww.ccieeeeeeeeeee. wwtprreee...c.eeeeeeeeeeeeeeeeee rtt.peec...ccoeeceeeeeeaaaaaaaaa 2020. 102348.
$00222496203002293
Beren Millidge, Alexander Tschantz, and Christopher Buckley, Buckey, Whence expee free energy?
CoRR, abs/2004.08122 2020. URI.ttps::/arxii.oonooee 28811
10

---

Under review as a conference paper at ICLR 2021
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioamnis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- Stra, Shane I Legg. and Demis Hassabis. Human-level control through deep reinforcement learning. Nat., 518((5400)552 5333333 2015.
doi: 10. 10.1038/mature14233. htt.:ttt//://...orrrt
1038/nature14233
Thomas Parr and Karl J. Friston. Generalised free energy and active inference. Biological
Cyhernetics, 11365-6):99--513, 2019. doi: 10.10077400422-11-000005 D1 011-0000222211--000-00000 00005-w
URL
https:
7/doi. orooi.org/00.. 10. 1000000000200
Dean A Pomerleau. Alvinm: An autonomous land vehicle in neural network. In Advances in neural information processing systems, pp. 305-313, 1989.
Antonin Raffin. RI baselines zoo. httpss//ggittub..cm/aaaaafffffi 2018.
Traselines 200
Konrad Cyrus Rawlik. probabilistic inference approaches stochastic optimal control. 2013.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as scalable alternative reinforcement learning. arXiv preprint arXiv: 2017.
Emanuel Todorov. General duality between optimal control and estimation. In Proceedings of the 47th IEEE Conference on Decision and Control, CDC 2008, December 9-11, 2008, Cancúnn Mexico, pp. 4286 4292. IEEE, 2008. doi: 10. N0.1109/CDDC.220044433933 URL https://doi.
org 3xg/10.1109/CDDS 2005 2008 439438
Alexander Tschantz, Manuel Baltieri, Anil K. Seth, and Christopher L. Buckley. Scaling active inference. CoRR, abs/1911 10601, 2019. URL htl.http:///rrre hwtp:/o.rrrgggoooooe 1911.
Alexander Tschantz, Beren Millidge, Anil K. Seth, and Christopher L. Buckley, Reinforcement learning through active inference. CoRR, 02222002211666.. 2020. /
abs/2003 12636
Kai Ueltzhöffer. Deep active inference. Biol. Cybern.,
112(6)547--73,, December 2018.
ISSN 0340-1200.
doi: 10. 1011007760042--088-07555 URL https://@oi.orr/100 s 500422-018-0785-1
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1438. Chicago, IL, USA, 2008
O. Çatal, T. Verbelen, Nauta, C. D. Boom, and B. Dhoedt. Learning perception and planning with deep active inference. In ICASSP 2020 2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 3952-3956, 2020.


Reviews:
[{"review_id": "buhWIYJh3o9", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a theoretical connection between active inference and reinforcement learning. The authors show that the concept of expected free energy (EFE) can be extended to a stochastic setting and propose an action-conditioned EFE that can be interpreted as the well-known RL Q-function. They also propose a prior preference learning approach to learn from expert demonstrations.\n\nThe paper sheds light on a novel interpretation of active inference from the point of view of RL and demonstrates a theoretical connection between the two. \n\nHowever, the concepts of active inference should be more clearly introduced and some intuition should be provided. It is quite hard for a reader not truly familiar with the field to follow.\nAlso, the experiment section is lacking comparison with traditional RL algorithms.\n\nA few comments:\n\nIn the derivations in page 4, some approximations are used. It would help to explain why these can be made.\n\nIn the experiment section, it is not clear which algorithms are compared. We can assume that PPL refers to Algorithm 1, although the latter is never referred to in the text. Also, it is not clear what \"conventional global preference\" refers to. Also,  it would help put things in perspective to compare the authors' approach to classic RL/IRL algorithms.\n\nMinor typo: in page 2, section 2, paragraph 1, line 5: space missing before “The agent”\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "R5qTmNEdop2", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The work in this paper draws connections between the active inference literature and reinforcement learning frameworks. The paper proposes a connection between these two methods more formally so that you can convert the active inference learning problem into a reinforcement learning problem. The paper also shows some success in being able to solve some simple control problems by providing some expert demonstrations it seems. It's not clear if this work is significantly different than the deep active inference paper which does something very similar and also runs experiments on the mountain car problem.\n\nhttps://arxiv.org/abs/1709.02341\n\n- The equations are not numbered in the paper, but the first equation in section 3 is a little unclear given the paragraph before it on how it would be obvious that this follows. It will help the reader to add more description about this.\n- Q appears to be overloaded many times in the mathematics of the paper and makes it a bit difficult to follow the theory and section 2.\n- The author's note and the experiment section that some type of expert preferences is greater than some type of global preference. This terminology is confusing and it's not clear where the \"global preference\" comes from.\n- Similarly, \"Expert Batch\" is used to help learning but there does not appear to be a definition for the expert batch. Is it related to the 5000 tuples collected early? Where does this expert data come from?\n\n---- Post Discussion ----\nThe discussion with the authors improved my understanding of how the paper fits with recent work.", "extra_scores": {}, "extra_text": {}}, {"review_id": "OJrxu0jIyry", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a theoretical connection between active inference and reinforcement learning and develops a method that can find a prior preference from experts. The new theory is derived from the concept of expected free energy (EFE) based on the free-energy principle.  Simulation experiments were conducted, and the effect of the prior preference learning was demonstrated.\n\nThe theoretical contribution of the paper is to find the relationship between EFE and negative value function and proposed a prior preference learning method. The theoretical connection is insightful and interesting. \n\nHowever, the originality of the proposed method itself is not clear from the theoretical and practical viewpoints.\nIn the experiment, they compared their method with a baseline method, i.e., global preference. \nThere is no comparison between the pre-existing baseline method.\nThough the EFE-based approach is very interesting, the authors did not succeed in providing evidence of the advantage of the proposed method.\nIt is questionable if this experiment is suitable for evaluating the main argument of this paper.\n\nAlso, from the viewpoint of the information-theoretic approach to RL and the relation to the free energy principle, studies related to \"control as inference\" is worth mentioning.\n\n- Levine, Sergey. \"Reinforcement learning and control as probabilistic inference: Tutorial and review.\" arXiv preprint arXiv:1805.00909 (2018).\n- Okada, Masashi, and Tadahiro Taniguchi. \"Variational inference mpc for bayesian model-based reinforcement learning.\" Conference on Robot Learning. 2020.\n- Hafner, Danijar, et al. \"Action and perception as divergence minimization.\" arXiv preprint arXiv:2009.01791 (2020).\n\n<Minor comments>\n\nCapitalized Q is used for representing a variational density function. Q is often used in action-value function in the context of RL. If this is not equivalent to Q-function, it cannot be very clear. I think using q is a better choice.\n\nIn 5.1.1, \"We did not run setting 2 in this study, because Acrobat is ambiguous in defining the global\npreference of the environment.\"\n-> This may be \"setting 4.\"\n\n\nThe definition of \"global preference\" is not given. To my understanding, the term is not so well-known in the community of imitation and reinforcement learning. That should be defined. Because of this, what the experiment showed is unclear to potential readers.\n\n\nIn conclusion, they describe, \"We also show that active inference can provide insights to solve the inverse RL problems.\" However, they did not provide any explicit discussion over \"inverse RL.\" This is actually the second time they mention \"inverse RL.\" The first one is just at the end of the introduction.\nThis should be explicitly mentioned if the authors put this statement in conclusion.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The meta-reviewer agrees with the reviewers that this is a marginal case. Conditioned on the quality of content and comparisons to other works:
Constrained Reinforcement Learning With Learned Constraints (https://openreview.net/forum?id=akgiLNAkC7P)
Parrot: Data-Driven Behavioral Priors for Reinforcement Learning (https://openreview.net/forum?id=Ysuv-WOFeKR)
PERIL: Probabilistic Embeddings for hybrid Meta-Reinforcement and Imitation Learning (https://openreview.net/forum?id=BIIwfP55pp)

We believe that the paper is not ready for publication yet. We would strongly encourage the authors to use the reviewers' feedback to improve the paper and resubmit to one of the upcoming conferences.

Decision: Reject