Combining Imitation and Reinforcement Learning with Free Energy Principle
Authors: Ryoya Ogishima
Izumi Karino
Yasuo Kuniyoshi
Venue:  (2021)
Keywords: Imitation
Reinforcement Learning
Free Energy Principle

Abstract: Imitation Learning (IL) and Reinforcement Learning (RL) from high dimensional sensory inputs are often introduced as separate problems, but a more realistic problem setting is how to merge the techniques so that the agent can reduce exploration costs by partially imitating experts at the same time it maximizes its return. Even when the experts are suboptimal (e.g. Experts learned halfway with other RL methods or human-crafted experts), it is expected that the agent outperforms the suboptimal experts’ performance. In this paper, we propose to address the issue by using and theoretically extending Free Energy Principle, a unified brain theory that explains perception, action and model learning in a Bayesian probabilistic way. We find that both IL and RL can be achieved based on the same free energy objective function. Our results show that our approach is promising in visual control tasks especially with sparse-reward environments.

Under review as a conference paper at ICLR 2021
COMBINING IMITATION AND REINFORCEMENT LEARNING WITH FREE ENERGY PRINCIPLE
Anonymous authors Paper under double-blind review
ABSTRACT
Imitation Learning (IL) and Reinforcement Learning (RL) from high dimensional sensory inputs are often introduced as separate problems, but more realistic prob lem setting is how to merge the techniques so that the agent can reduce exploration costs by partially imitating experts at the same time it maximizes its return. Even when the experts are suboptimal (e.g. Experts learned halfway with other RL methods or human-crafted experts), it is expected that the agent outperforms the suboptimal experts' performance. this paper, we propose to address the issue by using and theoretically extending Free Energy Principle, a unified brain theory that explains perception, action and model learning in a Bayesian probabilistic way. We find that both IL and RL can be achieved based on the same free en- ergy objective function. Our results show that our approach is promising in visual control tasks especially with sparse-reward environments.
INTRODUCTION
Imitation Learning (II.) is a framework to learn a policy to mimic expert trajectories. As the expert specifies model behaviors, there is no need to do exploration or to design complex reward functions. Reinforcement Learning (RL) does not have these features, so RL agents have no clue to realize desired behaviors sparse-reward settings and even when RL succeeds in reward maximization, the policy does not necessarily achieve behaviors that the reward designer has expected. The key drawbacks of IL are that the policy never exceeds the suboptimal expert performance and that the policy is vulnerable to distributional shift. Meanwhile, RL can achieve super-human performance and has potentials to transfer the policy to new tasks. As real-world applications often needs high sample efficiency and little preparation (rough rewards and suboptimal experts), it is important to find a way to effectively combine IL and RL.
When the sensory inputs are high-dimensional images in the real world, behavior learning such as IL and RL would be difficult without representation or model learning. Free Energy Principle (FEP), unified brain theory in computational neuroscience that explains perception, action and model learning in a Bayesian probabilistic way (Friston et al. [0066] Friston| 2010), can handle behavior learning and model learning the same time. In FEP, the brain has generative model of the world and computes a mathematical amount called Free Energy using the model prediction and sensory inputs the brain. By minimizing the Free Energy, the brain achieves model learning and behavior learning. Prior work about FEP only dealt with limited situations where part of the generative model given and the task is very low dimensional. As there are a lot in common between FEP and variational inference in machine learning, recent advancements in deep learning and latent variable models could be applied to scale up FEP agents to compatible with high dimensional tasks.
Recent work in model-based reinforcement learning succeeds in latent planning from high- dimensional image inputs by incorporating latent dynamics models. Behaviors can derived either by imagined-reward maximization (Ha & Schmidhuber) 20188) Hafner et al.) 2019a) by online planning (Hafner et al. 2019b). Although solving high dimensional visual control tasks with model- based methods is becoming feasible, prior methods have never tried to combine with imitation.
In this paper, we propose Deep Free Energy Network (FENet), an agent that combines the advan- tages of IL and RL so that the policy roughly leams from suboptimal expert data without the need of exploration or detailed reward crafting in the first place, then learns from sparsely specified reward functions exceed the suboptimal expert performance.

---

Under review as a conference paper at ICLR 2021
The key contributions of this work are summarized follows:
Extension of Free Energy Principle:
We theoretically extend Free Energy Principle, introducing policy prior and policy posterior combine IL and RL. We implement the proposed method on top of Recurrent State Space Model (Hainer et al., 2019b), a latent dynamics model with both deterministic and stochastic components.
Visual control tasks in realistic problem settings:
We solve Cheetah-rum runn Walker-wall and Quadruped- -walk tasks from DeepMind Control Suite (Tassa al. 2018). We do not only use the default problem settings, we also set up problems with sparse rewards and with suboptimal experts. We demonstrate that our agent outperforms model- -aaed RL using Recurrent State Space Model in sparse-reward settings. We also show that our agent can achieve higher returns than Behavioral Cloning (II.) with suboptimal experts.
2 BACKGROUNDS ON FREE ENERGY PRINCIPLE
2.1 PROBLEM SETUPS
We formulate visual control as partially observable Markov decision process (POMDP) with dis- crete time steps L, observations o,, hidden states 8t, continuous action vectors a, and scalar rewards rt. The goal is to develop an agent that maximizes expected return EE re].
2.2 FREE ENERGY PRINCIPLE
Perception, action and model learning are all achieved by minimizing the same objective function, Free Energy (Friston et al., 2006 Friston| 2010). FEP, the agent is equipped with generative model of the world, using prior p(s)) and likelihood p(o,]st).
p(o,, st) s:)sss:)p(s:)
Perceptual Inference Under the generative model, the posterior probability of hidden states given observations calculated with Bayes' theorem as follows.
p(s:lo))= p(or)se)p($t) p(o:)
p(o:)= - p(oc)ss)p((s)))
(2)
Since cannot compute p(or) due the integral, we think of approximating p(selo.) with vari- ational posterior q(st) by minimizing KL divergence KL(((s))ll((sslo)))
K K[(((s))ll((=slo))) = Inp((.) + KL( LL(((s))llp(o, st))
(3)
We define the Free Energy as (eq (eq4). 4 Since p(o:) does not depend on 8;, we can minimize (cq w.r.t. the parameters of the variational posterior by minimizing the Free Energy. Thus, the agent can infer the hidden states the observations by minimizing F,. This process is called 'perceptual inference' in FEP.
Perceptual Learning Free Energy is the same amount as negative Evidence Lower Bound (ELBO) in variational inference often in machine learning as follows.
p(o.) > -F.
(5)
By minimizing F; w.r.t. the parameters of the prior and the likelihood, the generative model learns to best explain the observations. This process is called "perceptual learning" in FEP.
Active Inference We can assume that the prior conditioned on the hidden states and actions at the previous time step as follows.
p(s:) (15(5(-11,0-11
2

---

Under review as a conference paper at ICLR 2021
The agent can change the future by choosing actions. Suppose the agent chooses as when it is at St, the prior can predict the next hidden state 84-1- Thus, we can think of the Expected Free Energy G at the next time step + 1 as follows (Friston et al. 2015).
G++1 = KT(g(s++))lp(04),.
= E. (n (Ing((+1)))))()))))))))))))))))) ( ((4041188411
Bu(ee:t)gpossssss==0 (() - Inp(0t11,
(I4g()+41) - 1np(st+1004+1) - Inp(0:+1)
Eelores [In ((((+)) - (ng(s++100+1)) - Inp(0+1)]
(8)
By(o (o -n Inp(
Since the agent has not experienced time step t + 1 yet and has not received observations 0++1. take expectation over 0 using the likelihood ((0r41|5(4)) as (eq[7) In (eq we approximate g($++1)0++1) According to the complete class theorem (Friston et al. 2012), any scalar rewards can be encoded as observation priors using p(o) oc (x) and the second term in (eq becomes a goal-directed value. This observation prior p(o(+1) can also be regarded as the probability of optimality variable p(O = 10+++)) where the binary optimality variable O++1 denotes that time step + is optimal and 0++1 0 denotes that it is not optimal as introduced in the context of control as probabilistic infereneeLevine 2018)) The first term in (eq 9 is called epistemic value that works as intrinsic motivation to further explore the world. Minimization of (g((((()+)) ((4+2100++))) (09+2)]9($(++1)) means that the agent tries to experience as different states 8t+1 as possible given some imagined observations Ot+1- By minimizing the Expected Free Energy, the agent can infer the actions that explores the world and maximize rewards. This process called 'active inference'.
0(0++1|$142) g(0++1 S1+1) and p(s++1/0(+1)
DEEP FREE ENERGY NETWORK (FENET)
Perceptual learning deals with learning the generative model best explain the agent's sensory inputs. If we think of not only observations but also actions given by the expert part of the sensory inputs, we can explain imitation leaning by using the concept of perceptual learning. Active inference deals with exploration and reward maximization, so it is compatible with reinforcement learning. By minimizing the same objective function, the Free Energy, we can deal with both imita- tion and RL.
In this section, we first introduce a policy prior for imitation and a policy posterior for RL. Second, extend the Free Energy Principle to be able accommodate these two policies in the same objective function, the Free Energy. Finally, we explain a detailed network architecture to implement the proposed method for solving image control tasks.
3.1 INTRODUCING POLICY PRIOR AND POLICY POSTERIOR
Free Energy We extend the Free Energy from (eq so that actions are part of sensory inputs that the generative model tries to explain.
F. KL(g(s))
st, a.))) 10:==KLNN KKl.leellllooossssaaaassllleeeee
(10)
((8))
Eg(s,) [In
(11)
=
Imp(aylss)) + ( En - Inp(a,)s))]]]]]] + ( K[g(())lpp()lss-nn 4(-1)
(12)
(13)
We define p(a, (x) as policy prior. When the agent observes expert trajectories, by minimizing Fr, the policy prior will be learned so that it can best explain the experts. Besides the policy prior, we introduce and define a policy posterior ((ar(se), which the very policy that the agent samples from when interacting with its environments. We explain how to learn the policy posterior in the following.
Expected Free Energy for imitation In a similar manner active inference in Section 2.2 think of the Expected Free Energy Ge+1 at the next time step t++1, 1, but this time we take expectation over the policy posterior g(aelst) because G44z is value expected under the next actions. Note that
3

---

Under review as a conference paper at ICLR 2021
in Section 2.2 ay given certain value, but here aj is sampled from the policy posterior. We calculate the expected variational posterior at time step t + 1 as follows.
((8t+1) Fy(er))(arle))
[x)]
(14)
g00+1 9141: a(41) ()(ser1)l((0 | (q(((+1)ss-1))
(15)
We extend the Expected Free Energy from (eq 12 so that the variational posterior makes inference on actions as follows.
1 = Eyfor Bufor1e41 Inpooo In ((((44118441) + +In (88+111,
Inp(8442 86, st)aa)]
(16)
B in - (4p Ing(e)
+ +KLig(s=))lppsstssss...
(17)
=
+ (L
(18)
(19)
+ K Mtt
(20)
In (eq 20), the first term is the entropy of the observation likelihood, and the second term is the KL
divergence between the policy prior and the policy posterior. By minimizing G the agent learns the policy posterior so that it matches the policy prior which has been leamned through minimizing Fr to encode the experts' behavior.
Expected Free Energy for RL We can get the Expected Free Energy in different way that has a reward component r(0++1) leading to the policy posterior maximizing rewards. We extend the Expected Free Energy from (eq so that the variational posterior makes inference actions as follows.
G
E
In Inp(att// (((1)))))))))))))))))))))))))( In ((+++1/0+))))--111/00000)))))))) In -
(21)
(aloe -
+ [11g)0+11)4)))//d4==+15111)
(22)
+
K |akp://||||i@@@ooeeeee/////////1
(23)
Egtorral - K |0410000000000000000000014
Bg( [
(24)
In a similar manner to active inference Section 2.2 we use p(o) expr(o) in (cq The
first KL term is the epistemic value that lets the agent explore the world, the second term is the expected reward under the action sampled from the policy posterior, and the last KI. divergence between the policy prior and the policy posterior. The last KI. term can be written as follows (eq[2)) 25). meaning that minimizing this term leads to maximizing the entropy of the policy posterior at the same time the policy posterior tries to match the policy prior. Thus, the expected free energy can be regarded as one of entropy maximizing RL methods.
the KI.
K KL(g(a-11sst) (lppp)))p((++)) ())+)) = - - H[q(a++1 (8441)) Eyco ulec ll (25)
Note that ((o++1) in (eq 24 can be calculated as follows.
q(o++1) = (plo::111)))
(26)
By minimizing GRI, the agent learns the policy posterior so that it explores the world and maxi- mizes the reward as long as it does not deviate too much from the policy prior which has encoded experts' behavior through minimizing Ft.
4

---

Under review as a conference paper at ICLR 2021
Frem Energy Calosbetio
Espected Tree Eneagy Calouaation
State pricr
at
a+++1
State pilor
qiaz posterior
plotos Mess) Porkey postorlor KX. Pokty prao Ploess
St
18++1
KL
Felicy orior p(cl/a,)
State porterior
oolted 1ow-1
a++1
4 (c((o))
Ot
Oc-1
Expected reward
data
(ot, at)
Figure 1: Deep Free Energy Network (FENet) calculation process. The left side shows how to calculate the Free Energy using data at hand. The right side shows how to calculate the Expected Free Energy for RL with latent imagination.
3.2 IMITATION AND RI OBJECTIVES
To account for the long term future, the agent has to calculate the Expected Free Energy at t + 1 to
F=
Σ Y G+
(27)
We define this curly F to be the objective that the Deep Free Energy Network should minimize. Note that y is a discount factor as case of general RL algorithms. As it is impossible to sum over infinity time steps, we introduce an Expected Free Energy Value function V(8++1) estimate the cumulative Expected Free Energy. Similarly to the case of Temporal Difference learning of Deep Q Network (Mnih et al. 2013), we target network Viarg($142) to stabilize the learning process and define the loss for the value function as follows.
L= 1[+++ + (Varrg(s+22) - ((8+1)111
(28)
We made design choice that the agent uses the value function only for RL, and not for imitation. In imitation, we use only the real value of the Expected Free Energy G++1 at the next time step + 1. This is because imitation learning be achieved without long term prediction as the agent given the experts' all time series data available. On the other hand, in RL, using the value function to predict rewards in the long-term future is essential avoid local minimum and achieve the desired goal.
In conclusion, the objective functions of Deep Free Energy Network (FENet) for data sequence (ot, Qe, Tt, 0r+1) are as follows.
F11. F, G
(29)
FRI. F, + G + + (8t+2)
(30)
L = ||G C + (VVarg(8442) ((8+1)111
(31)
The overall Free Energy calculation process is shown in Figure]
3.3 NETWORK ARCHITECTURE AND CALCULATION
For implementation, we made a design choice to use Recurrent State Space Model (Hafner et al. 2019b), a latent dynamics model with both deterministic and stochastic components. In this model, the hidden states 8, are split into two parts: stochastic hidden states and deterministic hidden states h4 The deterministic transition of h, modeled using Recurrent Neural Networks (RNN)
as follows.
h f(h--1, 81-1,
(32)
5

---

Under review as a conference paper at ICLR 2021
We model the probabilities in Deep Free Energy Networks as follows.
State prior
po(se
(33)
Observation likelihood
po(or]st, he)
(34)
Reward likelihood
po(re-ilse (1s,,)))),,))))))
(35)
State posterior
(s oz)
(36)
Policy prior
po po |se, he)
(37)
Policy posterior
qu (eelsse he) (ss)
(38)
Value network
(39)
Target Value Network
Vanarn (5t)
(40)
We model these probabilities as feedforward Neural Networks that output the mean and standard deviation the random variables according to the Gaussian distribution. The parameters 0, d, 1h w are network parameters to be learned. Using the network parameters, the objective loss functions can be written as follows.
FIL = Ft + G
(41)
FRL = Fi + Gf + Vanang (8t+2)
(42)
L = ||GH41 + +VVernng (81+2) (1((111)11
(43)
when
F, = -. (npol se h2) Ia lapo st, h.)] + (Lo 0z)
[po (h,))
(44)
G4 = Ey s
h h141 (ag
h++1)
(45)
G |o
+ (eg(e (K (at+1 1841, htw.://hhee........//./////////a
Varnng (84+2)
(46)
((5441) = ba (ar (Po (s++1 [h44)]
(47)
g(0141) = Eg(se=s) [Po [Po 1 |s+11, 1]]
(48)
Algorithm [ in Appendix shows overall caleulations using these losses. The agent minimizes FIL for expert data DE and the agent minimizes FRL for agent data DA that the agent collects on its own.
EXPERIMENTS
We evaluate FENet three continuous control tasks from images. We compare model with basel-based RL and model-based imitation RL dense and sparse reward setting when optimal expert is available. Then we compare model with imitation learning methods when only subop- timal experts are available. Finally, we investigate the merits of combining imitation and RL as an ablation study.
Control tasks We used Cheetah-run, Walker-walk, and Quadruped-walk tasks, image-based con- tinuous control tasks DeepMind Control Suite (Tassa et al., 2018)) shown in Figure 6 The agent gets rewards ranging from 0 to 1. Quadruped-ward walk is the most difficult as it has more action dimen- sions than the others. Walker-walk to stand up and then walk, meaning that the agent easily falls down on the ground, which is difficult to predict. The episode length 1000 steps starting from randomized initial states. We use action repeat R = for the Cheetah-run task, and R = 2 for the Walker-walk task and the Quadruped-walk task.
more challenging than Cheehtah-rui because an agent first has
4.1 PERFORMANCE IN STANDARD VISUAL CONTROL TASKS
We compare the performance of FENet to PlaNet (RL) and "PlaNet with demonstrations" (imitation RL) in standard visual control tasks mentioned above. We use PlaNet as a baseline method be- cause PlaNet is one of the most basic methods using Recurrent State Space Model, top of which

---

Under review as a conference paper at ICLR 2021
302
4.00
(a)cheetah-run
(b) walker-walk
(c) quadruped-wall
Figure 2: Comparison FENet to PlaNet and "PlaNet with demonstrations". Plots show test per- formance learning iterations. The lines show means and the areas show standard deviations over 10 trajectories.
30C
4000
10/0
10.00
Stenation
(a)cheetah-run
(b) walker-walk
(c) quadruped-walk
Figure 3: Comparison FENet to PlaNet and "PlaNet with demonstrations" in sparse- reward set- tings, where agents do not get rewards less than 0.5. Plots show test performance over learning iterations. FENet substantially outperforms PlaNet. The lines show means and the areas show stan- dard deviations over 10 trajectories.
build our model. As FENet uses expert data, create "PlaNet with demonstrations" for fair comparison. This variant of PlaNet has additional experience replay pre-pope peepoopulated with expert trajectories and minimize loss calculated from the expert data addition to PlaNet's original loss.
Figure 2 shows that "PlaNet with demonstrations" is always better than PlaNet and that FENet ranked higher as the difficulty tasks gets higher. In Cheetah-run, FENet gives competitive performance with PlaNet. In Walker walk, FENet and "PlaNet with demonstrations" are almost competitive, both which are substantially better than PlaNet thanks to expert knowledge being leveraged to increase sample efficiency. In Quadruped-walk, FENet is slightly better than the other two baselines.
4.2 PERFORMANCE IN SPARSE-REWARD VISUAL CONTROL TASKS
In real-world robot learning, is demanding to craft a dense reward function lead robots to desired behaviors. It would be helpful if an agent could acquire desired behaviors simply by giving sparse signals. We compare the performance of FENet to PlaNet and "PlaNet with demonstrations" in sparse-reward settings, where agents do not get rewards less than 0.5 per time step (Note that in the original implementation of Cheetah-run Walker-walk and Quadruped maalk, walk, agents get rewards ranging from 0 to per time step). Figure demonstrations" all three tasks. In Cheetah-run, PlaNet and "PlaNet with demonstrations" are not able to get single reward.
B shows that FENet outperforms PlaNet and "PlaNet with

---

Under review as a conference paper at ICLR 2021
Figure 4: Comparison of FENet to imitation learning methods when only suboptimal experts are available in Cheetah-run. Plots show test performance over learning iterations. Behavioral Cloning imitation methods cannot surpass the suboptimal expert's return which FENet successfully sur- passes. The lines show means and the areas show standard deviations over 10 trajectories.
4.3 PERFORMANCE WITH SUBOPTIMAL EXPERTS
In real-world robot learning. expert trajectories are often given by human experts. is natural to assume that expert trajectories suboptimal and that there remains much room for improvement. We compare the performance of FENet to Behavioral Cloning imitation methods. We use two types of networks for behavioral cloning methods: recurrent policy and recurrent decoder policy. The recurrent policy n (o(o))))) is neural networks with one gated recurrent unit cell and three dense layers. The recurrent decoder policy a 0t+1/o,) is neural networks with one gated recurrent unit cell and four dense layers and deconvolution layers as in the decoder of PlaNet. Both networks does not get pixel observations but take observations encoded by the same convolutional encoder as PlaNet's.
Figure shows that while imitation methods overfit to the expert and cannot surpass the suboptimal expert performance, FENet is able to substantially surpass the suboptimal expert's 's performance.
4.4 LEARNING STRATEGIES
Figure compares learning strategies of FENet in Cheetah-run and Walker-walk (ablation study). "Imitation RL' is the default FENet agent that does imitation learning and RL at the same time, minimizing FIL + FRL. Imiitaion--retrainee RL' is an agent that first learns the model only with imitation (minimizing FIL) and then does RL using the pre- trained model (minimizing FRL). RLL only' an agent that does RL only, minimizing FRI. 'Imitation only' is an agent that does imitation only, minimizing FrL. While 'imitation only' gives the best performance and 'imitation RL gives the second best in Cheetah-run, 'imitation RI. gives the best performance and 'imitation only' gives the worst performance in Walker-walk, We could say 'imitation RL' the most robust to the properties of tasks.
5 RELATED WORK
Active Inference Friston, who first proposed Active Inference, has evaluated the performance in simple control tasks and a low-dimensional maze (Friston al.) 2012. 2015). Ueltzhoffer imple- mented Active Inference with Deep Neural Networks and evaluated the performance in a simple control task (Ueltzhöffer, 2018). Millidge proposed Deep Active Inference framework with value functions to estimate the correct Free Energy and succeeded in solving Gym environments (Mil- lidge 2019). Our approach extends Deep Active Inference to combine imitation and RI, solving more challenging tasks.
RL from demonstration Reinforced Imitation Learning succeeds in reducing sample complex- ity by using imitation as pre-training before RL (Pfeiffer al.) 2018 Adding demonstrations into a replay buffer of off policy RL methods also demonstrates high sample efficiency (Vecerik et al, 2017 Nair et al. 2018 Paine et al. 2019). Demo Augmented Policy Gradient mixes the pol- icy gradient with a behavioral cloning gradient (Rajeswaran" et al., 2018). Deep Q-learning from Demonstrations (DQfD) not only use demonstrations for pre-training but also calculates gradients
8

---

Under review as a conference paper at ICLR 2021
3007
(a)chectah-run
(b) walker-walk
Figure 5: Comparison of FENet (imitation RL) to other learning strategies (ablation studies: imitation-pretrained RL, RL only, and imitation only with FENets). Plots show test performance over learning iterations. The lines show means and the areas show standard deviations over 10 trajectories.
from demonstrations and environment interaction data (Hester et al. 2018). Truncated HORizon Policy Search uses demonstrations shape rewards so that subsequent planning can achieve perior performance to RL even when experts are suboptimal (Sun et al. 2018). Soft Q Imitation Learning gives rewards that encourage the agent to return to demonstrated states in order to avoid policy collapse (Reddy et al. (2019). Our approach is similar to DQfD in terms of mixing gradi- ents calculated from demonstrations and from environment interaction data. One key difference that FENet concurrently learns the generative model of the world so that it can be robust to wider environment properties.
Control with latent dynamics model World Models acquire latent spaces and dynamics over the spaces separately, and evolve simple linear controllers to solve visual control tasks (Ha Schmid- huber 20188) PlaNet learns Recurrent State Space Model and does planning with Model Predictive Control at test phase (Hafner et al.) 2019b). Dreamer, which is recently built upon PlaNet, has a policy for latent imagination and achieved higher performance than PlaNet (Hafner al.) 2019a). Our approach also uses Recurrent State Space Model to describe variational inference, and we are the first to combine imitation and RL over latent dynamics models to the best of our knowledge.
CONCLUSION
We present FENet, an agent that combines Imitation Learning and Reinforcement Learning using Free Energy objectives. For this, we theoretically extend the Free Energy Principle and introduce policy prior that encodes experts' behaviors and policy posterior that learns to maximize expected rewards without deviating too much from the policy prior. FENet outperforms model-based RL and imitation RL especially in visual control tasks with sparse rewards and FENet also outperforms suboptimal experts' performance unlike Behavioral cloning. Strong potentials in sparse environment with suboptimal experts are important factors for real-world robot learning.
Directions for future work include learning the balance between imitation and RI, i.e. Free Energy and Expected Free Energy so that the agent can select the best approach to solve its confronting tasks by monitoring the value of Free Energy. It is also important evaluate FENet in real-world robotics tasks to show that our method is effective in more realistic settings that truly appear in the real world.
REFERENCES
Karl Friston. The free-energy principle: a unified brain theory? Nature reviews neuroscience, (2):177-138, 2010.
Karl Friston, James Kilner, and Lee Harrison. free energy principle for the brain. Journal of Physiology-Paris, 100(1-3):70-87, 2006.

---

Under review as a conference paper at ICLR 2021
Karl Friston, Spyridon Samothrakis, and Read Montague. Active inference and agency: optimal control without cost functions. Biological cybernetics, 106(8-9):523-5M4) 2012.
Karl Friston, Francesco Rigoli, Dimitri Ognibene, Christoph Mathys, Thomas Fitzgerald, and Gio- vanni Pezzulo. Active inference and epistemic value. Cognitive neuroscience, 6(4):187-214, 2015.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:/803. /0/22, 2018. 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01003 2019a.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 2555-2565, Long Beach, California, USA, 2019b. PMLR.
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In Thirly-Second AAAI Conference Artificial Intelligence, 2018.
Steven Kapturowski, Georg Ostrovski, Will Dabney, John Quan, and Remi Munos. Recurrent perience replay in distributed reinforcement learning. In International Conference on Learning Representations, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6900 2014.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv: 18t5::0880000000 2018
Beren Millidge. Deep active inference variational policy gradients.
arXiv preprint
arXiv:1907.03876, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602. 2013.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In 2018 IEEE International Conference Robotics and Automation (ICRA), pp. 6292-6299. IEEE, 2018.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807-814, 2010.
Tom Le Paine, Caglar Gulcehre, Bobak Shahriari, Misha Denil, Mat Hoffman, Hubert Soyer, Richard Tanburn, Steven Kapturowski, Neil Rabinowitz, Duncan Williams, et al. Making effi- cient use of demonstrations solve hard exploration problems. arXiv preprint arXiv:1909.012. 0937799999877 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vetioo, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Mark Pfeiffer, Samarth Shukla, Matteo Turchetta, Cesar Cadena, Andreas Krause, Roland Siegwart, and Juan Nieto. Reinforced imitation: Sample efficient deep reinforcement learning for mapless navigation by leveraging prior demonstrations. IEEE Robotics and Automation Letters, 3(4): 4423 4430, 2018
Aravind Rajeswaran", Vikash Kumar", Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforce- ment Learning and Demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2018.
10

---

Under review as a conference paper at ICLR 2021
Siddharth Reddy, Anca D Dragan, and Sergey Levine. Sqil: imitation learning via regularized behavioral cloning. arXiv preprint arXiv:1905.111000 2019.
Wen Sun, J. Andrew Bagnell, and Byron Boots, Truncated horizon policy search: Combining reinforcement learning & imitation learning. In International Conference on Learning Represen- tations, 2018
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXiv preprint arXiv: www.v::181.0006600 2018.
Kai Ueltzhöffer. Deep active inference. Biological cybernetics, 12(6)):577-573 2018.
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang. Olivier Pietquin, Bilal Piot, Nico- las Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstra- tions for deep reinforcement learning on roboties problems with sparse rewards. arXiv preprint 1rXiv:170.008817, 2017.
A APPENDIX
(a) Cheetah-run
(b) Walker-walk
(c) Quadruped-war
Figure Image-based control tasks used in our experiments.
A. FENET ALGGRRITH
See Algorithm !
A.2 IMPLEMENTATION
To stabilize the learning process, we adopt burn-in, technique to recover initial states of RNN's hidden variables he (Kapturowski al. 2019). As shown in Algorithm [ the agent calculates the Free Energy with mini batches sampled from the expert or agent experience replay buffer D, which means that h2 is initialized randomly in every mini batch calculation. Since the Free Energy heavily depends on ht, it is crucial to estimate the accurate hidden states. We set a burn-in period when a portion of the replay sequence is used only for unrolling the networks produce initial states, After the period, we update the networks only on the remaining part of the sequence.
We PyTorch (Paszke al. 2017) write neural networks and run experiments using NVIDIA GeForce GTX 1080 Ti / RTX 2080 Ti Tesla V100 GPU (1 GPU per experiment). The training time for our FENet implementation is about 24 hours on the Control Suite environment. As for the hyper parameters, we use the convolutional encoder and decoder networks from (Ha & Schmidhu- ber, 2018) and Recurrent State Space Model from (Hafner al. 2019b) and implement all other functions as three dense layers of size 200 with ReLU activations (Nair & Hinton, 2010). We made a design choice to make the policy prior, the policy posterior, and the observation likelihood, the reward likelihood deterministic functions while the state prior and the state posterior are stochastic. We use the batch size B = 25 for 'imitation RL' with FENet, and B = 50 for other types and baseline methods. We the chunk length L 50. the burn-in period 20. We seed episodes 40, expert episodes N = 10000 trained with PlaNet (Hafner et al. 2019b). collect interval C = 100 and action exploration noise Normal(O, 0.3). We the discount factor 7 0.99 and the
11

---

Under review as a conference paper at ICLR 2021
Algorithm Deep Free Energy Network (FENet)
Input:
Initialize expert dataset Dg with expert tra-
Seed episodes S
jectories
Collect interval C
Initialize agent dataset DA with S random
Batch size B
episodes
Chunk length L
Initialize neural network parameters 8, o, dh randomly
Expert episodes N
Target smoothing rate Learning rate a
while not converged do for update step c 1..C do
State prior (sel/tt
/ Imitation Learning
State posterior qo (sel/te, 0t)
Draw expert data { Ot, Tt. 0141) Jim1
1-L11
Policy prior pe (aa (s.ss,. h4)
DE
Policy posterior qu(a, (sc. h2)
Compute Free Energy FIL from equation
Likelihood pa (o(or 188, (,),, po (Te- 88, h2)
41
Value function (1))
// Reinforcement Learning
Target value function V Vineay (9t)
Draw agent data {(o Tt, 0141) Ji=1
k+
DA
Compute Free Energy FRL. from equation
42
Compute V function's Loss C from equa-
tion 43
// Update parameters
0 + aVo(FIL + FRL).
o t 6 aVo(FIL + FRL)
th
oV + FFL)
w + w avaL
Wtarg pubarg + (1 p)- end for
// Environment interaction 01 + anv.reset. () for time step 1..T do
Infer hidden states 8; +- q (hi Calculate actions ag < q (at 8t, hi)
Add exploration noise to actions
Tt. 0++1 f eeveeseep (at) end for
DA + DA U {(or, ar rt, (141) end while
12

---

Under review as a conference paper at ICLR 2021
target smoothing rate 0.01. We use Adam (Kingma & Ba 2014 with learning rates 10-3 and scale down gradient norms that exceed 1000. We scale the reward-related loss by 100, the relicy-prior-related loss by 10. We clip KL loss between the hidden states below 3 free nats and clip KI. loss between the policies below 0.6.
13


Reviews:
[{"review_id": "Z-8KAV9hQJS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper extends and explains how to apply the \"free energy principle\" and active inference to RL and imitation learning. They implement a neural network approximation of losses derived this way and test on some control tasks. Importantly the tasks focus on here are imitation + control tasks. That is, there is both a reward signal but also demonstration trajectories. The demonstrations may be suboptimal. The compare against PLaNet, a latent planning based approach.\n\nIt was difficult to evaluate this work well, as I found the approach difficult to follow. My primary concern about this paper is connecting it with other work and ensuring the terminology and approach is as clear as possible.\n\nFirstly, in the initial derivation up to equation 9. By defining the observation prior $p(o) \\propto \\exp r(o)$ I believe this results in the same thing as \"RL as inference\" (e.g. [1] which states that \"formulation proposed by Friston (2009) is similar to the maximum entropy approach outlined in this survey\"). It would be good to make this connection clear for the reader. In particular, at least from my perspective, connecting to existing maximum entropy RL is helpful for understanding this work.\n\nHowever, this leads to a concern of terminology. I realize the use of the term \"observation\" for $p(o)$ is to maintain the connection with the free energy formulation, but if the \"observation\" is really just the reward, then in the context of an RL paper this seems to lead to a lot of confusion if when you say \"observation\" you really just mean reward. In other places observation seems to mean the observation in a POMDP (e.g. in the algorithm, where reward is a separate variable).\n\nAnother source of confusion is using $s_t$ to refer to both the entire hidden state and just the stochastic part (3.3).\n\nIt would helpful to define the RL problem formally that is attempted to be solved here. It appears that is a partially observed MDP (which makes the use of \"observation\" even more confusing). It is not clear the the test environments are particularly partially observable, since the image reveals the full state of the world and these tasks have been solved with models that don't deal with partially observable states.\n\nI think another view of this paper is that it is applying maximum entropy RL approach to a partially observed MDP, learning a demonstration policy, and then using this demonstration policy as a prior on an RL objective policy. Approaching this topic from a different perspective is helpful, but it would be useful to make connection with existing literature much clearer and state an alternative way of viewing the approach from a more standard RL point of view. Alternatively, if these statements are incorrect contrasting what is different would benefit the reader.\n\nThe experimental comparisons seem weak for a few reasons. There seems to be a conflation between model-based RL solutions and RL approaches dealing with partial observability. As far as I can understand the approach introduced here is the second, a state transition function is learned but this never used for planning, only for inferring the hidden state of the world. For this reason, the only comparison being with a model-based control approach seems lacking. It would be interesting to compare against other model-free approaches that incorporate demonstrations such as [3] (this does not deal with partial observability, but it's not clear these problems are partially observable) or [4] ([2] is an example of using a GAIL objective as in [4] + and a reward as is done here). Finally, it appears one of the significant benefits of the algorithm introduced here is to work well with partially observed MDPs, it would be good to construct environments which test this more clearly.\n\nOverall, I think this work introduces interesting ideas. However, it needs to connect this with existing work such as RL as inference view of RL and make the approach clearer. The experimental section would benefit for more comparison with other methods and experiments on environments which are more straightforwardly partially observable.\n\n[1] Levine, Sergey. \"Reinforcement learning and control as probabilistic inference: Tutorial and review.\" arXiv preprint arXiv:1805.00909 (2018).\n\n[2] Zhu, Yuke, et al. \"Reinforcement and imitation learning for diverse visuomotor skills.\" arXiv preprint arXiv:1802.09564 (2018).\n\n[3] Vecerik, Mel, et al. \"Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards.\" arXiv preprint arXiv:1707.08817 (2017).\n\n[4] Torabi, Faraz, Garrett Warnell, and Peter Stone. \"Generative adversarial imitation from observation.\" arXiv preprint arXiv:1807.06158 (2018).", "extra_scores": {}, "extra_text": {}}, {"review_id": "JTk4S_Fv1EY", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This manuscript develops RL algorithms that can learn reward from expert data and even exceed the expert performance. It introduces free energy framework that combines ideas of imitation learning and reinforcement learning in a Bayesian probabilistic way. The methodology is well written. It is interesting that they find the algorithm can deal with both imitation learning and RL by minimizing the same Free energy objective function.  The expert data provides policy prior. Then, the policy posterior will try to match the policy prior as the imitation learning, and at the same time, it is motivated to maximize the reward and explore the environment by maximizing the KL divergence between the state posterior and the state posterior given imaged observations. \n\nWith the proposed objective function, the algorithm applies existing recurrent state space model to solve the problem. Though it applies existing model, it would enhance the paper if it can show further analysis of the algorithm in this context, particularly from perspective of imitation learning. And the paper may need to enhance the empirical study by testing more problems and provide convincing analysis.\n\nThe math notations could be more consistent, as in Section 3 it widely uses variables like q(s_{\\tau}), q(s_{\\tau}|o_{\\tau}), p(a_{t}|s_{t}), but in Appendix, there is no  q(s_{\\tau}), it defines the variables like q_{\\phi}(s_{t}|h_{t}, o_{t}), p_{\\theta}(a_{t}|s_{t}, h_{t}) etc. \n\nIt may lack details on the the gradient of energy function. In Algorithm 1, there are important steps to update the parameters, it is not straightforward to find how to get the gradient from the functions 27 - 29.\n\nIn section 3.2, it may need further clarification on why imitation learning doesn’t need longer term objective. The assumption seems not straightforward, as usually the agent mimics the expert behavior to guarantee its expected long term reward is similar as the expert.\n\nThough the abstract mentions the proposed algorithm can achieve better performance than standard IL or RL, there isn’t sufficient evidence. It may need theoretical analysis or rich empirical results. The experiments could study more simulation problems, and may compare with state-of-the-art inverse RL algorithms. Other questions related to experiments include:\n- How to set up the expert data for  “PlaNet with demonstrations” in Section 4.2?\n- Can experiments show the reward shaping benefit using virtualization to compare different reward estimations and the ground truth?\n- The behavior cloning algorithms compared in Section 4.3 seem basic in the field, it may be worth comparing with “PlaNet with suboptimal expert data”, or other inverse RL algorithms. \n- In Section 4.4, it may need to provide reference to the algorithms in study. Are they known algorithms, eg. which imitation RL algorithm is used,  which algorithm is the imitation pretrained RL (which algorithm is used to learn model and which RL is employed  after that), and which RL algorithm is used in ‘RL only’. \n\nSome questions are written below.\n- On page 2, it could make clear which parameters are optimized to minimize the Free Energy function of Eq. (4)\n- In Eq. (8) when it replaces p(s_{\\tau}|o_{\\tau}) with q(s_{\\tau}|o_{\\tau}), it may be better to explain how to get the approximation and whether there is error bound.\n- It would be interesting to know the convergence analysis of Algorithm 1, is it always convergent.\n- Section 3.1 mentions we give a_t a prior in Section 2.2, but it seems there isn't a prior of a_t in section 2.2.\n- It may be worth explaining q(s_{\\tau}) in Eq. 14, is it equal to the variational posterior  as an approximation of p(s_{t+1}|o_{t+1})  on page 2.?\n- It may need explanation on how to reach Eq.15 from Eq.12? Is it directly derived from KL(q(o_{\\tau}, s_{\\tau}, a_{\\tau})||p(o_{\\tau}, s_{\\tau}, a_{\\tau})?\n- Figure 1 may lack good explanation, as the graph looks complicated with many math notations, not self-explanatory.\n- The notation $s_{t}$ in section 3.3 may be confusing as $s_{t}$ refers to both hidden state and the stochastic part of it. In Figure 1, the deterministic transition only depends on h_t and a_t, while it depends on s_{t-1} too in  Eq.30.\n- In quadruped-walk experiments, like Figure 3,  it seems the solution is not convergent yet. Why do FEnet and planet w/demo have less number of iterations?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "BK_EjD3dSY", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper  introduces two different interpretations of free energy minimization as a form of behavior cloning and reinforcement learning.  \n\nStrength:\nThis approach seems to have significant gains on the environments evaluated.\nThe approach appears novel to my knowledge.\n\nWeaknesses:\nI found that I was confused by the presentation of section 3.1. In particular, I think the authors should clarify the difference between prior and posterior policies in both the RL and imitation learning setting as they appear to be different.\n\nWhy does equation 17 to 18 follow? Isn't the posterior policy different than the policy prior, leading to the likelihood to the next state this is distinct from the prior probability of the next state?\n\nFor equation 22 to 23, is the assumption that the likelihood of the next state is proportional to inverse exponentiated reward? I think the statement should be said in the text.\n\nHow does the approach compare with other approaches that encourage entropy in the policy? Such as something like soft Q learning? Or some type of curiosity?\n\nCan this approach be evaluated on more realistic environments other than deepmind control? \n\nPost Rebuttal Update:\n\nDue to the remaining confusion among reviewers about the equations in the manuscript, I maintain my score.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "6aOeaNg6_Ni", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims at merging Imitation Learning (IL) and Reinforcement Learning (RL) from high dimensional sensory inputs so that the agent can make use of expert trajectories, even when the experts are suboptimal. This paper aims at addressing this by theoretically using the \"Free Energy Principle\", which the authors define in the abstract as a unified brain theory that explains perception. \n\nThe paper tackles an important problem and develops many theoretical functionals. The approach is then tested on a few benchmarks from the DeepMind Control Suite where the approach is reported to work well.\n\nMy main concerns are related to the clarity of the paper, which does not allow me to understand some key parts.\n\nIt is unclear what are the minimized loss functions: it is mentioned that equations 27-29 sum up all objective functions. Since F_t and G_{t+1}^{RL} are appearing twice, does it mean that those losses have a two times more important contribution than for instance G_{t+1}^{IL}? In addition how exactly are equations 27-29 derived from the other equations?\n\nNotations are not always consistent:\n- F_{IL} F_{RL} are sometimes with curly F sometimes not (equations 27-29 and the few lines that follow Figure 1 page 5).\n- First line equation 7 and first line equation 10: Why is there a subscript for F and not G? In other parts of the paper, a subscript is used for G as well.\n\nSome sentences are unclear:\n- \"in RL, using the value function to predict rewards in the long-term future is essential to avoid a local minimum and achieve the desired goal.\" What kind of local minimum does that refer to?\n\nA few typos:\n- \"Then We (...)\"\n- \"the agent easily fall down\"", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new algorithm that combines imitation learning and reinforcement learning, based on an extension of the free energy principal. The expert's demonstrations are encoded as a policy prior, and a posterior policy is inferred by maximizing expected rewards. While at a high-level this is a promising direction, all the reviewers found the paper difficult to follow and verify its claims. This mostly due to a use of unusual and non-conistent notations. The authors are advised to take into account the issues about clarity that the reviewers raised and improve the readability of their paper accordingly.
Decision: Reject