Hellinger Distance Constrained Regression
Authors: Egor Rotinov
Venue:  (2021)
Keywords: offline
Reinforcement Learning
off-policy
control

Abstract: This paper introduces an off-policy reinforcement learning method that uses Hellinger distance between sampling policy (from what samples were collected) and current policy (policy being optimized) as a constraint. 
Hellinger distance squared multiplied by two is greater than or equal to total variation distance squared and less than or equal to Kullback-Leibler divergence, therefore a lower bound for expected discounted return for the new policy is improved compared to the lower bound for training with KL. 
Also, Hellinger distance is less than or equal to 1, so there is a policy-independent lower bound for expected discounted return. 
HDCR is capable of training with Experience Replay, a common setting for distributed RL when collecting trajectories using different policies and learning from this data centralized. 
HDCR shows results comparable to or better than Advantage-weighted Behavior Model and Advantage-Weighted Regression on MuJoCo tasks using tiny offline datasets collected by random agents. On bigger datasets (100k timesteps) obtained by pretrained behavioral policy, HDCR outperforms ABM and AWR methods on 3 out of 4 tasks. 

Under review as a conference paper at ICLR 2021
HELLINGER DISTANCE CONSTRAINED REGRESSION
Anonymous authors
Paper under double-blind review
ABSTRACT
This paper introduces an pollcy reinforcement learning method that uses Hellinger distance between sampling policy (from what samples were collected) and current policy (policy being optimized) as constraint. Hellinger distance squared multiplied by two is greater than or equal to total variation distance squared and less than or equal to Kullback-Leibler divergence, therefore a lower bound for expected discounted return for the new policy is improved compared to the lower bound for training with KL. Also, Hellinger distance is less than or equal to so there is policy-independen lower bound for expected discounted return. HDCR is capable of training with Experience Replay, a common set- ting for distributed RL when collecting trajectories using different policies and learning from this data centralized. HDCR shows results comparable to or better than ddvantage-weightee
Behavior Model and Advantage- Advantage-Weighted Regression on MuJoCo tasks using tiny offline datasets collected by random agents. On big- ger datasets (100k timesteps) obtained by pretrained behavioral policy, HDCR outperforms ABM and AWR methods on 3 out of 4 tasks.
INTRODUCTION
Policy gradient algorithms are methods of model-free reinforcement learning that optimize policy through differentiating expected discounted return. Despite the simplicity, to converge, these meth- ods should stay on-policy because of the first-order approximation of state visitation frequencies. This issue makes agents learn through trial-and-error, using data only once.
To make policy gradient updates more off-policy, we can add a constraint on the update to decrease the step size if the current policy is too far from the sampling policy. One of the first methods was to add total variation distance squared as a constraint for mixture policies. Later it was proven that there is a lower bound for new policy's expected discounted return (Kakade & Langford 2002). Recently it was proven that this lower bound exists for all types of updates (Schulman et al. 2015
Next, total variation distance squared was replaced by Kullback-Leibler divergence that is greater than or equal to the previous one (Pinker's inequality (Levin & Peres 2017)) so that the lower bound was decreased (Schulman al., 2015). Using Lagrangian, have been derived off off-policy method called Advantage- Weighted Regression (Peng al. 2019). which also used KI as a constraint.
This article proposes new method whose lower bound expected discounted return greater than or equal to the bound with KL. We achieve this by replacing total variation distance by Hellinger distance, which decreases lower bound. Therefore strictness stays the same. Then we derive an off- policy method called Hellinger Distance Constrained Regression using the new constraint. It can be used on discrete and continuous action spaces since derivation uses Lebesgue integrals rather than a summation or Riemann integrals.
PRELIMINARIES
To better present the problem, we start from basic definitions, go through the history of improve- ments, and then describe the disadvantages of using KL divergence as a constraint.
We consider infinite-horizon discounted Markov decision process (MDP), defined by the tu-
ple (S, A, P. r, Pn. 7), where is a set of states (finite or infinite), .A is a set of actions (finite infinite), P S
S R is the transition probability distribution, r S - R is

---

Under review as a conference paper at ICLR 2021
the reward function, Pu S R is the distribution of the initial state su. and (0, 1) is the discount factor.
Let denote stochastic policy S A
[0, 1}, and then its expected discounted return is:
nT(T ao........
Σ Ent((s))
where
(1)
90 po(+); as x (-1se)) s141 P()ss,,a))
This paper uses state-action value function Qr state value function Vi, and advantage function A with the following definitions:
Qx 0i) Bect ac+t............................
Ar(s
V=((t) E
tr(sce))
As(5t, 04) Q=(st, ae)
V=()).
Let Pa (s) be unnormalized visitation frequencies state s where actions are chosen according to m:
Pr
P(s s).
Following identity expresses the expected return of another policy म in terms of the advantage over n, accumulated over states (see Schulman al. (2015) for proof);
nf ()) n7()) f p*( /
[sals)An da ds.
In approximately optimal learning, replace state visitation frequency p# by Pa. since this drasti- cally decrease optimization complexity:
L (x(#)) = (7())) + pa(s) a ((als)) da ds.
(5)
Let wold denote current policy, then the lower bound for the expected discounted return for the new policy Nnww will be (see Schulman et al. (2015) for proof):
4ey
([new) 2 Lixiia (Tnew)
7)2 a
where e = max A (s, a)l:
8,0
(6)
max Drv (Totd(-)s)) (-1s)),
1 [Tad(als) knew (a|s) da.
(Ty Tnew (-18)
2
Theoretical Frust-Region Policy Optimization algorithm relays on Pinsker's inequality (see Tsy-
bakov 2009) for proof):
D (Tota (T 2 Drv (Fata-(( (Teeu (-1s))²
where DKL (Toia (-1s) (-sw())s)) = f (ota(als) log (awa(a(s) da.
Nnew (als)
To retain strictness and decrease calculation complexity, total variation distance squared was re- placed with Kellback--Libbee divergence D (
2

---

[s)) with Under review as a conference paper at ICLR 2021
[FFnww)) 2 Tomond P (Tnew) CDRL (Taldll
where max [Ax(s,)),,
5,a
4
= (1 7)22
DKL (Told) Tneww)
= max D KT. (Tara(- Tnew (-1s).
However, this replacement greatly decreases the lower bound for the expected discounted return for the new policy, Moreover, Kullback-Leibler divergence has no upper bound. Therefore we have no policy-independent lower bound for this type of update.
HELLINGER DISTANCE IN POLICY OPTIMIZATION
We can improve lower bound (compared to KL) by replacing Drv (Tooaa( [s) || Tnew (-
Hellinger distance H (Toidd-]s)) Tnew (- s)):
H (Toza (s) || Tnew (-1s))² =1 (11s))²=11- /
Tota(als) Fnewwaas da
(9)
Theorem [ (see Appendix A or (Tsybakov 2009
section 2.4) for proof) proves correctness and
improvement (compared to KL) to the lower bound.
Let p(v) and q(v) be two probability density functions then:
Drv(p()) xt)))² < (p((()) (1)))))) s (Rz(p() (l(())
(10)
Replacing p(v) and g(v) with T-ta(- s) and Knew(- 8) respectively, new lower bound follows:
SEY
([ 2 Lisore (Tnww)
(1 7)3
where max | (8, a));
(11)
a max
is worth to note that H (Tote 8) I| Tnew (. [s) S 1.
HELLINGER DISTANCE CONSTRAINED REGRESSION (HDCR)
We could use presented lower bound as in TRPO, but instead, we derive offline regression algo- rithm by introducing the following optimization problem where 4 is the sampling policy:
arg
Pm(s) x(als)A, a) da ds
s.t. / Pu(()) ( ds Sc
(12)
/ [(als) da 1, V.
Constructing Lagrangian, differenntaaing it with respect to n, and solving for n gives us the following optimal policy (see Appendix B for derivation):
g2
t (als) = u{ass)
where / is a Lagrangian multiplier.
(13)
[3
2
Constructing a regression problem of KL divergence between optimal policy "' and current pol- icy and simplifying gives us the following supervised regression problem (see Appendix B for derivation):
arg max Esnpe()) Earz((1s) log (Tals) (B Ap(s,a))2
(14)
3

---

Under review as a conference paper at ICLR 2021
Using notation of ABM paper (Siegel al. a()(s,a) = (0-A(s,a))2
2020
"udvantage-weighhing" function
If we use HDCR with Experience Replay, in equation
14
replace u(-1s) in expecta-
tion Eg ~ nl a) and advantage function Au(s, a). Let of sampling policies from which actions were sampled, w(T)) probability of selecting policy Ti then:
( #s+1, T&+N} be a set
((s,a)= J ((TT) di
Ap(s.a) a) Jn ((()))(()))) (Q+ V+(()) dr Jn w px) dv
(15)
V(s) Sn pm di
Sn w (( px) dm
The proof will repeat the proof for AWR with Experience Replay (Peng et al., 2019).
Practically we simply sample uniformly from the replay buffer and using value function estimator. This type of sampling provides us an approximation of expectation and state value function.
Let 'D denote a set of stored trajectories (replay buffer), A(s, a; Φ) is an advantage function parame- terized by vector Φb
The most popular method to obtain A(s, a; Φ) for offline learning is to use state-action function estimator Q(x, a; parameterized by @b:
A(s, Q(s,a; (aa (s,)))(ssss dd)
(16)
Despite Q(s, a; () being closer to the expectation of discounted return following policy (because of "taking" the first action according to æ rather then μt), we found Monte-Carlo return more effi- cient on tiny offline datasets. Greater performance using MC return can be explained by lack of experience "produced" by certain actions.
Monte-Carle estimation of A(s, Φ) can be described follows where R and (s; Φ) is state value function estimator parameterized by vector do:
Eto Tret
A(s, a; σ) = R.P. - (x;
(17)
Also, we use Generalized Advantage Estimation (Schulman al. 2016), where y 8thh is one-step temporal difference for state s calculated using old vector Φu:
[0, and
TI + (V dx) V(si: Φk)
T
Å(ss, as Φk)
E
(18)
A(st, ar o) A(s a;; + V(s Φx) - ((s;;)
Finally, we propose the following reinforcement learning algorithm:
Algorithm 1: Hellinger Distance Constrained Regression
0, +- random initial weights
D 4
for iteration k = 1, ..., kmaa do add trajectories {r.} sampled via To, to de+1 arg ming Ega-DA (8)
f++1 + arg max, Ey,anD
[log (als)) (8 -
end

---

Under review as a conference paper at ICLR 2021
Heppersd
Jet-2
480
230
130
ABM
AWR
HDCR (Ours)
Figure 1: Learning curves of ABM, AWR, and HDCR averaged across results of learning from 10 different datasets 10k timestamps (also seeds used to generate each dataset).
Task
ABM
AWR
HDCR (Ours)
Ant-v2
468 72
495 84
569 + 64
HalfCheetah-v2
-7 3
-7 4
-8 - 6
Hopper-v2
184 = 72
209 31
223 141
Walker2d-v2
132 76
125 + 92
145 + 117
Table 1: Final returns for different algorithms, with + corresponding to one standard deviation of the average return across 10 datasets of 10k timestamps.
EXPERIMENTS
In our experiments, we evaluate the algorithm on MuloCo (odorov et al., 2012) tasks.
5.1 TINY DATASETS
For evaluating on extremely small datasets we use setting inspired by Behavioral Modelling Priors for Offline Reinforcement Learning paper (Siegel et al.) 2020) but instead of using actions from behaviorial policy we use random actions while generating buffer.
First, collect 2048 timestamps more, until episode termination (whichever occurred later), from each of 5 seeds using random actions. Then we load collected trajectories to a replay buffer for the agent training. Separate networks with the same architecture (except the last layer) represent the policy and value function and consist of 2 hidden layers of 256 ELU units. Each train iteration only old data obtained by random agents. Each iteration, the value function is updating with gradient steps and policy with 50 steps using a uniformly sampled batches of 512 samples using all data from the replay buffer. Learning rates for Adam optimizer are x 10 and x 10 for critic and actor, respectively.
We compare 3 different "advantage--wighh" functions (AA(s,))):
Hellinger Distance Constrained Regression, where ((A(s,a) = (3-A(a0)]=
wevintage-weighted Behavior Model, where f(A(s, a)) = LA(s,a)20 Lado = 1 otherwise 4250 0;
Regression, where f(A(s, a)) exp A(s,a).
AWR method uses 3 = 1.0 as it is in implementation released by authors. HDCR uses 3 = 1.0. For TD(A) we use = 0.95.
On simple tasks Hopper-v2, all methods are able to leam (Figure D. and HDCR shows slightly better results [T. While on difficult tasks as Ant-v2, all algorithms do not improve their results through iterations. Moreover, evaluation returns decrease.
5.2
LARGE DATASETS
Next, we perform tests using buffers with the size of 100k timesteps. Buffer is filled by running pretrained behavioral policy. This setting replicates the setting from Off-Policy Deep Reinforce-
5

---

Under review as a conference paper at ICLR 2021
Hagperve
APH-2
Walterrd
7.0
11.0
1750
2.20
070
ABM
AWR
BCQ
HDCR (Ours)
Figure 2: Curves BCQ, ABM, AWR, and HDCR evaluation results averaged across 3 seeds.
Task
ABM
AWR
HDCR (Ours)
Ant-v2
179 45
281 + 25
217 = 54
HalfCheetah-v2
3358 + 76
3365 + 193
3485 - 205
Hopper-v2
495 + 115
478
576 + 123
Walker2d-v2
592 + 139
762 + 64
805 - 62
Table 2: Final returns for different algorithms, with + corresponding to one standard deviation of the average return across 3 seeds.
ment Learning without Exploration (Fujimoto et al. 2019) paper. Therefore we also provide results of BCQ method achieved by authors' implementation trained from the same datasets. For calculat- ing advantage obtained by 10 actions sampled from the policy.
use equation 16 where we approximate integral by taking mean of 10 Q-values
While BCQ outperforms all the presented methods, it uses gradient of Q-value function in actor training. which provides better generalization. This provides better results on evaluation but affeets stability and performance. Against other methods that update the policy function directly, HDCR shows better results on environments out (Figure and Table
DISCUSSION
We theoretically proved that Hellinger distance improves the lower bound of expected discounted return compared to Kullback-Leible divergence and proposed a simple off-policy reinforcement learning method that uses Hellinger distance as constraint. The expected discounted return for new policy now has a policy-independee lower bound. This bound guarantees that return will not decrease "one" shot.
Experiments show that HDCR outperforms both ABM and AWR on tiny datasets obtained by ran- dom agents. This performance proves the efficiency of using Hellinger distance by allowing bigger step sizes, retaining lower bound.
On bigger datasets, HDCR shows comparable or better results than AWR and ABM.
REFERENCES
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration, 2019.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In
Proceedings of the Nineteenth International Conference on Machine Learning, ICML '02, pp- 267-274, San Francisco, CA, USA, 2002. Morgan Kaufmann Publishers Inc. ISBN 1558608737.
David Levin and Yuval Peres. Markov Chains and Mixing Times. 2017. doi: 10.1090/mbl//07.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning, 2019.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. volume 37 of Proceedings of Machine Learning Research, pp. 1889-1897,

---

Under review as a conference paper at ICLR 2021
Lille, France, 07-09 Jul 2015. PMLR. URL http://ppooeeinng.......ss////// schulman15.htm html
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Picter Abbeel. High- dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Representations (ICLR), 2016.
Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Ne- unert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. In International Confer- ence on Leurning Representations, 2020. URL wke7geHtth!]
net fort/fformm?cc
E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012.
Alexandre B. Tsybakov. Introduction to Nonparmetrrcc Estimation. Springer New York, 2009. doi: 10.1007/b13794. URL 00t.///p..///d0..or///0000111113

---

Under review as a conference paper at ICLR 2021
A THEOREM 1 PROOF
Let (9, A) a measurable space, P and Q two probability measures on that space, e be o-finite measure on (S2, A) such that P < t (P(A) 0 for any A A such that (( = 0) and
Q < v. And let us denote Radon-Nikodym derivatives (in our derivations we can assume them as probability density functions) follows:
dP
dv dQ q dv
(19)
So, p and 9 satisfy following conditions:
P(A) p(u) do AAEΛ
(20)
Q(A) / g(v) de AAEA
Then we define distances between probability measures:
Dry(P11 = sup PP(A) - Q(A)) AEA
H(P 11) = 2 (Vp(v) v((1)) do===1 10-1-1 Vi(u)g(u) de
(21)
DKL(P11 ff log dQ
flog
dP dP
/ p(u) log p(v) du
g(o)
Following Lemmas will be used in Theorem proof:
Lemma 1. Given probability distributions and total variance divergence can be calculated as follows:
1 / (p(e) q(v)] de sup P(A) Q(A)) 2 AEA
Drv
(22)
Proof.
Let B = (0: P(u) ((:)
B
((+)) and A E A:
P(A) [(AB)))((()]](
- Q(B)
P(A) [(M)-Q(4)5 (
P(B))
sup AEA
IP(A) Q[(A]= [[A)]=P(BB)) =(B)-
(23)
Drv (P Q) = _ IP(B) QQ - QQB) P(B))
/ (p(v) q(v) de
Lemma 2.
1 min(p(()) a(1)) do
Drv Drv(P11)) = -
(24)
Proof.
Drv (P 11 = / (p(u) q(v)) du = f(- (())))))))))) (p(v) - q(1}] du 2
=l
=
p(v) du - (or q(v) du
(25)
(es p(u)<c(()) 1 / min(p(v), ((()) du
8

---

Under review as a conference paper at ICLR 2021
Lemma 3.
/ max((((),,(()) do + min(p(v), ((v)) du
(26)
Proof. Rewriting left part in 4 integrals over following sets (max(p(v), q(u)) p(v)) (max(p(v)) (())) = q(v),, (min(p(v), q(v)) = {(0)} and ((i((p()).(()) = q(v)} and stacking integrals back gives P(l)) + Q(2) 2.
m
Theorem 1. For any two probability density functions and the following double inequality is true:
Drv(p ll q)² v (H 11 g)² & Dkz(pll
(27)
Proof. First inequality can be proved follows:
/ Vp(u)g(u) du 1
(1 H( (--(PI1Q)33) 11))}
/ /min(p((), max((())) d(v)) du
1²
min(p(u).g(v)) du max(p(v), de
(28)
min(p(v).g(v) du 2 2- min(((()) ( du
(1 Drv (P Q1) + (rv(
1 - Drv (P (P 1)
DP || Q] < |[ ())(22 H(P (1))
Drv(P (P) || [12 (H Q)²
Proof of the second inequality:
DkI(PI1 = / p(u) log ((0)
p(v)
du =2 / p(v) log
p(v) q(v)
de
([v g(())
-2 / p(v) log
+ du
p(v)
(29)
((v)
du 2 / [Vllep(o) - du
2 -2 p(o)
[V
p(u)
/ Vp(u)g(o) du 2H( (1
DKI(P (P 2
B HDCR DERIVATION
Given optimization problem:
arg Pm(s) (ss a) da ds
s.t. / Pu(() ( ds Se,
(30)
/ T(|ss) da = Vs

---

Under review as a conference paper at ICLR 2021
Constructing Lagrangian where S - R is function for obtaining Lagrange multiplier for every state, 3 also a Lagrange multiplier:
C(T, = --- Pul() /. x(als)AA(ss,) a) da ds
Pp(s) 1- |-JVr(absuuaal)
(i(als)uuaas) da ds
(31)
+ J as 1 / r(aas) du ds
Diffcrentiating with respect to (a]s) gives us following result:
OC
p(als)
ea(a|s) Py (p(s)) + (s)
0
(32)
2 2((((a))((aas)))
Solving for <alss):
p(als)
Bpu(s)
Pp(8)
2 [{(as))((ass) 02
(33)
Vp(als)
Pr
as
VT((as)
B
Substituting <[ass) p(-ls) and taking expectation over actions taken according to l gives us expression for ory:
are
a)
= 2 Pa(e)
B
(34)
2 Au(ssaa = 3ppp(s)
Be)p((1)) = 00 Ba)()))
2
Then optimal policy x (a|s) can be written follows:
82
n" (als) p(als) (3 (A)
(35)
To obtain regression problem we construct Kullback-Leibler divergence between optimal policy " and current policy n:
arg min Eerp,.)) E (xI( (+((s || T(|ss)
arg min E Eappu)) / n (als) log T² (als) (a|s))
da
arg min E Bampa() / u(als))
g2
M(als) (3-24,.s.a)) da
log
(3 18-2A,10 a2)
(a|s))
8
arg min E Berpu() Bavu(-)e)
log (M(als)
(B 2A, (s, log m (als)
(B (A, a))
arg max EaaxE Eawn(-ls) log (als)
(3 - 2Ay(s,a))*
arg max (axpEe EL Baneet)s) log (3 A,(8, a))
(36)
Regression problem follows:
arg max Esmp,() Eas log ((ass) (B Ap(s,
(37)
10


Reviews:
[{"review_id": "j89VeJR7xh9", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\n \nThe paper provides a new metric - Hellinger distance to be combined with trust region ideas in policy optimization. The major difference from prior work is the change of this distance metric. The paper shows that with this distance metric, along with Lagrangian relaxation, one could show analytic results of improved policies. The paper also shows similar lower bound improvement results and compared with baselines on offline rl tasks.\n\n##########################################################################\n\nReasons for score: \n\n \nOverall, I vote for rejection. I think the idea of changing the distance metric is not novel enough. Critically, I do not think so far in the paper there is a strong enough motivation to use this distance metric: both innovation-wise and result-wise. I will explain in details below.\n \n##########################################################################Pros: \n\n \n1. Idea is not novel: the overall idea of using an alternative metric does not seem novel. Though the authors motivated an 'improved' version of the trust region lower bound, by using the fact that the Hellinger distance is upper bounded by KL - I think such an improvement in the lower bound is a bit trivial and does not provide new perspectives on the old results.\n \n2. This new lower bound also might not provide additional benefits in practice - because in practice such lower bounds are generally too conservative.\n\n3. Experiment results are also not strong enough. I will explain below.\n \n##########################################################################\n\nCons: \n\n \n1. The final performance of all three baseline algorithms are fairly bad in terms of final rewards (e.g. for halfcheetah, all returns are negative, yet we know that online algorithms could achieve >3000 at least and in some cases >6000). I wonder if this general inferior performance is a result of using offline dataset - in that sense, does the agent learn anything meaningful at all?\n\n2. From both fig 1 and fig 2, about for half of the tasks the performance seem to drop (or stay at the same level) as the case where no training is done (x-axis at the origin). Does this also corroborate my previous concern that these agents do not learn much at all?\n \n3. From the curves presented in Fig1,2, as well as mean+std results in Table 1,2, it does not seem that the new method provides much significant gains either.\n \n##########################################################################\n\nQuestions during rebuttal period: \n\nPlease address and clarify the cons above. Thanks.\n\n \n#########################################################################", "extra_scores": {}, "extra_text": {}}, {"review_id": "hV0TNUsII3F", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an algorithm for off-policy reinforcement learning using the Hellinger distance between the sampling policy and optimized policy as a constraint. The motivation for the proposed method is explained in the preliminaries section. The actual algorithm and experiments run using the proposed algorithm are also provided. \n\nThe derivation is easy to follow, and this is because of the well-known lower and upper bounds on the Hellinger distance. \n\nThe writing of the paper needs work. For example, the abstract talks about the sampling policy and current policy. By current policy, what the authors mean is the policy that is being optimized. The sampling policy is the policy that was run offline. Clarifying these terms would help. Similarly, I did not follow \"return for the new policy is improved comparing to KL\". \n\nIn paragraph 3: \"With the use of Lagrangian, have been derived\" needs proofreading. In eqn 13, what is beta?\n\nIn the figures, what are the axes?", "extra_scores": {}, "extra_text": {}}, {"review_id": "VBIr6lYuNZO", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose the use of the Hellinger distance instead of KL divergence to constrain new policies to remain close to the behavior policy. The technical aspects are straightforward, noting that Hellinger provides tighter bounds on total variation than KL, and can straightforwardly be plugged into the CPI/TRPO bounds for policy improvement. They also propose an offline reinforcement learning algorithm based on enforcing a Hellinger constraint to the data policy, deriving iterative optimization procedure, and evaluate it on offline\n\nI find the experimental evaluation highly lacking. It seems with the datasets and envs evaluated, policy performance actually *drops* as policy optimization is conducted, so it is not clear to me that these evaluations actually provide meaningful information towards which methods perform better in scenarious where we would want to use offline RL. I would like to see much more extensive evaluation of this method compared to other offline RL algorithms like BCQ https://arxiv.org/abs/1812.02900, BRAC https://arxiv.org/abs/1911.11361, or CQL https://arxiv.org/abs/2006.04779, over a much wider variety of datasets. \n\nIn general, I'm not convinced that simply using the Hellinger distance instead of KL will lead to significant improvements on its own, given that in the BRAC paper, the authors experimented with different trust regions including Wasserstein, MMD, and KL and didn't find huge differences in the tested domains. Overall, the contribution does not seem significant enough to warrant publication without strong experimental results, which this paper lacks.", "extra_scores": {}, "extra_text": {}}, {"review_id": "30wqdURjFu7", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \n \nThis paper proposes a supervised learning for off-policy reinforcement learning. It exploits the Hellinger distance instead of KL divergence. Thus it achieves tighter lower bound of the expected culmulative return than that using the KL divergence. Moreover, the new lower bound is policy independent. The experimental results show that the proposed method slightly outperforms other baselines when only small amount of data are given, while the algorithms fail to learn on several environments.\n \n\nReasons for score:\n \nThough it has some advantages, I vote to reject this paper. This is because, it has low novelty, the experiments are wrongly designed, and thus it is hard to believe the results. The specific details are below.\n \n\nPros\n \n+ Hellinger divergence is used instead of KL divergence, and thus the lower bound become tighter than that using KL divergence.\n \n+ The loss function for policy can be derived by theory\n \n\nCons\n \n- Changing KL distance to Hellinger divergence has low novelty.  Also, the derivation of the loss function using Hellinger distance isn't difficult.  Hellinger distance and KL divergence are all under the class of Amari alpha-divergence. When alpha = +/- 1, Amari alpha-divergence becomes KL and when alpha=0, Amari alpha-divergence becomes the Hellinger distance = integral [sqrt(p) - sqrt(q)]^2 dx.   Indeed, HD is symmetric and satisfies the axioms of distance. Basically, when we consider the HD on the space of probability distribution, we consider Euclidean geometry on the space of probability distribution, whereas the KLD induces the Boltzman interpretation, i.e., p ~ exp( -KLD).\n\n- In addition to the issue of significance in novelty,  the numerical results show that the performance improvement is insignificant or negligible. .\n \n- The experiments used data sampled by random policies or first few samples of on-policy data, but I think that this is a little strange training setting. Most of the previous works in this line use samples at a certain performance (NOT DRAWN BY RANDOM POLICY). For example, in ABM paper[1], it used first 10,000 episodes (if the length of an episode is 1,000, it uses first 10 million samples), or first 2,000 episodes (first 2 million samples) to show its performance when it uses high performed samples, or low performed samples, respectively. These contain good performed samples relative to the random samples. However, experiments in this paper use almost random samples to train policies. We cannot expect a good policy at a certain performance using these random samples. This expectation is also shown in the results. Some learning curves go down as learning proceeds, and this  means that the learning fails on these environments. If the proposed method learns successfully while the others fail to learn, it is a meaningful result, but it is not, otherwise. I think that the authors should evaluate performance using better samples to prove that the proposed method outperforms others.\n \n\nReference\n \n[1] Noah Siegel, et al. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. In International Conference on Learning Representations, 2020.\n \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewer concerns generally centered around the novelty of replacing the distance metric for a policy constraint. While the authors clarified many of the reviewer concerns and added some additional comparisons, in the end it was not clear why the proposed approach was interesting: while it is true that this particular distance metric has not been evaluated in prior work, and the result would have been interesting if it resulted in some clear benefits either empirically or theoretically, in the absence of clear and unambiguous benefit, it's not clear how valuable this concept really is. After discussion, the reviewers generally found the paper to not be ready for publication in its present state.
Decision: Reject