SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this paper, we introduced BARFI, a novel framework that empowers RL practitioners—who may not be experts in the field—to incorporate domain knowledge through heuristic auxiliary reward functions. Our framework allows for more expressive reward functions to be learned while ensuring they remain aligned with a designer's original intentions. BARFI can also identify reward functions that foster faster learning while mitigating various limitations and biases in underlying RL algorithms. We empirically show that BARFI is effective in training agents in sparse-reward scenarios where (possibly poorly-specified) auxiliary reward information is available. If the provided auxiliary rewards are determined to be misaligned with the designer's intended goals, BARFI autonomously adapts and effectively disincentivizes their use as needed. This adaptability results in a reliable pathway to identifying high-performing policies. The conceptual insights offered by this work provide RL practitioners with a structured way to design more robust and easy-to-optimize reward functions. We believe this will contribute to making RL more accessible to a broader audience.
Acknowledgement and Funding Disclosures
We thank Andy Barto for invaluable discussions and insightful feedback on an earlier version of this manuscript, which significantly improved the quality of our work.
This work is partially supported by the National Science Foundation under grant no. CCF-2018372 and by a gift from the Berkeley Existential Risk Initiative.
[1] Joshua Achiam, Ethan Knight, and Pieter Abbeel. Towards characterizing divergence in deep q-learning. arXiv preprint arXiv:1903.08894, 2019.
[2] Akshay Agrawal, Brandon Amos, Shane Barratt, Stephen Boyd, Steven Diamond, and Zico Kolter. Differentiable convex optimization layers. arXiv preprint arXiv:1910.12430, 2019.
[3] Ron Amit, Ron Meir, and Kamil Ciosek. Discount factor as a regularizer in reinforcement learning. In International Conference on Machine Learning, 2020.
[4] Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning, 2017.
[5] Stuart Armstrong, Jan Leike, Laurent Orseau, and Shane Legg. Pitfalls of learning a reward function online. arXiv preprint arXiv:2004.13654, 2020.
[6] Karl J Åström and Tore Hägglund. Pid control. IEEE Control Systems Magazine, 2006.
[7] Sarah Bechtle, Artem Molchanov, Yevgen Chebotar, Edward Grefenstette, Ludovic Righetti, Gaurav Sukhatme, and Franziska Meier. Meta-learning via learned loss. arXiv preprint arXiv:1906.05374, 2019.
[8] Yoshua Bengio. Gradient-based optimization of hyperparameters. Neural computation, 2000.
[9] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
[10] Souradip Chakraborty, Amrit Singh Bedi, Alec Koppel, Dinesh Manocha, Huazheng Wang, Furong Huang, and Mengdi Wang. Aligning agent policy with externalities: Reward design via bilevel rl. arXiv preprint arXiv:2308.02585, 2023.
[11] Xu Chu Dennis Ding, Stephen L Smith, Calin Belta, and Daniela Rus. LTL control in uncertain environments with probabilistic satisfaction guarantees. IFAC Proceedings Volumes, 2011.
[12] Chuong B Do, Chuan-Sheng Foo, and Andrew Y Ng. Efficient multiple hyperparameter learning for log-linear models. In Advances in Neural Information Processing Systems, 2007.
[13] Justin Domke. Generic methods for optimization-based modeling. In Artificial Intelligence and Statistics, 2012.
[14] Asen L Dontchev and R Tyrrell Rockafellar. Implicit functions and solution mappings, volume 543. Springer, 2009.
[15] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In , Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, 2017.
[16] Razvan V Florian. Correct equations for the dynamics of the cart-pole system. Center for Cognitive and Neural Studies (Coneural), Romania, 2007.
[17] Zhao-Yang Fu, De-Chuan Zhan, Xin-Chun Li, and Yi-Xing Lu. Automatic successive reinforcement learning with multiple auxiliary rewards. In IJCAI, 2019.
[55] Jonathan Sorg, Satinder P Singh, and Richard L Lewis. Internal rewards mitigate agent boundedness. In Proceedings of the 27th international conference on machine learning (ICML-10), 2010.
[56] Jonathan Sorg, Satinder P Singh, and Richard L Lewis. Optimal rewards versus leaf-evaluation heuristics in planning agents. In Proceedings of the AAAI Conference on Artificial Intelligence, 2011.
[57] Jonathan Daniel Sorg. The Optimal Reward Problem: Designing Effective Reward for Bounded Agents. PhD thesis, University of Michigan, 2011.
[58] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[59] Philip Thomas. Bias in natural actor-critic algorithms. In International Conference on Machine Learning, 2014.
[60] John N Tsitsiklis and Benjamin Van Roy. An analysis of temporal-difference learning with function approximation. IEEE Transactions on Automatic Control, 1997.
[61] Marin Vlastelica, Anselm Paulus, Vít Musil, Georg Martius, and Michal Rolínek. Differentiation of blackbox combinatorial solvers. arXiv preprint arXiv:1912.02175, 2019.
[62] Yufei Wang, Qiwei Ye, and Tie-Yan Liu. Beyond exponentially discounted sum: Automatic learning of return function. arXiv preprint arXiv:1905.11591, 2019.
[63] Martha White. Unifying task specification in reinforcement learning. In International Conference on Machine Learning, 2017.
[64] Eric Wiewiora. Potential-based shaping and q-value initialization are equivalent. Journal of Artificial Intelligence Research, 2003.
[65] Eric M Wolff, Ufuk Topcu, and Richard M Murray. Robust control of uncertain markov decision processes with temporal logic specifications. In 2012 IEEE 51st IEEE Conference on Decision and Control (CDC), 2012.
[66] David H Wolpert and Kagan Tumer. Optimal reward functions in distributed reinforcement learning. In Intelligent agent technology: Research and development. World Scientific, 2001.
[67] Yuhuai Wu, Mengye Ren, Renjie Liao, and Roger Grosse. Understanding short-horizon bias in stochastic meta-optimization. arXiv preprint arXiv:1803.02021, 2018.
[68] Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-gradient reinforcement learning. In Advances in Neural Information Processing Systems, 2018.
[69] Zhongwen Xu, Hado P van Hasselt, Matteo Hessel, Junhyuk Oh, Satinder Singh, and David Silver. Meta-gradient reinforcement learning with an objective discovered online. Advances in Neural Information Processing Systems, 2020.
[70] Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. In Advances in Neural Information Processing Systems, 2018.
[71] Zeyu Zheng, Junhyuk Oh, Matteo Hessel, Zhongwen Xu, Manuel Kroiss, Hado Van Hasselt, David Silver, and Satinder Singh. What can be learned intrinsic rewards capture? In International Conference on Machine Learning, 2020.
[72] Haosheng Zou, Tongzheng Ren, Dong Yan, Hang Su, and Jun Zhu. Reward shaping via meta-learning. arXiv preprint arXiv:1901.09330, 2019.
Behavior Alignment via Reward Function Optimization (Supplemental Material)
Table 2: Notations
Symbol	Description
θ	Parameters for policy π
φ	Parameters for reward function
φ	Parameters for learned γ
πθ, rφ, γφ	Functional form of policy, reward and γ with their respective parameters
αθ, αφ, αφ	Step sizes for the respective parameters
λθ, λφ, λφ	Regularization for policy, reward and γ function
δ	Number of on-policy samples collected between subsequent updates to φ, φ
η	Neumann Approximator Eigen value scaling factor
n	Number of loops used in Neumann Approximation
optim	Any standard optimizer like Adam, RMSprop, SGD, which takes input as gradients and outputs the appropriate update
E	Total Number of episodes to sample from the environment
Ni	Number of updates to be performed for updating the π by Alg
N0	Number of initial updates to be peformed
τ	Sample of a trajectory from a full episode
A Proofs for Theoretical Results
In this section, we provide proofs for Property 1, Property 2, and Property 3. For the purpose of these proofs, we introduce some additional notation. To have a unified MDP notation for goal-based and time-based tasks, we first consider that in the time-based task, time is a part of the state such that Markovian dynamics is ensured.
The (un-normalized) discounted and (un-normalized) undiscounted visitation probability is denoted as
$$
d _ {\gamma} ^ {\pi} (s, a) := \sum_ {t = 0} ^ {T} \gamma^ {t} \Pr \left(S _ {t} = s, A _ {t} = a; \pi\right), \tag {9}
$$
$$
\bar {d} ^ {\pi} (s, a) := \sum_ {t = 0} ^ {T} \Pr \left(S _ {t} = s, A _ {t} = a; \pi\right). \tag {10}
$$
We can normalize it so that it is a distribution as follows :
$$
d ^ {\pi} (s, a) := \frac {\bar {d} ^ {\pi} (s , a)}{\sum_ {s ^ {\prime} \in \mathcal {S} , a ^ {\prime} \in \mathcal {A}} \bar {d} ^ {\pi} \left(s ^ {\prime} , a ^ {\prime}\right)}.
$$
Property 1. The expected update performed by the biased policy gradient update is same when using the primary reward and reward modified with potential-based shaping, i.e.,  $\Delta (\theta ,\tilde{r}) = \Delta (\theta ,r_p)$ . Further, the variance of the update when using potential-based reward shaping can be higher than the variance of the update performed using the primary reward, i.e.,  $\operatorname {Var}\left(\hat{\Delta} (\theta ,\tilde{r})\right)\geq \operatorname {Var}\left(\hat{\Delta} (\theta ,r_p)\right)$
Proof. Part 1: Equality of the expected update
$$
\begin{array}{l} \Delta (\theta , \tilde {r}) = \mathbb {E} _ {\pi_ {\theta}} \left[ \sum_ {t = 0} ^ {T} \psi_ {\theta} (S _ {t}, A _ {t}) \sum_ {j = t} ^ {T} \gamma^ {j - t} \tilde {r} (S _ {j}, A _ {j}) \right] \\ = \mathbb {E} _ {\pi_ {\theta}} \left[ \sum_ {t = 0} ^ {T} \psi_ {\theta} \left(S _ {t}, A _ {t}\right) \left(\sum_ {j = t} ^ {T} \gamma^ {j - t} \left(r _ {p} \left(S _ {j}, A _ {j}\right) + \gamma \Phi \left(S _ {j + 1}\right) - \Phi \left(S _ {j}\right)\right)\right) \right] \\ = \mathbb {E} _ {\pi_ {\theta}} \left[ \sum_ {t = 0} ^ {T} \psi_ {\theta} (S _ {t}, A _ {t}) \sum_ {j = t} ^ {T} \gamma^ {j - t} r _ {p} (S _ {j}, A _ {j}) \right] + \mathbb {E} _ {\pi_ {\theta}} \left[ \sum_ {t = 0} ^ {T} \psi_ {\theta} (S _ {t}, A _ {t}) \sum_ {j = t} ^ {T} \gamma^ {j - t} (\gamma \Phi (S _ {j + 1}) - \Phi (S _ {j})) \right] \\ = \Delta (\theta , r _ {p}) + \mathbb {E} _ {\pi_ {\theta}} \left[ \sum_ {t = 0} ^ {T} \psi_ {\theta} \left(S _ {t}, A _ {t}\right) \sum_ {j = t} ^ {T} \gamma^ {j - t} \left(\gamma \Phi \left(S _ {j + 1}\right) - \Phi \left(S _ {j}\right)\right) \right] \\ \stackrel {(a)} {=} \Delta (\theta , r _ {p}) + \mathbb {E} _ {\pi_ {\theta}} \left[ \sum_ {t = 0} ^ {T} \psi_ {\theta} (S _ {t}, A _ {t}) \left(\gamma^ {T - t + 1} \Phi \left(S _ {T + 1}\right) - \Phi (S _ {t})\right) \right] \\ \stackrel {(b)} {=} \Delta (\theta , r _ {p}) + \mathbb {E} _ {\pi_ {\theta}} \left[ \sum_ {t = 0} ^ {T} \psi_ {\theta} (S _ {t}, A _ {t}) \left(\gamma^ {T - t + 1} c - \Phi (S _ {t})\right) \right] \\ \stackrel {(c)} {=} \Delta (\theta , r _ {p}) + \mathbb {E} _ {\pi_ {\theta}} \left[ \sum_ {t = 0} ^ {T} \left(\gamma^ {T - t + 1} c - \Phi (S _ {t})\right) \mathbb {E} _ {\pi_ {\theta}} [ \psi_ {\theta} (S _ {t}, A _ {t}) | S _ {t} ] \right] \\ \stackrel {(d)} {=} \Delta (\theta , r _ {p}), \\ \end{array}
$$
where (a) holds because on the expansion of future return, intermediate potential values cancel out, (b) holds because  $S_{T + 1}$  is the terminal state and potential function is defined to be a fixed constant  $c$  for any terminal state [41], (c) holds from the law of total expectation, and (d) holds because,
$$
\mathbb {E} _ {\pi_ {\theta}} \left[ \psi_ {\theta} (S _ {t}, A _ {t}) | S _ {t} \right] = \sum_ {a \in \mathcal {A}} \pi_ {\theta} (S _ {t}, a) \frac {\partial \ln \pi_ {\theta} (S _ {t} , a)}{\partial \theta} = \sum_ {a \in \mathcal {A}} \frac {\partial \pi_ {\theta} (S _ {t} , a)}{\partial \theta} = \frac {\partial}{\partial \theta} \sum_ {a \in \mathcal {A}} \pi_ {\theta} (S _ {t}, a) = 0.
$$
In the stochastic setting, i.e., when using sample average estimates instead of the true expectation,  $\gamma^{T - t + 1}c - \phi (S_t)$  is analogous to a state-dependent baseline for the sum of discounted future primary rewards. It may reduce or increase the variance of  $\Delta (\theta ,r_p)$ , depending on this baseline's co-variance with  $\sum_{j = t}^{T}\gamma^{j - t}r_{p}(S_{j},A_{j})$ .
Note: As we encountered the potential at the terminal state to be  $c$ , as it is a constant, we will use the value of  $c = 0$  in accordance with [41].
Part 2: Variance characterization
For this result that discusses the possibility of the variance being higher when using potential-based reward shaping, we demonstrate the result using a simple example. We will consider the single-step case wherein an episode lasts for one time step. That is, the agent takes an action  $A_0$  at the starting state  $S_0$  and then transitions to the terminal state. Hence, the stochastic update  $\hat{\Delta}(\theta, \tilde{r})$  can be written as:
$$
\begin{array}{l} \hat {\Delta} (\theta , \tilde {r}) = \psi_ {\theta} (S _ {0}, A _ {0}) \tilde {r} (S _ {0}, A _ {0}) \\ = \psi_ {\theta} (S _ {0}, A _ {0}) \left(r _ {p} (S _ {0}, A _ {0}) - \Phi (S _ {0})\right), \\ \end{array}
$$
wherein, we assume that  $\Phi$  for terminal states is 0 and similarly,  $\hat{\Delta} (\theta ,r_p) = \psi_\theta (S_0,A_0)r_p(S_0,A_0)$ .
For the purpose of this proof we will consider the case wherein we have a scalar  $\theta$ , i.e.,  $\theta \in \mathbb{R}$ , such that,  $\psi_{\theta}(.,.) \in \mathbb{R}$ .
Hence,  $\operatorname{Var}\left(\hat{\Delta}(\theta, \tilde{r})\right)$  can be written as:
$$
\begin{array}{l} \operatorname {V a r} \left(\hat {\Delta} (\theta , \tilde {r})\right) = \mathbb {E} \left[ \hat {\Delta} (\theta , \tilde {r}) ^ {2} \right] - \mathbb {E} \left[ \hat {\Delta} (\theta , \tilde {r}) \right] ^ {2} \\ = \mathbb {E} \left[ \left(\psi_ {\theta} (S _ {0}, A _ {0}) \left(r _ {p} (S _ {0}, A _ {0}) - \Phi (S _ {0})\right)\right) ^ {2} \right] - \mathbb {E} \left[ \psi_ {\theta} (S _ {0}, A _ {0}) \left(r _ {p} (S _ {0}, A _ {0}) - \Phi (S _ {0})\right) \right] ^ {2} \\ \stackrel {(a)} {=} \mathbb {E} \left[ \psi_ {\theta} \left(S _ {0}, A _ {0}\right) ^ {2} \left(r _ {p} \left(S _ {0}, A _ {0}\right) ^ {2} + \Phi \left(S _ {0}\right) ^ {2} - 2 \Phi \left(S _ {0}\right) r _ {p} \left(S _ {0}, A _ {0}\right)\right) \right] - \mathbb {E} \left[ \psi_ {\theta} \left(S _ {0}, A _ {0}\right) \left(r _ {p} \left(S _ {0}, A _ {0}\right)\right) \right] ^ {2} \\ = \mathbb {E} \left[ \psi_ {\theta} \left(S _ {0}, A _ {0}\right) ^ {2} \left(\Phi \left(S _ {0}\right) ^ {2} - 2 \Phi (S _ {0}) r _ {p} \left(S _ {0}, A _ {0}\right)\right) \right] + \\ \underbrace {\mathbb {E} \big [ \psi_ {\theta} (S _ {0} , A _ {0}) ^ {2} r _ {p} (S _ {0} , A _ {0}) ^ {2} \big ] - \mathbb {E} [ \psi_ {\theta} (S _ {0} , A _ {0}) (r _ {p} (S _ {0} , A _ {0})) ] ^ {2}} _ {\operatorname {V a r} (\hat {\Delta} (\theta , r _ {p}))}. \\ \end{array}
$$
$$
\left. \right. \operatorname {V a r} \left(\hat {\Delta} (\theta , \tilde {r})\right) - \operatorname {V a r} \left(\hat {\Delta} (\theta , r _ {p})\right) = \mathbb {E} \left[ \psi_ {\theta} \left(S _ {0}, A _ {0}\right) ^ {2} \left(\Phi \left(S _ {0}\right) ^ {2} - 2 \Phi \left(S _ {0}\right) r _ {p} \left(S _ {0}, A _ {0}\right)\right)\right].
$$
Subsequently, variance of  $\hat{\Delta} (\theta ,\tilde{r})$  will be higher than that of  $\hat{\Delta} (\theta ,r_p)$  if  $\mathbb{E}\big[\psi_{\theta}(S_0,A_0)^2\Phi (S_0)^2\big] - 2\mathbb{E}\big[\psi_{\theta}(S_0,A_0)^2\Phi (S_0)r_p(S_0,A_0)\big] > 0.$
Example: Let us look at an example where the above condition can be true. Let us consider an MDP with a single state and a single-step horizon. In that case, we can consider the variance of the update to the policy at the said state, i.e.,
$$
\operatorname {V a r} _ {\pi} \Big (\hat {\Delta} (\theta , \tilde {r}) \Big) - \operatorname {V a r} _ {\pi} \Big (\hat {\Delta} (\theta , r _ {p}) \Big) = \Phi (s) ^ {2} \mathbb {E} _ {\pi} \big [ \psi_ {\theta} (s, A) ^ {2} \big ] - 2 \Phi (s) \mathbb {E} _ {\pi} \big [ \psi_ {\theta} (s, A) ^ {2} (r _ {p} (s, A)) \big ],
$$
where  $s$  is the fixed state. Hence, the variance of the potential-based method might be more than the variance from using only the primary reward when
$$
\begin{array}{l} \Phi (s) ^ {2} \mathbb {E} _ {\pi} \left[ \psi_ {\theta} (s, A) ^ {2} \right] - 2 \Phi (s) \mathbb {E} _ {\pi} \left[ \psi_ {\theta} (s, A) ^ {2} \left(r _ {p} (s, A)\right) \right] > 0 \\ \Phi (s) ^ {2} \mathbb {E} _ {\pi} \left[ \psi_ {\theta} (s, A) ^ {2} \right] > 2 \Phi (s) \mathbb {E} _ {\pi} \left[ \psi_ {\theta} (s, A) ^ {2} \left(r _ {p} (s, A)\right) \right]. \\ \end{array}
$$
Further, let us consider the case where  $\Phi(s) \neq 0$ , because otherwise the variance of the update for those states would be same, and  $\Phi(s) > 0$ .
$$
\begin{array}{l} \Phi (s) ^ {2} \mathbb {E} _ {\pi} \left[ \psi_ {\theta} (s, A) ^ {2} \right] > 2 \Phi (s) \mathbb {E} _ {\pi} \left[ \psi_ {\theta} (s, A) ^ {2} \left(r _ {p} (s, A)\right) \right] \\ \Phi (s) \mathbb {E} _ {\pi} \left[ \psi_ {\theta} (s, A) ^ {2} \right] > 2 \mathbb {E} _ {\pi} \left[ \psi_ {\theta} (s, A) ^ {2} \left(r _ {p} (s, A)\right) \right]. \\ \end{array}
$$
We can see that the above condition can be satisfied by choosing a potential function that might be overly optimistic about the average reward of the state  $s$ , i.e. any  $\Phi(s)$ , s.t.  $\Phi(s) > 2r_p(s, a)\forall a$  would lead to an increase in variance. A common place where this might be true is the use of an optimal value function (as hinted by [41]) as a baseline for a bad/mediocre policy initially.
Property 2. There exists  $r_{\phi}:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$  and  $\gamma_{\varphi}\in [0,1)$  such that  $\Delta_{on}(\theta ,\phi ,\varphi) = \Delta_{\gamma}(\theta ,r_p)$
Proof. Recall the definition of  $\Delta_{\gamma}(\theta, r_p)$  from Section 4.1:
$$
\Delta_ {\gamma} (\theta , r _ {p}) = \mathbb {E} _ {\pi_ {\theta}} \left[ \sum_ {t = 0} ^ {T} \gamma^ {t} \psi_ {\theta} (S _ {t}, A _ {t}) \sum_ {j = t} ^ {T} \gamma^ {j - t} r _ {p} (S _ {j}, A _ {j}) \right].
$$
Using the law of total expectation,
$$
\begin{array}{l} \Delta_ {\gamma} (\theta , r _ {p}) = \mathbb {E} _ {\pi_ {\theta}} \left[ \sum_ {t = 0} ^ {T} \gamma^ {t} \psi_ {\theta} (S _ {t}, A _ {t}) \mathbb {E} _ {\pi_ {\theta}} \left[ \sum_ {j = t} ^ {T} \gamma^ {j - t} r _ {p} (S _ {j}, A _ {j}) \Bigg | S _ {t}, A _ {t} \right] \right] \\ = \mathbb {E} _ {\pi_ {\theta}} \left[ \sum_ {t = 0} ^ {T} \gamma^ {t} \psi_ {\theta} \left(S _ {t}, A _ {t}\right) q ^ {\pi_ {\theta}} \left(S _ {t}, A _ {t}\right) \right] \\ = \sum_ {s \in \mathcal {S}, a \in \mathcal {A}} \sum_ {t = 0} ^ {T} \gamma^ {t} \Pr (S _ {t} = s, A _ {t} = a; \pi_ {\theta}) \psi_ {\theta} (s, a) q ^ {\pi_ {\theta}} (s, a) \\ = \sum_ {s \in S, a \in A} \psi_ {\theta} (s, a) q ^ {\pi_ {\theta}} (s, a) \sum_ {t = 0} ^ {T} \gamma^ {t} \Pr \left(S _ {t} = s, A _ {t} = a; \pi_ {\theta}\right) \\ = \sum_ {s \in S, a \in A} \psi_ {\theta} (s, a) q ^ {\pi_ {\theta}} (s, a) d _ {\gamma} ^ {\pi_ {\theta}} (s, a). \tag {11} \\ \end{array}
$$
Notice from (9) and (10) that for any  $(s,a)$  pair, if  $d_{\gamma}^{\pi_{\theta}}(s,a) > 0$ , then  $\bar{d}^{\pi_{\theta}}(s,a) > 0$  since  $\gamma \geq 0$ . Therefore, dividing and multiplying by  $\bar{d}^{\pi_{\theta}}(s,a)$  leads to:
$$
\begin{array}{l} \Delta_ {\gamma} (\theta , r _ {p}) = \sum_ {s \in \mathcal {S}, a \in \mathcal {A}} \bar {d} ^ {\pi_ {\theta}} (s, a) \psi_ {\theta} (s, a) q ^ {\pi_ {\theta}} (s, a) \frac {d _ {\gamma} ^ {\pi_ {\theta}} (s , a)}{\bar {d} ^ {\pi_ {\theta}} (s , a)} \\ = \sum_ {s \in S, a \in \mathcal {A}} \sum_ {t = 0} ^ {T} \Pr (S _ {t} = s, A _ {t} = a; \pi_ {\theta}) \psi_ {\theta} (s, a) q ^ {\pi_ {\theta}} (s, a) \frac {d _ {\gamma} ^ {\pi_ {\theta}} (s , a)}{\bar {d} ^ {\pi_ {\theta}} (s , a)} \\ = \mathbb {E} _ {\pi_ {\theta}} \left[ \sum_ {t = 0} ^ {T} \psi_ {\theta} \left(S _ {t}, A _ {t}\right) q ^ {\pi_ {\theta}} \left(S _ {t}, A _ {t}\right) \frac {d _ {\gamma} ^ {\pi_ {\theta}} \left(S _ {t} , A _ {t}\right)}{\bar {d} ^ {\pi_ {\theta}} \left(S _ {t} , A _ {t}\right)} \right]. \\ \end{array}
$$
Now, notice that if  $\gamma_{\varphi} = 0$  and  $r_{\phi}(s,a) = q^{\pi_{\theta}}(s,a)\frac{d_{\gamma}^{\pi_{\theta}}(s,a)}{d^{\pi_{\theta}}(s,a)}$ , for all  $s\in S$  and  $a\in \mathcal{A}$ , then
$$
\Delta_ {\mathrm {o n}} (\theta , \phi , \varphi) = \Delta_ {\gamma} (\theta , r _ {p}).
$$
Property 3. There exists  $r_{\phi}:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$  and  $\gamma_{\varphi}\in [0,1)$  such that  $\Delta_{off}(\theta ,\phi ,\varphi) = \Delta_{off}(\theta ,r_p)$
Proof. This proof follows a similar technique as the proof for Property 2. Recall the definition of  $\Delta_{\mathrm{off}}(\theta, r_p)$ :
$$
\begin{array}{l} \Delta_ {\mathrm {o f f}} (\theta , r _ {p}) := \mathbb {E} _ {\beta} \left[ \sum_ {t = 0} ^ {T} \gamma^ {t} \psi_ {\theta} (S _ {t}, A _ {t}) \sum_ {j = t} ^ {T} \gamma^ {j - t} \rho_ {j} r _ {p} (S _ {j}, A _ {j}) \right] \\ := \mathbb {E} _ {\beta} \left[ \sum_ {t = 0} ^ {T} \gamma^ {t} \rho_ {t} \psi_ {\theta} (S _ {t}, A _ {t}) \sum_ {j = t} ^ {T} \gamma^ {j - t} \rho_ {j - t} r _ {p} (S _ {j}, A _ {j}) \right]. \\ \end{array}
$$
Now using the law of total expectations,
$$
\begin{array}{l} \Delta_ {\text {o f f}} (\theta , r _ {p}) = \mathbb {E} _ {\beta} \left[ \sum_ {t = 0} ^ {T} \gamma^ {t} \rho_ {t} \psi_ {\theta} (S _ {t}, A _ {t}) \mathbb {E} _ {\beta} \left[ \sum_ {j = t} ^ {T} \gamma^ {j - t} \rho_ {j - t} r _ {p} (S _ {j}, A _ {j}) \Bigg | S _ {t}, A _ {t} \right] \right] \\ = \mathbb {E} _ {\pi_ {\theta}} \left[ \sum_ {t = 0} ^ {T} \gamma^ {t} \psi_ {\theta} \left(S _ {t}, A _ {t}\right) \mathbb {E} _ {\pi_ {\theta}} \left[ \sum_ {j = t} ^ {T} \gamma^ {j - t} r _ {p} \left(S _ {j}, A _ {j}\right) \mid S _ {t}, A _ {t} \right] \right] \\ = \mathbb {E} _ {\pi_ {\theta}} \left[ \sum_ {t = 0} ^ {T} \gamma^ {t} \psi_ {\theta} (S _ {t}, A _ {t}) q ^ {\pi_ {\theta}} (S _ {j}, A _ {j}) \right] \\ = \sum_ {s \in \mathcal {S}, a \in \mathcal {A}} \psi_ {\theta} (s, a) q ^ {\pi_ {\theta}} (s, a) d _ {\gamma} ^ {\pi_ {\theta}} (s, a), \\ \end{array}
$$
where the last line follows similar to (11). Now, notice that for any  $(s,a)$  pair, the assumption that  $\pi_{\theta}(s,a) / \beta (s,a) < \infty$  for all  $s\in S,a\in \mathcal{A}$ , implies  $d_{\gamma}^{\pi_{\theta}}(s,a) / d_{\gamma}^{\beta}(s,a) < \infty$ . Further, if  $d_{\gamma}^{\beta}(s,a) > 0$  it has to be that  $d^{\beta}(s,a) > 0$  as well. Therefore,  $d_{\gamma}^{\pi_{\theta}}(s,a) / d^{\beta}(s,a) < \infty$  as well. Multiplying and dividing by  $d^{\beta}(s,a)$  results in:
$$
\begin{array}{l} \Delta_ {\mathrm {o f f}} (\theta , r _ {p}) = \sum_ {s \in \mathcal {S}, a \in \mathcal {A}} \bar {d} ^ {\beta} (s, a) \psi_ {\theta} (s, a) q ^ {\pi_ {\theta}} (s, a) \frac {d _ {\gamma} ^ {\pi_ {\theta}} (s , a)}{\bar {d} ^ {\beta} (s , a)} \\ = \sum_ {s \in \mathcal {S}, a \in \mathcal {A}} \sum_ {t = 0} ^ {T} \Pr (S _ {t} = s, A _ {t} = a; \beta) \psi_ {\theta} (s, a) q ^ {\pi_ {\theta}} (s, a) \frac {d _ {\gamma} ^ {\pi_ {\theta}} (s , a)}{\bar {d} ^ {\beta} (s , a)} \\ = \mathbb {E} _ {\beta} \left[ \sum_ {t = 0} ^ {T} \psi_ {\theta} \left(S _ {t}, A _ {t}\right) q ^ {\pi_ {\theta}} \left(S _ {t}, A _ {t}\right) \frac {d _ {\gamma} ^ {\pi_ {\theta}} \left(S _ {t} , A _ {t}\right)}{\bar {d} ^ {\beta} \left(S _ {t} , A _ {t}\right)} \right]. \\ \end{array}
$$
Finally, notice that if  $\gamma_{\varphi} = 0$  and  $r_{\phi}(s,a) = q^{\pi_{\theta}}(s,a)\frac{d_{\gamma}^{\pi_{\theta}}(s,a)}{d^{\beta}(s,a)}$  for all  $s\in S$  and  $a\in A$
$$
\Delta_ {\mathrm {o f f}} (\theta , \phi , \varphi) = \Delta_ {\mathrm {o f f}} (\theta , r _ {p}).
$$