SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we propose a perturbed quantile regression (PQR) that is a practical algorithm for distributional reinforcement learning. Our quantile model is updated by minimizing the objective function (1) induced by PDBOO. Since we employ a quantile model, sampling a reweight function  $\xi$  can be reduced into sampling an  $N$ -dimensional weight vector  $\pmb{\xi} \coloneqq [\xi_1, \dots, \xi_N]$  where  $\sum_{i=1}^{N} \xi_i = N$  and  $\xi_i \geq 0$  for all  $i \in \{1, \dots, N\}$ . Based on the QR-DQN setup, note that the condition  $\int_{w \in \Omega} \xi(w) \mathbb{P}(dw) = 1$  turns into  $\sum_{i=1}^{N} \frac{1}{N} \xi_i = 1$ , since the quantile level is set as  $\tau_i = \frac{i}{N}$ .
A key issue is how to construct an ambiguity set with bound  $\Delta_t$  and then sample  $\xi$ . A natural class of distribution for practical use is the symmetric Dirichlet distribution with concentration  $\beta$ , which represents distribution over distributions. (i.e.  $\pmb{x} \sim \mathrm{Dir}(\beta)$ .) We sample a random vector,  $\pmb{x} \sim \mathrm{Dir}(\beta)$ , and define the reweight distribution as  $\pmb{\xi} := \mathbf{1}^N + \alpha (N\pmb{x} - \mathbf{1}^N)$ . From the construction of  $\xi$ , we have  $1 - \alpha \leq \xi_i \leq 1 + \alpha (N - 1)$  for all  $i$  and it follows that  $|1 - \xi_i| \leq \alpha (N - 1)$ . By controlling  $\alpha$ , we can bound the deviation of  $\xi_i$  from 1 and bound the perturbation gap as
$$
\begin{array}{l} \sup  _ {s, a} | \mathbb {E} [ Z (s, a) ] - \mathbb {E} _ {\xi} [ Z (s, a) ] | = \sup  _ {s, a} \left| \int_ {w \in \Omega} Z (w; s, a) (1 - \xi (w)) \mathbb {P} (d w) \right| \\ \leq \sup  _ {w \in \Omega} | 1 - \xi (w) | \sup  _ {s, a} \mathbb {E} [ | Z (s, a) | ] \leq \sup  _ {w \in \Omega} | 1 - \xi (w) | V _ {\max } \leq \alpha (N - 1) V _ {\max }. \\ \end{array}
$$
Hence, letting  $\alpha \leq \frac{\Delta}{(N - 1)V_{\max}}$  is sufficient to obtain  $d(Z;\xi) \leq \Delta$  in the quantile setting. We set  $\beta = 0.05 \cdot \mathbf{1}^N$  to generate a constructive perturbation  $\xi_n$  which gap is close to the bound  $\Delta_n$ . For Assumption 3.2, our default schedule is set as  $\Delta_t = \Delta_0 t^{-(1 + \epsilon)}$  where  $\epsilon = 0.001$ .
Our experiments aim to investigate the following questions.
1. Does randomizing risk criterion successfully escape from the biased exploration in stochastic environments?
2. Can PQR accurately estimate a return distribution?
3. Can a perturbation-based exploration perform successfully as a behavior policy for the full Atari benchmark?
Figure 3: Illustration of the N-Chain environment [27] with high uncertainty starting from state  $s_2$ . To emphasize the intrinsic uncertainty, the reward of state  $s_4$  was set as a mixture model composed of two Gaussian distributions. Blue arrows indicate the risk-neutral optimal policy in this MDPs.
Figure 4: Empirical return distribution plot in N-Chain environment. The ground truth of each distribution is  $\gamma^2\mathcal{N}(10,0.1^2)$  and  $\gamma^2[\frac{1}{2}\mathcal{N}(5,0.1^2) + \frac{1}{2}\mathcal{N}(13,0.1^2)]$ . Each dot represents an indicator for choosing action. Since QR-DQN does not depend on other criterion, the dots are omitted.
4.1 Learning on Stochastic Environments with High Intrinsic Uncertainty
For intuitive comparison between optimism and randomized criterion, we design  $\mathbf{p}$ -DLTV, a perturbed variant of DLTV, where coefficient  $c_{t}$  is multiplied by a normal distribution  $\mathcal{N}(0,1^2)$ . Every experimental setup, pseudocodes, and implementation details can be found in Appendix C.
N-Chain with high intrinsic uncertainty. We extend N-Chain environment [27] with stochastic reward to evaluate action selection methods. A schematic diagram of the stochastic N-Chain environment is depicted in Figure 3. The reward is only given in the leftmost and rightmost states and the game terminates when one of the reward states is reached. We set the leftmost reward as  $\mathcal{N}(10,0.1^2)$  and the rightmost reward as  $\frac{1}{2}\mathcal{N}(5,0.1^2) + \frac{1}{2}\mathcal{N}(13,0.1^2)$  which has a lower mean as 9 but higher variance. The agent always starts from the middle state  $s_2$  and should move toward the leftmost state  $s_0$  to achieve the greatest expected return. For each state, the agent can take one of six available actions: left, right, and 4 no-op actions. The optimal policy with respect to mean is to move left twice from the start. We set the discount factor  $\gamma = 0.9$  and the coefficient  $c = 50$ .
Despite the simple configuration, the possibility to obtain higher reward in suboptimal state than the optimal state makes it difficult for an agent to determine which policy is optimal until it experiences enough to discern the characteristics of each distribution. Thus, the goal of our toy experiment is to evaluate how rapidly each algorithm could find a risk-neutral optimal policy. The results of varying the size of variance are reported in Appendix D.1.
Analysis of Experimental Results. As we design the mean of each return is intended to be similar, examining the learning behavior of the empirical return distribution for each algorithm can provide fruitful insights. Figure 4 shows the empirical PDF of return distribution by using Gaussian kernel density estimation. In Figure 4(b), DLTV fails to estimate the true optimal return distribution. While the return of  $(s_2, \text{right})$  (red line) is correctly estimated toward the ground truth,  $(s_2, \text{left})$  (blue line) does not capture the shape and mean due to the lack of experience. At  $20\mathrm{K}$
Figure 5: Total count of performing true optimal action. The oracle (dashed line) is to perform the optimal action from start to end.
timestep, the agent begins to see other actions, but the monotonic scheduling already makes the decision like exploitation. Hence, decaying schedule of optimism is not a way to solve the underlying problem. Notably, p-DLTV made a much better estimate than DLTV only by changing from optimism to a randomized scheme. In comparison, PQR estimates the ground truth much better than other baselines with much closer mean and standard-deviation.
Figure 5 shows the number of timesteps when the optimal policy was actually performed to see the interference of biased criterion. Since the optimal policy consists of the same index  $a_1$ , we plot the total count of performing the optimal action with 10 seeds. From the slope of each line, it is observed that DLTV selects the suboptimal action even if the optimal policy was initially performed. In contrast, p-DLTV avoids getting stuck by randomizing criterion and eventually finds the true optimal policy. The experimental results demonstrate that randomizing the criterion is a simple but effective way for exploration on training process.
Figure 6: 2-Wasserstein distance between the empirical return distribution and the ground truth  $\mathcal{N}(8.1,0.081^2)$ . We use QR-DQN with a fixed setting of  $\epsilon$ -greedy as a reference baseline, because the hyperparameter of  $\epsilon$ -greedy is not related to the scale of Q-values.
Hyperparameter Sensitivity. In Figure 6, we compute the 2-Wasserstein distance from the ground truth return distribution  $\mathcal{N}(10\gamma^2,(0.1\gamma^2)^2)$ . Except for QR-DQN, each initial hyperparameter  $\{c,\Delta_0\}$  was implemented with grid search on  $[1,5,10,50,100,500,1000,5000]$  in 5 different seeds. As the hyperparameter decreases, each agent is likely to behave as exploitation. One interesting aspect is that, while it may be difficult for DLTV and p-DLTV to balance the scale between the return and bonus term, PQR shows robust performance to the initial hyperparameter. This is because the distorted return is bounded by the support of return distribution, so that PQR implicitly tunes the scale of exploration. In practice, we set  $\Delta_0$  to be sufficiently large. See Table2 in Appendix C.1.
4.2 Full Atari Results
We compare our algorithm to various DistRL baselines, which have demonstrated good performance on RL benchmarks. In Table 1, we evaluated 55 Atari results, averaging over 5 different seeds at 50M frames. We compared with the published score of QR-DQN [13], IQN [12], and Rainbow [16] via the report of DQN-Zoo [29] and Dopamine [4] benchmark for reliability. This comparison is particularly noteworthy since our proposed method only applies perturbation-based exploration strategy and outperforms advanced variants of QR-DQN.
Table 1: Mean and median of best scores across 55 Atari games, measured as percentages of human baseline. Reference values are from Quan and Ostrovski [29] and Castro et al. [4].
50M Performance	Mean	Median	>human	>DQN
DQN-zoo (no-ops)	314%	55%	18	0
DQN-dopamine (sticky)	401%	51%	15	0
QR-DQN-zoo (no-ops)	559%	118%	29	47
QR-DQN-dopamine (sticky)	562%	93%	27	46
IQN-zoo (no-ops)	902%	131%	21	50
IQN-dopamine (sticky)	940%	124%	32	51
RAINBOW-zoo (no-ops)	1160%	154%	37	52
RAINBOW-dopamine (sticky)	965%	123%	35	53
PQR-zoo (no-ops)	1121%	124%	33	53
PQR-dopamine (sticky)	962%	123%	35	51
2In Dopamine framework, IQN was implemented with  $n$ -step updates with  $n = 3$ , which improves performance.
Figure 7: Evaluation curves on 8 Atari games with 3 random seeds for 50 million frames following sticky actions protocol [20]. Reference values are from Castro et al. [4].
No-ops Protocol. First, we follow the evaluation protocol of [1, 22] on full set of Atari games implemented in OpenAI's Gym [3]. Even if it is well known that the no-ops protocol does not provide enough stochasticity to avoid memorization, intrinsic uncertainty still exists due to the random frame skipping [20]. While PQR cannot enjoy the environmental stochasticity by the deterministic dynamics of Atari games, PQR achieved  $562\%$  performance gain in the mean of human-normalized score over QR-DQN, which is comparable results to Rainbow. From the raw scores of 55 games, PQR wins 39 games against QR-DQN and 34 games against IQN.
Sticky actions protocol. To prevent the deterministic dynamics of Atari games, Machado et al. [20] proposes injecting stochasticity scheme, called sticky actions, by forcing to repeat the previous action with probability  $p = 0.25$ . Sticky actions protocol prevents agents from relying on memorization and allows robust evaluation. In Figure 7, PQR shows steeper learning curves, even without any support of advanced schemes, such as  $n$ -step updates for Rainbow or IQN. In particular, PQR dramatically improves over IQN and Rainbow in ASSAULT, BATTLEZONE, BEAMRIDER, BERZERK and BOWLING. In Table 1, PQR shows robust median score against the injected stochasticity.
It should be noted that IQN benefits from the generalized form of distributional outputs, which reduces the approximation error from the number of quantiles output. Compare to IQN, PQR does not rely on prior distortion risk measure such as CVaR [7], Wang [37] or CPW [36], but instead randomly samples the risk measure and evaluates it with a risk-neutral criterion. Another notable difference is that PQR shows the better or competitive performance solely through its exploration strategies, compared to  $\epsilon$ -greedy baselines, such as QR-DQN, IQN, and especially Rainbow. Note that Rainbow enjoys a combination of several orthogonal improvements such as double Q-learning, prioritized replay,ueling networks, and  $n$ -step updates.
Randomized or perturbation-based exploration has been focused due to its strong empirical performance and simplicity. In tabular RL, Osband et al. [28] proposed randomized least-squares value iteration (RLSVI) using random perturbations for statistically and computationally efficient exploration. Ishfaq et al. [17] leveraged the idea into optimistic reward sampling by perturbing rewards and regularizers. However, existing perturbation-based methods require tuning of the hyperparameter for the variance of injected Gaussian noise and depend on well-crafted feature vectors in advance. On the other hand, PDBOO does not rely on the scale of rewards or uncertainties due to the built-in scaling mechanism of risk measures. Additionally, we successfully extend PQR to deep RL scenarios in distributional lens, where feature vectors are not provided, but learned during training.
In this paper, we proposed a general framework of perturbation in distributional RL which is based on the characteristics of a return distribution. Without resorting to a pre-defined risk criterion,
we revealed and resolved the underlying problem where one-sided tendency on risk can lead to biased action selection under the stochastic environment. To our best knowledge, this paper is the first attempt to integrate risk-sensitivity and exploration by using time-varying Bellman objective with theoretical analysis. In order to validate the effectiveness of PQR, we evaluate on various environments including 55 Atari games with several distributional RL baselines. Without separating the two uncertainties, the results show that perturbing the risk criterion is an effective approach to resolve the biased exploration. We believe that PQR can be combined with other distributional RL or risk-sensitive algorithms as a perturbation-based exploration method without sacrificing their original objectives.
This work was supported in part by National Research Foundation of Korea (NRF, 2021R1A2C2014504(20%) and 2021M3F3A2A02037893(20%)), in part by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Ministry of Science and ICT (MSIT) (2021-0-00106(15%), 2021-0-02068(15%), 2021-0-01080(15%), and 2021-0-01341(15%)), and in part by AI Graduate School Program, CAU, INMAC and BK21 FOUR program.
[1] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[2] Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In International Conference on Machine Learning, pages 449-458. PMLR, 2017.
[3] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
[4] Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G Bellemare. Dopamine: A research framework for deep reinforcement learning. arXiv preprint arXiv:1812.06110, 2018.
[5] Richard Y Chen, Szymon Sidor, Pieter Abbeel, and John Schulman. Ucb exploration via q-ensembles. arXiv preprint arXiv:1706.01502, 2017.
[6] Yunho Choi, Kyungjae Lee, and Songhwai Oh. Distributional deep reinforcement learning with a mixture of gaussians. In 2019 International Conference on Robotics and Automation (ICRA), pages 9791-9797. IEEE, 2019.
[7] Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for cvar optimization in mdps. Advances in neural information processing systems, 27, 2014.
[8] Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decision-making: a cvar optimization approach. arXiv preprint arXiv:1506.02188, 2015.
[9] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 18(1):6070-6120, 2017.
[10] Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann. Better exploration with optimistic actor-critic. arXiv preprint arXiv:1910.12807, 2019.
[11] William R Clements, Bastien Van Delft, Benoit-Marie Robaglia, Reda Bahi Slaoui, and Sébastien Toth. Estimating risk and uncertainty in deep reinforcement learning. arXiv preprint arXiv:1905.09638, 2019.
[29] John Quan and Georg Ostrovski. DQN Zoo: Reference implementations of DQN-based agents, 2020. URL http://github.com/deepmind/dqn_zoo.
[30] Marc Rigter, Bruno Lacerda, and Nick Hawes. Risk-averse bayes-adaptive reinforcement learning. arXiv preprint arXiv:2102.05762, 2021.
[31] R Tyrrell Rockafellar, Stanislav Uryasev, et al. Optimization of conditional value-at-risk. Journal of risk, 2:21-42, 2000.
[32] Alexander Shapiro, Darinka Dentcheva, and Andrzej Ruszczyński. Lectures on stochastic programming: modeling and theory. SIAM, 2021.
[33] Elena Smirnova, Elvis Dohmatob, and Jérémie Mary. Distributionally robust reinforcement learning. arXiv preprint arXiv:1902.08708, 2019.
[34] Silvestr Stanko and Karel Macek. Risk-averse distributional reinforcement learning: A cvar optimization approach. In IJCCI, pages 412–423, 2019.
[35] Yunhao Tang and Shipra Agrawal. Exploration by distributional reinforcement learning. arXiv preprint arXiv:1805.01907, 2018.
[36] Amos Tversky and Daniel Kahneman. Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and uncertainty, 5(4):297-323, 1992.
[37] Shaun S Wang. A class of distortion operators for pricing financial and insurance risks. Journal of risk and insurance, pages 15-36, 2000.
[38] Derek Yang, Li Zhao, Zichuan Lin, Tao Qin, Jiang Bian, and Tie-Yan Liu. Fully parameterized quantile function for distributional reinforcement learning. Advances in neural information processing systems, 32:6193-6202, 2019.
[39] Insoon Yang. Wasserstein distributionally robust stochastic control: A data-driven approach. IEEE Transactions on Automatic Control, 2020.
[40] Shangtong Zhang and Hengshuai Yao. Quota: The quantile option architecture for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5797-5804, 2019.
[41] Shangtong Zhang, Bo Liu, and Shimon Whiteson. Mean-variance policy iteration for risk-averse reinforcement learning. arXiv preprint arXiv:2004.10888, 2020.
[42] Fan Zhou, Zhoufan Zhu, Qi Kuang, and Liwen Zhang. Non-decreasing quantile function network with efficient exploration for distributional reinforcement learning. arXiv preprint arXiv:2105.06696, 2021.
A.1 Technical Lemma
Before proving our theoretical results, we present two inequalities for supremum to clear the description.
1.  $\sup_{x\in X}|f(x) + g(x)|\leq \sup_{x\in X}|f(x)| + \sup_{x\in X}|g(x)|$
2.  $\left|\sup_{x\in X}f(x) - \sup_{x'\in X}g(x')\right|\leq \sup_{x,x'\in X}|f(x) - g(x')|$
Proof of 1. Since  $|f(x) + g(x)| \leq |f(x)| + |g(x)|$  holds for all  $x \in X$ ,
$$
\begin{array}{l} \sup  _ {x \in X} | f (x) + g (x) | \leq \sup  _ {x \in X} (| f (x) | + | g (x) |) \\ \leq \sup  _ {x \in X} | f (x) | + \sup  _ {x \in X} | g (x) | \\ \end{array}
$$
Proof of 2. Since  $\left|\|a\| - \|b\|\right| \leq \|a - b\|$  for any norm  $\| \cdot \|$  and for a large enough  $M$ ,
$$
\begin{array}{l} \sup  _ {x, x ^ {\prime} \in X} | f (x) - g \left(x ^ {\prime}\right) | \geq \sup  _ {x \in X} | f (x) - g (x) | \\ = \sup  _ {x \in X} | (f (x) + M) - (g (x) + M) | \\ \geq \left| \sup  _ {x \in X} (f (x) + M) - \sup  _ {x \in X} (g (x) + M) \right| \\ = \left| \sup  _ {x \in X} f (x) - \sup  _ {x ^ {\prime} \in X} g \left(x ^ {\prime}\right) \right| \\ \end{array}
$$
A.2 Proof of Theorem A.3
Theorem A.3. If  $\xi_{t}$  converges to 1 in probability on  $\Omega$ , then  $\mathbb{E}\mathcal{T}_{\xi_t}$  converges to  $\mathbb{E}\mathcal{T}$  uniformly on  $\mathcal{Z}$  for all  $s\in S$  and  $a\in \mathcal{A}$ .
Proof. Recall that  $\mathcal{Z} = \left\{Z:\mathcal{S}\times \mathcal{A}\to \mathcal{P}(\mathbb{R})\big|\mathbb{E}[|Z(s,a)|]\leq V_{\max},\forall (s,a)\right\}$ . Then for any  $Z\in \mathcal{Z}$  and  $\xi \in \Xi$
$$
\mathbb {E} \left[ | \mathcal {T} _ {\xi} Z | \right] \leq R _ {\max } + \gamma \frac {R _ {\max }}{1 - \gamma} = \frac {R _ {\max }}{1 - \gamma} = V _ {\max }.
$$
which implies PDBOO is closed in  $\mathcal{Z}$ , i.e.  $\mathcal{T}_{\xi}Z \in \mathcal{Z}$  for all  $\xi \in \Xi$ . Hence, for any sequence  $\xi_{t}$ ,  $Z^{(n)} = \mathcal{T}_{\xi_{n:1}}Z \in \mathcal{Z}$  for any  $n \geq 0$ .
Since  $\xi_{t}$  converges to 1 in probability on  $\Omega$ , there exists  $T$  such that for any  $\epsilon, \delta > 0$  and  $t > T$ ,
$$
\mathbb {P} \left(\Omega_ {t}\right) := \mathbb {P} \left(\left\{w \in \Omega : \sup  _ {w \in \Omega} | \xi_ {t} (w) - 1 | \geq \epsilon \right\}\right) \leq \delta
$$
For any  $Z\in \mathcal{Z},s\in S,a\in \mathcal{A}$  , and  $t > T$  , by using Hölder's inequality,
$$
\begin{array}{l} \sup  _ {Z \in \mathcal {Z}} \sup  _ {s, a} | \mathbb {E} _ {\xi_ {t}} [ Z (s, a) ] - \mathbb {E} [ Z (s, a) ] | = \sup  _ {Z \in \mathcal {Z}} \sup  _ {s, a} \left| \int_ {w \in \Omega} (1 - \xi_ {t} (w)) Z (s, a, w) \mathbb {P} (d w) \right| \\ = \sup  _ {Z \in \mathcal {Z}} \sup  _ {s, a} \left| \int_ {w \in \Omega_ {t}} (1 - \xi_ {t} (w)) Z (s, a, w) \mathbb {P} (d w) + \int_ {w \in \Omega \backslash \Omega_ {t}} (1 - \xi_ {t} (w)) Z (s, a, w) \mathbb {P} (d w) \right| \\ \leq \mathbb {P} (\Omega_ {t}) \sup  _ {w \in \Omega_ {t}} | \xi_ {t} (w) - 1 | V _ {\max } + \mathbb {P} (\Omega \backslash \Omega_ {t}) \sup  _ {w \in \Omega \backslash \Omega_ {t}} | \xi_ {t} (w) - 1 | V _ {\max } \\ \leq \delta \left| B _ {\xi} - 1 \right| V _ {\max } + \epsilon V _ {\max } \\ \end{array}
$$
which implies that  $\mathbb{E}_{\xi_t}$  converges to  $\mathbb{E}$  uniformly on  $\mathcal{Z}$  for all  $s, a$ .
By using A.1, we can get the desired result.
$$
\begin{array}{l} \sup  \sup  | \mathbb {E} [ \mathcal {T} _ {\xi_ {t}} Z (s, a) ] - \mathbb {E} [ \mathcal {T} Z (s, a) ] | \\ Z \in \mathcal {Z} \quad s, a \\ \leq \sup  _ {Z \in \mathcal {Z}} \sup  _ {s, a} | \mathbb {E} [ \mathcal {T} _ {\xi_ {t}} Z (s, a) ] - \mathbb {E} _ {\xi_ {t}} [ \mathcal {T} _ {\xi_ {t}} Z (s, a) ] | + \sup  _ {Z \in \mathcal {Z}} \sup  _ {s, a} | \mathbb {E} _ {\xi_ {t}} [ \mathcal {T} _ {\xi_ {t}} Z (s, a) ] - \mathbb {E} [ \mathcal {T} Z (s, a) ] | \\ \leq \left(\delta \left| B _ {\xi} - 1 \right| V _ {\max } + \epsilon V _ {\max }\right) + \gamma \sup  _ {Z \in \mathcal {Z}} \sup  _ {s, a} \mathbb {E} _ {s ^ {\prime}} \left[ \left| \sup  _ {a ^ {\prime}} \mathbb {E} _ {\xi_ {t}} \left[ Z \left(s ^ {\prime}, a ^ {\prime}\right) \right] - \sup  _ {a ^ {\prime \prime}} \mathbb {E} [ Z \left(s ^ {\prime}, a ^ {\prime \prime}\right) ] \right| \right] \\ \leq \left(\delta \left| B _ {\xi} - 1 \right| V _ {\max } + \epsilon V _ {\max }\right) + \gamma \sup  _ {Z \in \mathcal {Z}} \sup  _ {s ^ {\prime}, a ^ {\prime}} \left| \mathbb {E} _ {\xi_ {t}} \left[ Z \left(s ^ {\prime}, a ^ {\prime}\right) \right] - \mathbb {E} [ Z \left(s ^ {\prime}, a ^ {\prime}\right) ] \right| \\ \leq \left(\delta \left| B _ {\xi} - 1 \right| V _ {\max } + \epsilon V _ {\max }\right) + \gamma \left(\delta \left| B _ {\xi} - 1 \right| V _ {\max } + \epsilon V _ {\max }\right) \\ = (1 + \gamma) (\delta | B _ {\xi} - 1 | V _ {\max } + \epsilon V _ {\max }) \\ \end{array}
$$
A.3 Proof of Theorem 3.3
Theorem 3.3. Let  $\xi_{n}$  be sampled from  $\bar{\mathcal{U}}_{\Delta_n}(Z^{(n - 1)})$  for every iteration. If Assumption 3.2 holds, then the expectation of any composition of operators  $\mathbb{E}\mathcal{T}_{\xi_{n:1}}$  converges, i.e.  $\mathbb{E}\mathcal{T}_{\xi_{n:1}}[Z]\to \mathbb{E}[Z^* ]$
Moreover, the following bound holds,
Proof. We denote  $a_{i}^{*}(\xi_{n}) = \operatorname*{argmax}_{a^{\prime}}\mathbb{E}_{\xi_{n}}[Z_{i}^{(n - 1)}(s^{\prime},a^{\prime})]$  as the greedy action of  $Z_{i}^{(n - 1)}$  under perturbation  $\xi_{n}$ . Also, we denote  $\sup_{s,a}\big|\cdot \big|$  which is the supremum norm over  $s$  and  $a$  as  $\| \cdot \|_{sa}$ .
Before we start from the term  $\left\| \mathbb{E}[Z^{(k + 1)}] - \mathbb{E}[Z^{(k)}]\right\|_{sa}$ , for a given  $(s,a)$