SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lingwei Zhu
University of Alberta lingwei4@ualberta.ca
Zheng Chen
Osaka University chenz@sanken.osaka-u.ac.jp
Matthew Schlegel
University of Alberta mkschleg@ualberta.ca
Martha White
University of Alberta CIFAR Canada AI Chair, Amii whitem@ualberta.ca
Many policy optimization approaches in reinforcement learning incorporate a Kullback-Leibler (KL) divergence to the previous policy, to prevent the policy from changing too quickly. This idea was initially proposed in a seminal paper on Conservative Policy Iteration, with approximations given by algorithms like TRPO and Munchausen Value Iteration (MVI). We continue this line of work by investigating a generalized KL divergence—called the Tsallis KL divergence. Tsallis KL defined by the  $q$ -logarithm is a strict generalization, as  $q = 1$  corresponds to the standard KL divergence;  $q > 1$  provides a range of new options. We characterize the types of policies learned under the Tsallis KL, and motivate when  $q > 1$  could be beneficial. To obtain a practical algorithm that incorporates Tsallis KL regularization, we extend MVI, which is one of the simplest approaches to incorporate KL regularization. We show that this generalized MVI( $q$ ) obtains significant improvements over the standard MVI( $q = 1$ ) across 35 Atari games.
There is ample theoretical evidence that it is useful to incorporate KL regularization into policy optimization in reinforcement learning. The most basic approach is to regularize towards a uniform policy, resulting in entropy regularization. More effective, however, is to regularize towards the previous policy. By choosing KL regularization between consecutively updated policies, the optimal policy becomes a softmax over a uniform average of the full history of action value estimates [Vieillard et al., 2020a]. This averaging smooths out noise, allowing for better theoretical results [Azar et al., 2012, Kozuno et al., 2019, Vieillard et al., 2020a, Kozuno et al., 2022].
Despite these theoretical benefits, there are some issues with using KL regularization in practice. It is well-known that the uniform average is susceptible to outliers; this issue is inherent to KL divergence [Futami et al., 2018]. In practice, heuristics such as assigning vanishing regularization coefficients to some estimates have been implemented widely to increase robustness and accelerate learning [Grau-Moya et al., 2019, Haarnoja et al., 2018, Kitamura et al., 2021]. However, theoretical guarantees no longer hold for those heuristics [Vieillard et al., 2020a, Kozuno et al., 2022]. A natural question is what alternatives we can consider to this KL divergence regularization, that allows us to overcome some of these disadvantages while maintaining the benefits associate with restricting aggressive policy changes and smoothing errors.
In this work, we explore one possible direction by generalizing to Tsallis KL divergences. Tsallis KL divergences were introduced for physics [Tsallis, 1988, 2009] using a simple idea: replacing the use of the logarithm with the deformed  $q$ -logarithm. The implications for policy optimization, however, are that we get quite a different form for the resulting policy. Tsallis entropy with  $q = 2$  has actually already been considered for policy optimization [Chow et al., 2018, Lee et al., 2018], by replacing Shannon entropy with Tsallis entropy to maintain stochasticity in the policy. The resulting policies are called sparsemax policies, because they concentrate the probability on higher-valued actions and truncate the probability to zero for lower-valued actions. Intuitively, this should have the benefit of maintaining stochasticity, but only amongst the most promising actions, unlike the Boltzmann policy which maintains nonzero probability on all actions. Unfortunately, using only Tsallis entropy did not provide significant benefits, and in fact often performed worse than existing methods. We find, however, that using a Tsallis KL divergence to the previous policy does provide notable gains.
We first show how to incorporate Tsallis KL regularization into the standard value iteration updates, and prove that we maintain convergence under this generalization from KL regularization to Tsallis KL regularization. We then characterize the types of policies learned under Tsallis KL, highlighting that there is now a more complex relationship to past action-values than a simple uniform average. We then show how to extend Munchausen Value Iteration (MVI) [Vieillard et al., 2020b], to use Tsallis KL regularization, which we call  $\mathrm{MVI}(q)$ . We use this naming convention to highlight that this is a strict generalization of MVI: by setting  $q = 1$ , we exactly recover MVI. We then compare  $\mathrm{MVI}(q = 2)$  with MVI (namely the standard choice where  $q = 1$ ), and find that we obtain significant performance improvements in Atari.
Remark: There is a growing body of literature studying generalizations of KL divergence in RL [Nachum et al., 2019, Zhang et al., 2020]. Futami et al. [2018] discussed the inherent drawback of KL divergence in generative modeling and proposed to use  $\beta$ - and  $\gamma$ -divergence to allow for weighted average of sample contribution. These divergences fall under the category known as the  $f$ -divergence [Sason and Verdú, 2016], commonly used in other machine learning domains including generative modeling [Nowozin et al., 2016, Wan et al., 2020, Yu et al., 2020] and imitation learning [Ghasemipour et al., 2019, Ke et al., 2019]. In RL, Wang et al. [2018] discussed using tail adaptive  $f$ -divergence to enforce the mass-covering property. Belousov and Peters [2019] discussed the use of  $\alpha$ -divergence. Tsallis KL divergence, however, has not yet been studied in RL.
We focus on discrete-time discounted Markov Decision Processes (MDPs) expressed by the tuple  $(\mathcal{S},\mathcal{A},d,P,r,\gamma)$ , where  $\mathcal{S}$  and  $\mathcal{A}$  denote state space and finite action space, respectively. Let  $\Delta (\mathcal{X})$  denote the set of probability distributions over  $\mathcal{X}$ .  $d\in \Delta (\mathcal{S})$  denotes the initial state distribution.  $P:S\times \mathcal{A}\to \Delta (\mathcal{S})$  denotes the transition probability function, and  $r(s,a)$  defines the reward associated with that transition.  $\gamma \in (0,1)$  is the discount factor. A policy  $\pi :\mathcal{S}\rightarrow \Delta (\mathcal{A})$  is a mapping from the state space to distributions over actions. We define the action value function following policy  $\pi$  and starting from  $s_0\sim d(\cdot)$  with action  $a_0$  taken as  $Q_{\pi}(s,a) = \mathbb{E}_{\pi}\left[\sum_{t = 0}^{\infty}\gamma^{t}r_{t}|s_{0} = s,a_{0} = a\right]$ . A standard approach to find the optimal value function  $Q_{*}$  is value iteration. To define the formulas for value iteration, it will be convenient to write the action value function as a matrix  $Q_{\pi}\in \mathbb{R}^{|\mathcal{S}|\times |\mathcal{A}|}$ . For notational convenience, we define the inner product for any two functions  $F_{1},F_{2}\in \mathbb{R}^{|\mathcal{S}|\times |\mathcal{A}|}$  over actions as  $\langle F_1,F_2\rangle \in \mathbb{R}^{|\mathcal{S}|}$ .
We are interested in the entropy-regularized MDPs where the recursion is augmented with  $\Omega (\pi)$ :
$$
\left\{ \begin{array}{l} \pi_ {k + 1} = \arg \max  _ {\pi} \left(\left\langle \pi , Q _ {k} \right\rangle - \tau \Omega (\pi)\right), \\ Q _ {k + 1} = r + \gamma P \left(\left\langle \pi_ {k + 1}, Q _ {k} \right\rangle - \tau \Omega \left(\pi_ {k + 1}\right)\right) \end{array} \right. \tag {1}
$$
This modified recursion is guaranteed to converge if  $\Omega$  is concave in  $\pi$ . For standard (Shannon) entropy regularization, we use  $\Omega(\pi) = -\mathcal{H}(\pi) = \langle \pi, \ln \pi \rangle$ . The resulting optimal policy has  $\pi_{k+1} \propto \exp(\tau^{-1}Q_k)$ , where  $\propto$  indicates proportional to up to a constant not depending on actions.
More generally, we can consider a broad class of regularizers known as  $f$ -divergences [Sason and Verdú, 2016]:  $\Omega(\pi) = D_f(\pi ||\mu) \coloneqq \langle \mu, f(\pi / \mu) \rangle$ , where  $f$  is a convex function. For example, the KL divergence  $D_{KL}(\pi \| \mu) = \langle \pi, \ln \pi - \ln \mu \rangle$  can be recovered by  $f(t) = -\ln t$ . In this work, when we say KL regularization, we mean the standard choice of setting  $\mu = \pi_k$ , the estimate from the previous update. Therefore,  $D_{\mathrm{KL}}$  serves as a penalty to penalize aggressive policy changes. The
Figure 1:  $\ln_q x$ ,  $\exp_q x$  and Tsallis entropy component  $-\pi^q \ln_q \pi$  for  $q = 1$  to 5. When  $q = 1$  they respectively recover their standard counterpart.  $\pi$  is chosen to be Gaussian  $\mathcal{N}(2,1)$ . As  $q$  gets larger  $\ln_q x$  (and hence Tsallis entropy) becomes more flat and  $\exp_q x$  more steep.
optimal policy in this case takes the form  $\pi_{k + 1}\propto \pi_k\exp \left(\tau^{-1}Q_k\right)$ . By induction, we can show this KL-regularized optimal policy  $\pi_{k + 1}$  is a softmax over a uniform average over the history of action value estimates [Vieillard et al., 2020a]:  $\pi_{k + 1}\propto \pi_k\exp \left(\tau^{-1}Q_k\right)\propto \dots \propto \exp \left(\tau^{-1}\sum_{j = 1}^{k}Q_j\right)$ . Using KL regularization has been shown to be theoretically superior to entropy regularization in terms of error tolerance [Azar et al., 2012, Vieillard et al., 2020a, Kozuno et al., 2022, Chan et al., 2022].
The definitions of  $\mathcal{H}(\cdot)$  and  $D_{\mathrm{KL}}(\cdot ||\cdot)$  rely on the standard logarithm and both induce softmax policies as an exponential (inverse function) over (weighted) action-values [Hiriart-Urruty and Lemarechal, 2004, Nachum and Dai, 2020]. Convergence properties of the resulting regularized algorithms have been well studied [Kozuno et al., 2019, Geist et al., 2019, Vieillard et al., 2020a]. In this paper, we investigate Tsallis entropy and Tsallis KL divergence as the regularizer, which generalize Shannon entropy and KL divergence respectively.
3 Generalizing to Tsallis Regularization
We can easily incorporate other regularizers in to the value iteration recursion, and maintain convergence as long as those regularizers are strongly convex in  $\pi$ . We characterize the types of policies that arise from using this regularizer, and prove the convergence of resulting regularized recursion.
3.1 Tsallis Entropy Regularization
Tsallis entropy was first proposed by Tsallis [1988] and is defined by the  $q$ -logarithm. The  $q$ -logarithm and its unique inverse function, the  $q$ -exponential, are defined as:
$$
\ln_ {q} x := \frac {x ^ {1 - q} - 1}{1 - q}, \quad \exp_ {q} x := [ 1 + (1 - q) x ] _ {+} ^ {\frac {1}{1 - q}}, \quad \text {f o r} q \in \mathbb {R} \backslash \{1 \} \tag {2}
$$
where  $[\cdot]_{+} := \max\{\cdot, 0\}$ . We define  $\ln_1 = \ln$ ,  $\exp_1 = \exp$ , as in the limit  $q \to 1$ , the formulas in Eq. (2) approach these functions. Tsallis entropy can be defined by  $S_q(\pi) := p\langle -\pi^q, \ln_q\pi \rangle$ ,  $p \in \mathbb{R}$  [Suyari and Tsukada, 2005]. We visualize the  $q$ -logarithm,  $q$ -exponential and Tsallis entropy for different  $q$  in Figure 1. As  $q$  gets larger,  $q$ -logarithm (and hence Tsallis entropy) becomes more flat and  $q$ -exponential more steep<sup>1</sup>. Note that  $\exp_q$  is only invertible for  $x > \frac{-1}{1-q}$ .
Tsallis policies have a similar form to softmax, but using the  $q$ -exponential instead. Let us provide some intuition for these policies. When  $p = \frac{1}{2}, q = 2$ ,  $S_{2}(\pi) = \frac{1}{2}\langle \pi, 1 - \pi \rangle$ , the optimization problem  $\arg \max_{\pi \in \Delta(\mathcal{A})} \langle \pi, Q \rangle + S_{2}(\pi) = \arg \min_{\pi \in \Delta(\mathcal{A})} \| \pi - Q \|_{2}^{2}$  is known to be the Euclidean projection onto the probability simplex. Its solution  $[Q - \psi]_{+}$  is called the sparsemax [Martins and Astudillo, 2016, Lee et al., 2018] and has sparse support [Duchi et al., 2008, Condat, 2016, Blondel et al., 2020].  $\psi : S \times \mathcal{A} \to S$  is the unique function satisfying  $\langle 1, [Q - \psi]_{+} \rangle = 1$ .
As our first result, we unify the Tsallis entropy regularized policies for all  $q \in \mathbb{R}_+$  with the  $q$ -exponential, and show that  $q$  and  $\tau$  are interchangeable for controlling the truncation.
<sup>1</sup>The  $q$ -logarithm defined here is consistent with the physics literature and different from prior RL works [Lee et al., 2020], where a change of variable  $q^{*} = 2 - q$  is made. We analyze both cases in Appendix A.
Figure 2: (Left) Tsallis KL component  $-\pi_1 \ln_q \frac{\pi_2}{\pi_1}$  between two Gaussian policies  $\pi_1 = \mathcal{N}(2.75, 1)$ ,  $\pi_2 = \mathcal{N}(3.25, 1)$  for  $q = 1$  to 5. When  $q = 1$  TKL recovers KL. For  $q > 1$ , TKL is more mode-covering than KL. (Mid) The sparsemax operator acting on a Boltzmann policy when  $q = 2$ . (Right) The sparsemax when  $q = 50$ . Truncation gets stronger as  $q$  gets larger. The same effect can be also controlled by  $\tau$ .
Theorem 1. Let  $\Omega (\pi) = -S_q(\pi)$  in Eq.(1). Then the regularized optimal policies can be expressed:
$$
\pi (a | s) = \sqrt [ 1 - q ]{\left[ \frac {Q (s , a)}{\tau} - \tilde {\psi} _ {q} \left(\frac {Q (s , \cdot)}{\tau}\right) \right] _ {+} (1 - q)} = \exp_ {q} \left(\frac {Q (s , a)}{\tau} - \psi_ {q} \left(\frac {Q (s , \cdot)}{\tau}\right)\right) \tag {3}
$$
where  $\psi_q = \tilde{\psi}_q + \frac{1}{1 - q}$ . Additionally, for an arbitrary  $(q,\tau)$  pair with  $q > 1$ , the same truncation effect (support) can be achieved using  $(q = 2,\frac{\tau}{q - 1})$ .
Proof. See Appendix B for the full proof.
Theorem 1 characterizes the role played by  $q$ : controlling the degree of truncation. We show the truncation effect when  $q = 2$  and  $q = 50$  in Figure 2, confirming that Tsallis policies tend to truncate more as  $q$  gets larger. The theorem also highlights that we can set  $q = 2$  and still get more or less truncation using different  $\tau$ , helping to explain why in our experiments  $q = 2$  is a generally effective choice.
Unfortunately, the threshold  $\tilde{\psi}_q$  (and  $\psi_q$ ) does not have a closed-form solution for  $q \neq 1,2,\infty$ . Note that  $q = 1$  corresponds to Shannon entropy and  $q = \infty$  to no regularization. However, we can resort to Taylor's expansion to obtain approximate sparsemax policies.
Theorem 2. For  $q \neq 1, \infty$ , we can obtain approximate threshold  $\hat{\psi}_q \approx \psi_q$  using Taylor's expansion, and therefore an approximate policy:
$$
\hat {\pi} (a | s) \propto \exp_ {q} \left(\frac {Q (s , a)}{\tau} - \hat {\psi} _ {q} \left(\frac {Q (s , \cdot)}{\tau}\right)\right), \hat {\psi} _ {q} \left(\frac {Q (s , \cdot)}{\tau}\right) \doteq \frac {\sum_ {a \in K (s)} \frac {Q (s , \cdot)}{\tau} - 1}{| K (s) |} + 1. \tag {4}
$$
$K(s)$  is the set of highest-valued actions, satisfying the relation  $1 + i\frac{Q(s,a_{(i)})}{\tau} > \sum_{j=1}^{i}\frac{Q(s,a_{(j)})}{\tau}$ , where  $a_{(j)}$  indicates the action with  $j$ th largest action value. The sparsemax policy sets the probabilities of lowest-valued actions to zero:  $\pi(a_{(i)}|s) = 0, i = z+1,\ldots,|\mathcal{A}|$  where  $Q(s,a_{(z)}) > \tau^{-1}\hat{\psi}_q\left(\frac{Q(s,\cdot)}{\tau}\right) > Q(s,a_{(z+1)})$ . When  $q = 2$ ,  $\hat{\psi}_q$  recovers  $\psi_q$ .
Lee et al. [2020] also used  $\exp_q$  to represent policies but they consider the continuous action setting and do not give any computable threshold. By contrast, Theorem 2 presents an easily computable  $\hat{\psi}_q$  for all  $q \notin \{1, \infty\}$ .
3.2 Tsallis KL Regularization and Convergence Results
The Tsallis KL divergence is defined as  $D_{KL}^{q}(\pi \parallel \mu) \coloneqq \left\langle \pi, -\ln_{q}\frac{\mu}{\pi}\right\rangle$  [Furuichi et al., 2004]. It is a member of  $f$ -divergence and can be recovered by choosing  $f(t) = -\ln_q t$ . As a divergence penalty,
it is required that  $q > 0$  since  $f(t)$  should be convex. We further assume that  $q > 1$  to align with standard divergences; i.e. penalize large value of  $\frac{\pi}{\mu}$ , since for  $0 < q < 1$  the regularization would penalize  $\frac{\mu}{\pi}$  instead. In practice, we find that  $0 < q < 1$  tend to perform poorly. In contrast to KL, Tsallis KL is more mass-covering; i.e. its value is proportional to the  $q$ -th power of the ratio  $\frac{\pi}{\mu}$ . When  $q$  is big, large values of  $\frac{\pi}{\mu}$  are strongly penalized [Wang et al., 2018]. This behavior of Tsallis KL divergence can also be found in other well-known divergences: the  $\alpha$ -divergence [Wang et al., 2018, Belousov and Peters, 2019] coincides with Tsallis KL when  $\alpha = 2$ ; Rényi's divergence also penalizes large policy ratio by raising it to the power  $q$ , but inside the logarithm, which is therefore an additive extension of KL [Li and Turner, 2016]. In the limit of  $q \to 1$ , Tsallis entropy recovers Shannon entropy and the Tsallis KL divergence recovers the KL divergence. We plot the Tsallis KL divergence behavior in Figure 2.
Now let us turn to formalizing when value iteration under Tsallis regularization converges. The  $q$ -logarithm has the following properties: Convexity:  $\ln_q\pi$  is convex for  $q \leq 0$ , concave for  $q > 0$ . When  $q = 0$ , both  $\ln_q$ ,  $\exp_q$  become linear. Monotonicity:  $\ln_q\pi$  is monotonically increasing with respect to  $\pi$ . These two properties can be simply verified by checking the first and second order derivative. We prove in Appendix A the following similarity between Shannon entropy (reps. KL) and Tsallis entropy (resp. Tsallis KL). Bounded entropy: we have  $0 \leq \mathcal{H}(\pi) \leq \ln |\mathcal{A}|$ ; and  $\forall q, 0 \leq S_q(\pi) \leq \ln_q|\mathcal{A}|$ . Generalized KL property:  $\forall q, D_{KL}^q(\pi \parallel \mu) \geq 0$ .  $D_{KL}^q(\pi \parallel \mu) = 0$  if and only if  $\pi = \mu$  almost everywhere, and  $D_{KL}^q(\pi \parallel \mu) \to \infty$  whenever  $\pi(a|s) > 0$  and  $\mu(a|s) = 0$ .
However, despite their similarity, a crucial difference is that  $\ln_q$  is non-extensive, which means it is not additive [Tsallis, 1988]. In fact,  $\ln_q$  is only pseudo-additive:
$$
\ln_ {q} \pi \mu = \ln_ {q} \pi + \ln_ {q} \mu + (1 - q) \ln_ {q} \pi \ln_ {q} \mu . \tag {5}
$$
Pseudo-additivity complicates obtaining convergence results for Eq. (1) with  $q$ -logarithm regularizers, since the techniques used for Shannon entropy and KL divergence are generally not applicable to their  $\ln_q$  counterparts. Moreover, deriving the optimal policy may be nontrivial. Convergence results have only been established for Tsallis entropy [Lee et al., 2018, Chow et al., 2018].
We know that Eq. (1) with  $\Omega (\pi) = D_{KL}^{q}(\pi \| \mu)$ , for any  $\mu$ , converges for  $q$  that make  $D_{KL}^{q}(\pi \| \mu)$  strictly convex [Geist et al., 2019]. When  $q = 2$ , it is strongly convex, and so also strictly convex, guaranteeing convergence.
Theorem 3. The regularized recursion Eq. (1) with  $\Omega (\pi) = D_{KL}^{q}(\pi \| \cdot)$  when  $q = 2$  converges to the unique regularized optimal policy.
Proof. See Appendix C. It simply involves proving that this regularizer is strongly convex.
3.3 TKL Regularized Policies Do More Than Averaging
We next show that the optimal regularized policy under Tsallis KL regularization does more than uniform averaging. It can be seen as performing a weighted average where the degree of weighting is controlled by  $q$ . Consider the recursion
$$
\left\{ \begin{array}{l} \pi_ {k + 1} = \arg \max  _ {\pi} \left\langle \pi , Q _ {k} - D _ {K L} ^ {q} \left(\pi \| \pi_ {k}\right) \right\rangle , \\ Q _ {k + 1} = r + \gamma P \left\langle \pi_ {k + 1}, Q _ {k} - D _ {K L} ^ {q} \left(\pi_ {k + 1} \| \pi_ {k}\right) \right\rangle , \end{array} \right. \tag {6}
$$
where we dropped the regularization coefficient  $\tau$  for convenience.
Theorem 4. The greedy policy  $\pi_{k + 1}$  in Equation (6) satisfies
$$
\pi_ {k + 1} \propto \left(\exp_ {q} Q _ {1} \dots \exp_ {q} Q _ {k}\right) = \left[ \exp_ {q} \left(\sum_ {j = 1} ^ {k} Q _ {j}\right) + \sum_ {j = 2} ^ {q - 1} (q - 1) ^ {j} \sum_ {i _ {1} = 1 <   \dots <   i _ {j}} ^ {k} Q _ {i _ {1}} \dots Q _ {i _ {j}} \right] ^ {\frac {1}{q - 1}}. \tag {7}
$$
When  $q = 1$ , Eq. (6) reduces to KL regularized recursion and hence Eq. (7) reduces to the KL-regularized policy. When  $q = 2$ , Eq. (7) becomes:
$$
\exp_{2}Q_{1}\dots \exp_{2}Q_{k} = \exp_{2}\left(\sum_{j = 1}^{k}Q_{j}\right) + \sum_{\substack{j = 2\\ i_{1} = 1 <   \dots <  i_{j}}}^{k}Q_{i_{1}}\dots Q_{i_{j}}.
$$
i.e., Tsallis KL regularized policies average over the history of value estimates as well as computing the interaction between them  $\sum_{j=2}^{k}\sum_{i_1 < \ldots < i_j}^{k}Q_{i_1}\ldots Q_{i_j}$ .
Proof. See Appendix D for the full proof. The proof comprises two parts: the first part shows  $\pi_{k + 1}\propto \exp_qQ_1\ldots \exp_qQ_k$ , and the second part establishes the more-than-averaging property by two-point equation [Yamano, 2002] and the  $2 - q$  duality [Naudts, 2002, Suyari and Tsukada, 2005] to conclude  $\left(\exp_qx\cdot \exp_qy\right)^{q - 1} = \exp_q(x + y)^{q - 1} + (q - 1)^2 xy.$
The form of this policy is harder to intuit, but we can try to understand each component. The first component actually corresponds to a weighted averaging by the property of the  $\exp_q$ :
$$
\exp_ {q} \left(\sum_ {i = 1} ^ {k} Q _ {i}\right) = \exp_ {q} Q _ {1} \exp_ {q} \left(\frac {Q _ {2}}{1 + (1 - q) Q _ {1}}\right) \dots \exp_ {q} \left(\frac {Q _ {k}}{1 + (1 - q) \sum_ {i = 1} ^ {k - 1} Q _ {i}}\right). \tag {8}
$$
Eq. (8) is a possible way to expand the summation: the left-hand side of the equation is what one might expect from conventional KL regularization; while the right-hand side shows a weighted scheme such that any estimate  $Q_{j}$  is weighted by the summation of estimates before  $Q_{j}$  times  $1 - q$  (Note that we can exchange 1 and  $q$ , see Appendix A). Weighting down numerator by the sum of components in the denominator has been analyzed before in the literature of weighted average by robust divergences, e.g., the  $\gamma$ -divergence [Futami et al., 2018, Table 1]. Therefore, we conjecture this functional form helps weighting down the magnitude of excessively large  $Q_{k}$ , which can also be controlled by choosing  $q$ . In fact, obtaining a weighted average has been an important topic in RL, where many proposed heuristics coincide with weighted averaging [Grau-Moya et al., 2019, Haarnoja et al., 2018, Kitamura et al., 2021].
Now let us consider the second term with  $q = 2$ , therefore the leading  $(q - 1)^j$  vanishes. The action-value cross-product term can be intuitively understood as further increasing the probability for any actions that have had consistently larger values across iterations. This observation agrees with the mode-covering property of Tsallis KL. However, there is no concrete evidence yet how the average inside  $q$ -exponential and the cross-product action values may work jointly to benefit the policy, and their benefits may depend on the task and environments, requiring further categorization and discussion. Empirically, we find that the nonlinearity of Tsallis KL policies bring superior performance to the uniform averaging KL policies on the testbed considered.
4 A Practical Algorithm for Tsallis KL Regularization
In this section we provide a practical algorithm for implementing Tsallis regularization. We first explain why this is not straightforward to simply implement KL-regularized value iteration, and how Munchausen Value Iteration (MVI) overcomes this issue with a clever implicit regularization trick. We then extend this algorithm to  $q > 1$  using a similar approach, though now with some approximation due once again to the difficulties of pseudo-additivity.
4.1 Implicit Regularization With MVI