SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Even for the standard KL, it is difficult to implement KL-regularized value iteration with function approximation. The difficulty arises from the fact that we cannot exactly obtain  $\pi_{k + 1}\propto \pi_k\exp (Q_k)$ . This policy might not be representable by our function approximator. For  $q = 1$ , one needs to store all past  $Q_{k}$  which is computationally infeasible.
An alternative direction has been to construct a different value function iteration scheme, which is equivalent to the original KL regularized value iteration [Azar et al., 2012, Kozuno et al., 2019]. A recent method of this family is Munchausen VI (MVI) [Vieillard et al., 2020b]. MVI implicitly enforces KL regularization using the recursion
$$
\left\{\pi_ {k + 1} = \arg \max  _ {\pi} \left\langle \pi , Q _ {k} - \tau \ln \pi \right\rangle \right. \tag {9}
$$
$$
\left\{Q _ {k + 1} = r + \alpha \tau \ln \pi_ {k + 1} + \gamma P \left\langle \pi_ {k + 1}, Q _ {k} - \tau \ln \pi_ {k + 1} \right\rangle \right.
$$
We see that Eq. (9) is Eq. (1) with  $\Omega (\pi) = -\mathcal{H}(\pi)$  (blue) plus an additional red Munchausen term, with coefficient  $\alpha$ . Vieillard et al. [2020b] showed that implicit KL regularization was performed
Figure 3:  $\mathrm{MVI}(q)$  on CartPole-v1 for  $q = 2,3,4,5$ , averaged over 50 seeds, with  $\tau = 0.03$ ,  $\alpha = 0.9$ . (Left) The difference between the proposed action gap  $Q_{k} - \mathcal{M}_{q,\tau}Q_{k}$  and the general Munchausen term  $\ln_q\pi_{k + 1}$  converges to a constant. (Right) The residual  $R_{q}(\pi_{k + 1},\pi_{k})$  becomes larger as  $q$  increases. For  $q = 2$ , it remains negligible throughout the learning.
under the hood, even though we still have tractable  $\pi_{k + 1}\propto \exp \left(\tau^{-1}Q_k\right)$
$$
\begin{array}{l} Q _ {k + 1} = r + \alpha \tau \ln \pi_ {k + 1} + \gamma P \left\langle \pi_ {k + 1}, Q _ {k} - \tau \ln \pi_ {k + 1} \right\rangle \Leftrightarrow Q _ {k + 1} - \alpha \tau \ln \pi_ {k + 1} = \\ r + \gamma P \left(\left\langle \pi_ {k + 1}, Q _ {k} - \alpha \tau \ln \pi_ {k} \right\rangle - \left\langle \pi_ {k + 1}, \alpha \tau \left(\ln \pi_ {k + 1} - \ln \pi_ {k}\right) - (1 - \alpha) \tau \ln \pi_ {k + 1} \right\rangle\right) \\ \Leftrightarrow Q _ {k + 1} ^ {\prime} = r + \gamma P \left(\left\langle \pi_ {k + 1}, Q _ {k} ^ {\prime} \right\rangle - \alpha \tau D _ {K L} \left(\pi_ {k + 1} \| \pi_ {k}\right) + (1 - \alpha) \tau \mathcal {H} \left(\pi_ {k + 1}\right)\right) \tag {10} \\ \end{array}
$$
where  $Q_{k + 1}^{\prime}\coloneqq Q_{k + 1} - \alpha \tau \ln \pi_{k + 1}$  is the generalized action value function.
The implementation of this idea uses the fact that  $\alpha \tau \ln \pi_{k + 1} = \alpha (Q_k - \mathcal{M}_\tau Q_k)$ , where  $\mathcal{M}_{\tau}Q_{k}\coloneqq \frac{1}{Z_{k}}\left\langle \exp \left(\tau^{-1}Q_{k}\right),Q_{k}\right\rangle ,Z_{k} = \left\langle \mathbf{1},\exp \left(\tau^{-1}Q_{k}\right)\right\rangle$  is the Boltzmann softmax operator. In the original work, computing this advantage term was found to be more stable than directly using the log of the policy. In our extension, we use the same form.
4.2 MVI(q) For General  $q$
The  $\mathrm{MVI}(q)$  algorithm is a simple extension of MVI: it replaces the standard exponential in the definition of the advantage with the  $q$ -exponential. We can express this action gap as  $Q_{k} - \mathcal{M}_{q,\tau}Q_{k}$ , where  $\mathcal{M}_{q,\tau}Q_{k} = \left\langle \exp_{q}\left(\frac{Q_{k}}{\tau} -\psi_{q}\left(\frac{Q_{k}}{\tau}\right)\right),Q_{k}\right\rangle$ . When  $q = 1$ , it recovers  $Q_{k} - \mathcal{M}_{\tau}Q_{k}$ . We summarize this  $\mathrm{MVI}(q)$  algorithm in Algorithm B in the Appendix. When  $q = 1$ , we recover MVI. For  $q = \infty$ , we get that  $\mathcal{M}_{\infty ,\tau}Q_k$  is  $\max_a Q_k(s,a)$  no regularization and we recover advantage learning [Baird and Moore, 1999]. Similar to the original MVI algorithm,  $\mathrm{MVI}(q)$  enjoys tractable policy expression with  $\pi_{k + 1}\propto \exp_q\left(\tau^{-1}Q_k\right)$ .
Unlike MVI, however,  $\mathrm{MVI}(q)$  no longer exactly implements the implicit regularization shown in Eq. (10). Below, we go through a similar derivation as MVI, show why there is an approximation and motivate why the above advantage term is a reasonable approximation. In addition to this reasoning, our primary motivation for this extension of MVI to use  $q > 1$  was to inherit the same simple form as MVI as well as because empirically we found it to be effective.
Let us similarly define a generalized action value function  $Q_{k + 1}^{\prime} = Q_{k + 1} - \alpha \tau \ln_q\pi_{k + 1}$ . Using the relationship  $\ln_q\pi_k = \ln_q\frac{\pi_k}{\pi_{k + 1}} -\ln_q\frac{1}{\pi_{k + 1}} -(1 - q)\ln_q\pi_k\ln_q\frac{1}{\pi_{k + 1}}$ , we get
$$
\begin{array}{l} Q _ {k + 1} - \alpha \tau \ln_ {q} \pi_ {k + 1} = r + \gamma P \left\langle \pi_ {k + 1}, Q _ {k} + \alpha \tau \ln_ {q} \pi_ {k} - \alpha \tau \ln_ {q} \pi_ {k} + \tau S _ {q} (\pi_ {k + 1}) \right\rangle \\ \Leftrightarrow Q _ {k + 1} ^ {\prime} = r + \gamma P \left\langle \pi_ {k + 1}, Q _ {k} ^ {\prime} + \tau S _ {q} \left(\pi_ {k + 1}\right) \right\rangle + \\ \gamma P \left\langle \pi_ {k + 1}, \alpha \tau \left(\ln_ {q} \frac {\pi_ {k}}{\pi_ {k + 1}} - \ln_ {q} \frac {1}{\pi_ {k + 1}} - (1 - q) \ln_ {q} \frac {1}{\pi_ {k + 1}} \ln_ {q} \pi_ {k}\right) \right\rangle \tag {11} \\ = r + \gamma P \left\langle \pi_ {k + 1}, Q _ {k} ^ {\prime} + (1 - \alpha) \tau S _ {q} \left(\pi_ {k + 1}\right) \right\rangle - \gamma P \left\langle \pi_ {k + 1}, \alpha \tau D _ {K L} ^ {q} \left(\pi_ {k + 1} \right| | \pi_ {k}\right) - \alpha \tau R _ {q} \left(\pi_ {k + 1}, \pi_ {k}\right) \rangle \\ \end{array}
$$
$^2$  Using  $\mathcal{M}_{\tau}Q$  is equivalent to the log-sum-exp operator up to a constant shift [Azar et al., 2012].
Figure 4: Learning curves of  $\mathrm{MVI}(q)$  and M-VI on the selected Atari games, averaged over 3 independent runs, with ribbon denoting the standard error. On some environments  $\mathrm{MVI}(q)$  significantly improve upon M-VI. Quantitative improvements over M-VI and Tsallis-VI are shown in Figures 5.
where we leveraged the fact that  $-\alpha \tau \left\langle \pi_{k+1}, \ln_q \frac{1}{\pi_{k+1}} \right\rangle = -\alpha \tau S_q(\pi_{k+1})$  and defined the residual term  $R_q(\pi_{k+1}, \pi_k) := (1-q) \ln_q \frac{1}{\pi_{k+1}} \ln_q \pi_k$ . When  $q = 2$ , it is expected that the residual term remains negligible, but can become larger as  $q$  increases. We visualize the trend of the residual  $R_q(\pi_{k+1}, \pi_k)$  for  $q = 2, 3, 4, 5$  on the CartPole-v1 environment [Brockman et al., 2016] in Figure 3. Learning consists of  $2.5 \times 10^5$  steps, evaluated every 2500 steps (one iteration), averaged over 50 independent runs. It is visible that the magnitude of residual jumps from  $q = 4$  to 5, while  $q = 2$  remains negligible throughout.
A reasonable approximation, therefore, is to use  $\ln_q\pi_{k + 1}$  and omit this residual term. Even this approximation, however, has an issue. When the actions are in the support,  $\ln_q$  is the unique inverse function of  $\exp_q$  and  $\ln_q\pi_{k + 1}$  yields  $\frac{Q_k}{\tau} -\psi_q\left(\frac{Q_k}{\tau}\right)$ . However, for actions outside the support, we cannot get the inverse, because many inputs to  $\exp_q$  can result in zero. We could still use  $\frac{Q_k}{\tau} -\psi_q\left(\frac{Q_k}{\tau}\right)$  as a sensible choice, and it appropriately does use negative values for the Munchausen term for these zero-probability actions. Empirically, however, we found this to be less effective than using the action gap.
Though the action gap is yet another approximation, there are clear similarities between using  $\frac{Q_k}{\tau} - \psi_q\left(\frac{Q_k}{\tau}\right)$  and the action gap  $Q_{k} - \mathcal{M}_{q,\tau}Q_{k}$ . The primary difference is in how the values are centered. We can see  $\psi_{q}$  as using a uniform average value of the actions in the support, as characterized in Theorem 2.  $\mathcal{M}_{q,\tau}Q_{k}$ , on the other hand, is a weighted average of action-values.
We plot the difference between  $Q_{k} - \mathcal{M}_{q,\tau}Q_{k}$  and  $\ln_q\pi_{k + 1}$  in Figure 3, again in Cartpole. The difference stabilizes around -0.5 for most of learning—in other words primarily just shifting by a constant—but in early learning  $\ln_q\pi_{k + 1}$  is larger, across all  $q$ . This difference in magnitude might explain why using the action gap results in more stable learning, though more investigation is needed to truly understand the difference. For the purposes of this initial work, we pursue the use of the action gap, both as itself a natural extension of the current implementation of MVI and from our own experiments suggesting improved stability with this form.
In this section we investigate the utility of  $\mathrm{MVI}(q)$  in the Atari 2600 benchmark [Bellemare et al., 2013]. We test whether this result holds in more challenging environments. Specifically, we compare to standard MVI  $(q = 1)$ , which was already shown to have competitive performance on Atari [Vieillard et al., 2020b]. We restrict our attention to  $q = 2$ , which was generally effective in other settings and also allows us to contrast to previous work [Lee et al., 2020] that only used entropy regularization with KL regularization. For  $\mathrm{MVI}(q = 2)$ , we take the exact same learning setup—hyperparameters and architecture—as  $\mathrm{MVI}(q = 1)$  and simply modify the term added to the VI update, as in Algorithm 1.
Figure 5: (Left) The percent improvement of  $\mathrm{MVI}(q)$  with  $q = 2$  over standard MVI (where  $q = 1$ ) on select Atari games. The improvement is computed by subtracting the scores from  $\mathrm{MVI}(q)$  and MVI and normalizing by the MVI scores. (Right) Improvement over Tsallis-VI on Atari environments, normalized with Tsallis-VI scores.
For the Atari games we implemented  $\mathrm{MVI}(q)$ , Tsallis-VI and M-VI based on the Quantile Regression DQN [Dabney et al., 2018]. We leverage the optimized Stable-Baselines3 architecture [Raffin et al., 2021] for best performance and average over 3 independent runs following [Vieillard et al., 2020b], though we run 50 million frames instead of 200 million. From Figure 4 it is visible that  $\mathrm{MVI}(q)$  is stable with no wild variance shown, suggesting 3 seeds might be sufficient. We perform grid searches for the algorithmic hyperparameters on two environments Asterix and Seaquest: the latter environment is regarded as a hard exploration environment.  $\mathrm{MVI}(q)\alpha : \{0.01,0.1,0.5,0.9,0.99\}$ ;  $\tau : \{0.01,0.1,1.0,10,100\}$ . Tsallis-VI  $\tau : \{0.01,0.1,1.0,10,100\}$ . For MVI we use the reported hyperparameters in [Vieillard et al., 2020b]. Hyperparameters can be seen from Table 2 and full results are provided in Appendix E.
5.1 Comparing  $\mathbf{MVI}(q)$  with  $q = 1$  to  $q = 2$
We provide the overall performance of MVI versus  $\mathrm{MVI}(q = 2)$  in Figure 5. Using  $q = 2$  provides a large improvement in about 5 games, about double the performance in the next 5 games, comparable performance in the next 7 games and then slightly worse performance in 3 games (PrivateEye, Chopper and Seaquest). Both PrivateEye and Seaquest are considered harder exploration games, which might explain this discrepancy. The Tsallis policy with  $q = 2$  reduces the support on actions, truncating some probabilities to zero. In general, with a higher  $q$ , the resulting policy is greedier, with  $q = \infty$  corresponding to exactly the greedy policy. It is possible that for these harder exploration games, the higher stochasticity in the softmax policy from MVI where  $q = 1$  promoted more exploration. A natural next step is to consider incorporating more directed exploration approaches, into  $\mathrm{MVI}(q = 2)$ , to benefit from the fact that lower-value actions are removed (avoiding taking poor actions) while exploring in a more directed way when needed.
We examine the learning curves for the games where  $\mathrm{MVI}(q)$  had the most significant improvement, in Figure 4. Particularly notable is how much more quickly  $\mathrm{MVI}(q)$  learned with  $q = 2$ , in addition to plateauing at a higher point. In Hero,  $\mathrm{MVI}(q)$  learned a stably across the runs, whereas standard MVI with  $q = 1$  clearly has some failures.
These results are quite surprising. The algorithms are otherwise very similar, with the seemingly small change of using Munchausen term  $Q_{k}(s,a) - \mathcal{M}_{q = 2,\tau}Q_{k}$  instead of  $Q_{k}(s,a) - \mathcal{M}_{q = 1,\tau}Q_{k}$  and using the  $q$ -logarithm and  $q$ -exponential for the entropy regularization and policy parameterization. Previous work using  $q = 2$  to get the sparsemax with entropy regularization generally harmed performance [Lee et al., 2018, 2020]. It seems that to get the benefits of the generalization to  $q > 1$ , the addition of the KL regularization might be key. We validate this in the next section.
5.2 The Importance of Including KL Regularization
In the policy evaluation step of Eq. (11), if we set  $\alpha = 0$  then we recover Tsallis-VI which uses regularization  $\Omega (\pi) = -S_q(\pi)$  in Eq. (1). In other words, we recover the algorithm that incorporates entropy regularization using the  $q$ -logarithm and the resulting sparsemax policy. Unlike MVI, Tsallis-
VI has not been comprehensively evaluated on Atari games, so we include results for the larger benchmark set comprising 35 Atari games. We plot the percentage improvement of  $\mathrm{MVI}(q)$  over Tsallis-VI in Figure 5.
The improvement from including the Munchausen term  $(\alpha > 0)$  is stark. For more than half of the games,  $\mathrm{MVI}(q)$  resulted in more than  $100\%$  improvement. For the remaining games it was comparable. For 10 games, it provided more than  $400\%$  improvement. Looking more specifically at which games there was notable improvement, it seems that exploration may again have played a role.  $\mathrm{MVI}(q)$  performs much better on Seaquest and PrivateEye. Both  $\mathrm{MVI}(q)$  and Tsallis-VI have policy parameterizations that truncate action support, setting probabilities to zero for some actions. The KL regularization term, however, likely slows this down. It is possible the Tsallis-VI is concentrating too quickly, resulting in insufficient exploration.
We investigated the use of the more general  $q$ -logarithm for entropy regularization and KL regularization, instead of the standard logarithm ( $q = 1$ ), which gave rise to Tsallis entropy and Tsallis KL regularization. We extended several results previously shown for  $q = 1$ , namely we proved (a) the form of the Tsallis policy can be expressed by  $q$ -exponential function; (b) Tsallis KL-regularized policies are weighted average of past action-values; (c) the convergence of value iteration for  $q = 2$  and (d) a relationship between adding a  $q$ -logarithm of policy to the action-value update, to provide implicit Tsallis KL regularization and entropy regularization, generalizing the original Munchausen Value Iteration (MVI). We used these results to propose a generalization to MVI, which we call MVI( $q$ ), because for  $q = 1$  we exactly recover MVI. We showed empirically that the generalization to  $q > 1$  can be beneficial, providing notable improvements in the Atari 2600 benchmark.
M. G. Azar, V. Gomez, and H. J. Kappen. Dynamic policy programming. Journal of Machine Learning Research, 13(1):3207-3245, 2012.
L. Baird and A. Moore. Gradient descent for general reinforcement learning. In Proceedings of the 1998 Conference on Advances in Neural Information Processing Systems II, page 968-974, 1999.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47(1):253-279, 2013. ISSN 1076-9757.
B. Belousov and J. Peters. Entropic regularization of markov decision processes. Entropy, 21(7), 2019.
M. Blondel, A. F. Martins, and V. Niculae. Learning with fenchel-young losses. Journal of Machine Learning Research, 21(35):1-69, 2020.
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
A. Chan, H. Silva, S. Lim, T. Kozuno, A. R. Mahmood, and M. White. Greedification operators for policy optimization: Investigating forward and reverse kl divergences. Journal of Machine Learning Research, 23(253):1-79, 2022.
G. Chen, Y. Peng, and M. Zhang. Effective exploration for deep reinforcement learning via bootstrapped q-ensembles under tsallis entropy regularization. arXiv:abs/1809.00403, 2018. URL http://arxiv.org/abs/1809.00403.
Y. Chow, O. Nachum, and M. Ghavamzadeh. Path consistency learning in Tsallis entropy regularized MDPs. In International Conference on Machine Learning, pages 979-988, 2018.
L. Condat. Fast projection onto the simplex and the 11 ball. Mathematical Programming, 158: 575-585, 2016.
O. Nachum and B. Dai. Reinforcement learning via fenchel-rockafellar duality. 2020. URL http://arxiv.org/abs/2001.01866.
O. Nachum, B. Dai, I. Kostrikov, Y. Chow, L. Li, and D. Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv:1912.02074, 2019.
J. Naudts. Deformed exponentials and logarithms in generalized thermostatistics. Physica A-statistical Mechanics and Its Applications, 316:323-334, 2002.
S. Nowozin, B. Cseke, and R. Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Advances in Neural Information Processing Systems, volume 29, pages 1-9, 2016.
J. Prehl, C. Essex, and K. H. Hoffmann. Tsallis relative entropy and anomalous diffusion. Entropy, 14(4):701-716, 2012.
A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22 (268):1-8, 2021.
I. Sason and S. Verdu. f-divergence inequalities. IEEE Transactions on Information Theory, 62: 5973-6006, 2016.
H. Suyari and M. Tsukada. Law of error in tsallis statistics. IEEE Transactions on Information Theory, 51(2):753-757, 2005.
H. Suyari, H. Matsuzoe, and A. M. Scarfone. Advantages of q-logarithm representation over q-exponential representation from the sense of scale and shift on nonlinear systems. The European Physical Journal Special Topics, 229(5):773-785, 2020.
C. Tsallis. Possible generalization of boltzmann-gibbs statistics. Journal of Statistical Physics, 52: 479-487, 1988.
C. Tsallis. Introduction to Nonextensive Statistical Mechanics: Approaching a Complex World. Springer New York, 2009. ISBN 9780387853581.
N. Vieillard, T. Kozuno, B. Scherrer, O. Pietquin, R. Munos, and M. Geist. Leverage the average: an analysis of regularization in rl. In Advances in Neural Information Processing Systems 33, pages 1-12, 2020a.
N. Vieillard, O. Pietquin, and M. Geist. Munchhausen reinforcement learning. In Advances in Neural Information Processing Systems 33, pages 1-11. 2020b.
N. Wan, D. Li, and N. Hovakimyan. f-divergence variational inference. In Advances in Neural Information Processing Systems, volume 33, pages 17370-17379, 2020.
D. Wang, H. Liu, and Q. Liu. Variational inference with tail-adaptive f-divergence. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 5742-5752, 2018.
T. Yamano. Some properties of q-logarithm and q-exponential functions in tsallis statistics. Physica A: Statistical Mechanics and its Applications, 305(3):486-496, 2002.
L. Yu, Y. Song, J. Song, and S. Ermon. Training deep energy-based models with f-divergence minimization. In Proceedings of the 37th International Conference on Machine Learning, ICML'20, pages 1-11, 2020.
R. Zhang, B. Dai, L. Li, and D. Schuurmans. Gendice: Generalized offline estimation of stationary values. In International Conference on Learning Representations, 2020.
A Basic facts of Tsallis KL divergence
We present some basic facts about  $q$ -logarithm and Tsallis KL divergence.
We begin by introducing the  $2 - q$  duality for Tsallis statistics. Recall that the  $q$ -logarithm and Tsallis entropy defined in the main paper are:
$$
\ln_ {q} x = \frac {x ^ {1 - q} - 1}{1 - q}, S _ {q} (x) = - \left\langle x ^ {q}, \ln_ {q} x \right\rangle .
$$
In the RL literature, another definition  $q^{*} = 2 - q$  is more often used [Lee et al., 2020]. This is called the  $2 - q$  duality [Naudts, 2002, Suyari and Tsukada, 2005], which refers to that the Tsallis entropy can be equivalently defined as:
$$
\ln_ {q ^ {*}} x = \frac {x ^ {q ^ {*} - 1} - 1}{q ^ {*} - 1}, \quad S _ {q ^ {*}} (x) = - \langle x, \ln_ {q ^ {*}} x \rangle ,
$$
By the duality we can show [Suyari and Tsukada, 2005, Eq.(12)]:
$$
S _ {q} (x) := - \left\langle x ^ {q}, \frac {x ^ {1 - q} - 1}{1 - q} \right\rangle = \frac {\left\langle \mathbf {1} , x ^ {q} \right\rangle - 1}{1 - q} = \frac {\left\langle \mathbf {1} , x ^ {q ^ {*}} \right\rangle - 1}{1 - q ^ {*}} = - \left\langle x, \frac {x ^ {q ^ {*} - 1} - 1}{q ^ {*} - 1} \right\rangle =: S _ {q ^ {*}} (x),
$$
i.e. the duality between logarithms  $\ln_{q^*}x$  and  $\ln_q x$  allows us to define Tsallis entropy by an alternative notation  $q^{*}$  that eventually reaches to the same functional form.
We now come to examine Tsallis KL divergence (or Tsallis relative entropy) defined in another form:  $D_{KL}^{q}(\pi \parallel \mu) = \left\langle \pi ,\ln_{q^{*}}\frac{\pi}{\mu}\right\rangle$  [Prehl et al., 2012]. In the main paper we used the definition  $D_{KL}^{q}(\pi \parallel \mu) = \left\langle \pi , - \ln_{q}\frac{\mu}{\pi}\right\rangle$  [Furuichi et al., 2004]. We show they are equivalent by the same logic:
$$
\left\langle \pi , - \ln_ {q} \frac {\mu}{\pi} \right\rangle = \left\langle \pi , - \frac {\left(\frac {\mu}{\pi}\right) ^ {1 - q} - 1}{1 - q} \right\rangle = \left\langle \pi , \frac {\left(\frac {\pi}{\mu}\right) ^ {q - 1} - 1}{q - 1} \right\rangle = \left\langle \pi , \ln_ {q ^ {*}} \frac {\pi}{\mu} \right\rangle . \tag {12}
$$
The equivalence allows us to work with whichever of  $\ln_q$  and  $\ln_{q^*}$  that makes the proof easier to work out the following useful properties of Tsallis KL divergence:
- Nonnegativity  $D_{KL}^{q}(\pi \parallel \mu) \geq 0$ : since the function  $-\ln_q\pi$  is convex, by Jensen's inequality
$$
\left\langle \pi , - \ln_ {q} \frac {\mu}{\pi} \right\rangle \geq - \ln_ {q} \left\langle \pi , \frac {\mu}{\pi} \right\rangle = 0,
$$
- Conditions of  $D_{KL}^{q}(\pi \| \mu) = 0$ : directly from the above, in Jensen's inequality the equality holds only when  $\frac{\mu}{\pi} = 1$  almost everywhere, i.e.  $D_{KL}^{q}(\pi \| \mu) = 0$  implies  $\mu = \pi$  almost everywhere.
- Conditions of  $D_{KL}^{q}(\pi \parallel \mu) = \infty$ : To better align with the standard KL divergence, let us work with  $\ln_{q^*}$ , following [Cover and Thomas, 2006], let us define
$$
0 \ln_ {q ^ {*}} \frac {0}{0} = 0, \quad 0 \ln_ {q ^ {*}} \frac {0}{\mu} = 0, \quad \pi \ln_ {q ^ {*}} \frac {\pi}{0} = \infty .
$$
We conclude that  $D_{KL}^{q}(\pi \parallel \mu) = \infty$  whenever  $\pi >0$  and  $\mu = 0$
- Bounded entropy  $\forall q, 0 \leq S_q(\pi) \leq \ln_q |\mathcal{A}|$ : let  $\mu = \frac{1}{|\mathcal{A}|}$ , by the nonnegativity of Tsallis KL divergence:
$$
\begin{array}{l} D _ {K L} ^ {q} (\pi \| \mu) = \left\langle \pi , - \ln_ {q} \frac {1}{(| \mathcal {A} | \cdot \pi)} \right\rangle = \left\langle \pi , \frac {(| \mathcal {A} | \cdot \pi) ^ {q - 1} - 1}{q - 1} \right\rangle \\ = | \mathcal {A} | ^ {q - 1} \left(\frac {\langle \mathbf {1} , \pi^ {q} \rangle - 1}{q - 1} - \frac {\frac {1}{| \mathcal {A} | ^ {q - 1}} - 1}{q - 1}\right) \geq 0. \\ \end{array}
$$
Notice that  $\frac{\langle\mathbf{1},\pi^q\rangle - 1}{q - 1} = \left\langle \pi^q,\frac{1 - \pi^{1 - q}}{1 - q}\right\rangle = \langle \pi ,\ln_q\pi \rangle = -S_q(\pi)$  and  $\frac{\frac{1}{|\mathcal{A}|q - 1} - 1}{q - 1} = \ln_q|\mathcal{A}|$ , we conclude
$$
S _ {q} (\pi) \leq \ln_ {q} | \mathcal {A} |.
$$
B Proof of Theorem 1 and 2
We structure this section as the following three parts: