Title: General Munchausen Reinforcement Learning with Tsallis Kullback-Leibler Divergence
Venue: NeurIPS 2023
Accepted: yes
Keywords: reinforcement learning

entropy regularization

Tsallis KL divergence
Abstract: Many policy optimization approaches in reinforcement learning incorporate a Kullback-Leilbler (KL) divergence to the previous policy, to prevent the policy from changing too quickly. This idea was initially proposed in a seminal paper on Conservative Policy Iteration, with approximations given by algorithms like TRPO and Munchausen Value Iteration (MVI). We continue this line of work by investigating a generalized KL divergence---called the Tsallis KL divergence. Tsallis KL defined by the $q$-logarithm is a strict generalization, as $q = 1$ corresponds to the standard KL divergence; $q > 1$ provides a range of new options. We characterize the types of policies learned under the Tsallis KL, and motivate when $q >1$ could be beneficial.  To obtain a practical algorithm that incorporates Tsallis KL regularization, we extend MVI, which is one of the simplest approaches to incorporate KL regularization. We show that this generalized MVI($q$) obtains significant improvements over the standard MVI($q = 1$) across 35 Atari games.

General Munchausen Reinforcement Learning with Tsallis Kullback-Leibler Divergence

Abstract

Many policy optimization approaches in reinforcement learning incorporate a Kullback-Leilbler (KL) divergence to the previous policy, to prevent the policy from changing too quickly. This idea was initially proposed in a seminal paper on Conservative Policy Iteration, with approximations given by algorithms like TRPO and Munchausen Value Iteration (MVI). We continue this line of work by investigating a generalized KL divergence-called the Tsallis KL divergence. Tsallis KL defined by the q -logarithm is a strict generalization, as q = 1 corresponds to the standard KL divergence; q > 1 provides a range of new options. We characterize the types of policies learned under the Tsallis KL, and motivate when q > 1 could be beneficial. To obtain a practical algorithm that incorporates Tsallis KL regularization, we extend MVI, which is one of the simplest approaches to incorporate KL regularization. We show that this generalized MVI( q ) obtains significant improvements over the standard MVI( q = 1 ) across 35 Atari games.

1 Introduction

There is ample theoretical evidence that it is useful to incorporate KL regularization into policy optimization in reinforcement learning. The most basic approach is to regularize towards a uniform policy, resulting in entropy regularization. More effective, however, is to regularize towards the previous policy. By choosing KL regularization between consecutively updated policies, the optimal policy becomes a softmax over a uniform average of the full history of action value estimates [Vieillard et al., 2020a]. This averaging smooths out noise, allowing for better theoretical results [Azar et al., 2012, Kozuno et al., 2019, Vieillard et al., 2020a, Kozuno et al., 2022].

Despite these theoretical benefits, there are some issues with using KL regularization in practice. It is well-known that the uniform average is susceptible to outliers; this issue is inherent to KL divergence [Futami et al., 2018]. In practice, heuristics such as assigning vanishing regularization coefficients to some estimates have been implemented widely to increase robustness and accelerate learning [Grau-Moya et al., 2019, Haarnoja et al., 2018, Kitamura et al., 2021]. However, theoretical guarantees no longer hold for those heuristics [Vieillard et al., 2020a, Kozuno et al., 2022]. A natural question is what alternatives we can consider to this KL divergence regularization, that allows us to overcome some of these disadvantages while maintaining the benefits associate with restricting aggressive policy changes and smoothing errors.

In this work, we explore one possible direction by generalizing to Tsallis KL divergences. Tsallis KL divergences were introduced for physics [Tsallis, 1988, 2009] using a simple idea: replacing the use of the logarithm with the deformed q -logarithm. The implications for policy optimization, however, are that we get quite a different form for the resulting policy. Tsallis entropy with q = 2 has actually already been considered for policy optimization [Chow et al., 2018, Lee et al., 2018], by replacing Shannon entropy with Tsallis entropy to maintain stochasticity in the policy. The resulting policies are called sparsemax policies, because they concentrate the probability on higher-valued actions and truncate the probability to zero for lower-valued actions. Intuitively, this should have the benefit of maintaining stochasticity, but only amongst the most promising actions, unlike the Boltzmann policy which maintains nonzero probability on all actions. Unfortunately, using only Tsallis entropy did not provide significant benefits, and in fact often performed worse than existing methods. We find, however, that using a Tsallis KL divergence to the previous policy does provide notable gains.

We first show how to incorporate Tsallis KL regularization into the standard value iteration updates, and prove that we maintain convergence under this generalization from KL regularization to Tsallis KL regularization. We then characterize the types of policies learned under Tsallis KL, highlighting that there is now a more complex relationship to past action-values than a simple uniform average. We then show how to extend Munchausen Value Iteration (MVI) [Vieillard et al., 2020b], to use Tsallis KL regularization, which we call MVI( q ). We use this naming convention to highlight that this is a strict generalization of MVI: by setting q = 1 , we exactly recover MVI. We then compare MVI( q = 2 ) with MVI (namely the standard choice where q = 1 ), and find that we obtain significant performance improvements in Atari.

Remark: There is a growing body of literature studying generalizations of KL divergence in RL [Nachum et al., 2019, Zhang et al., 2020]. Futami et al. [2018] discussed the inherent drawback of KL divergence in generative modeling and proposed to use β - and γ -divergence to allow for weighted average of sample contribution. These divergences fall under the category known as the f -divergence [Sason and Verdú, 2016], commonly used in other machine learning domains including generative modeling [Nowozin et al., 2016, Wan et al., 2020, Yu et al., 2020] and imitation learning [Ghasemipour et al., 2019, Ke et al., 2019]. In RL, Wang et al. [2018] discussed using tail adaptive f -divergence to enforce the mass-covering property. Belousov and Peters [2019] discussed the use of α -divergence. Tsallis KL divergence, however, has not yet been studied in RL.

2 Problem Setting

We focus on discrete-time discounted Markov Decision Processes (MDPs) expressed by the tuple ( S , A , d, P, r, γ ) , where S and A denote state space and finite action space, respectively. Let ∆( X ) denote the set of probability distributions over X . d ∈ ∆( S ) denotes the initial state distribution. P : S × A → ∆( S ) denotes the transition probability function, and r ( s, a ) defines the reward associated with that transition. γ ∈ (0 , 1) is the discount factor. A policy π : S → ∆( A ) is a mapping from the state space to distributions over actions. We define the action value function following policy π and starting from s 0 ∼ d ( · ) with action a 0 taken as Q π ( s, a ) = E π [ ∑ ∞ t =0 γ t r t | s 0 = s, a 0 = a ] . A standard approach to find the optimal value function Q ∗ is value iteration. To define the formulas for value iteration, it will be convenient to write the action value function as a matrix Q π ∈ R |S|×|A| . For notational convenience, we define the inner product for any two functions F 1 , F 2 ∈ R |S|×|A| over actions as ⟨ F 1 , F 2 ⟩ ∈ R |S| .

We are interested in the entropy-regularized MDPs where the recursion is augmented with Ω( π ) :

This modified recursion is guaranteed to converge if Ω is concave in π . For standard (Shannon) entropy regularization, we use Ω( π ) = -H ( π ) = ⟨ π, ln π ⟩ . The resulting optimal policy has π k +1 ∝ exp ( τ -1 Q k ) , where ∝ indicates proportional to up to a constant not depending on actions.

More generally, we can consider a broad class of regularizers known as f -divergences [Sason and Verdú, 2016]: Ω( π ) = D f ( π || µ ) := ⟨ µ, f ( π/µ ) ⟩ , where f is a convex function. For example, the KL divergence D KL ( π | | µ ) = ⟨ π, ln π -ln µ ⟩ can be recovered by f ( t ) = -ln t . In this work, when we say KL regularization, we mean the standard choice of setting µ = π k , the estimate from the previous update. Therefore, D KL serves as a penalty to penalize aggressive policy changes. The

2

0.4

3

π

三

0.3

2

三

=

bdxa

2

Entropy

0.2

q=

9-

q=

3

1

0.1

b

=

4

0.0

-10

0

1

2

3

0

3

-2

-1

0

1

2

0

2

3

4

L

π = N(2,1)

C

C

optimal policy in this case takes the form π k +1 ∝ π k exp ( τ -1 Q k ) . By induction, we can show this KL-regularized optimal policy π k +1 is a softmax over a uniform average over the history of action value estimates [Vieillard et al., 2020a]: π k +1 ∝ π k exp ( τ -1 Q k ) ∝ · · · ∝ exp ( τ -1 ∑ k j =1 Q j ) . Using KL regularization has been shown to be theoretically superior to entropy regularization in terms of error tolerance [Azar et al., 2012, Vieillard et al., 2020a, Kozuno et al., 2022, Chan et al., 2022].

The definitions of H ( · ) and D KL ( ·||· ) rely on the standard logarithm and both induce softmax policies as an exponential (inverse function) over (weighted) action-values [Hiriart-Urruty and Lemaréchal, 2004, Nachum and Dai, 2020]. Convergence properties of the resulting regularized algorithms have been well studied [Kozuno et al., 2019, Geist et al., 2019, Vieillard et al., 2020a]. In this paper, we investigate Tsallis entropy and Tsallis KL divergence as the regularizer, which generalize Shannon entropy and KL divergence respectively.

3 Generalizing to Tsallis Regularization

We can easily incorporate other regularizers in to the value iteration recursion, and maintain convergence as long as those regularizers are strongly convex in π . We characterize the types of policies that arise from using this regularizer, and prove the convergence of resulting regularized recursion.

3.1 Tsallis Entropy Regularization

Tsallis entropy was first proposed by Tsallis [1988] and is defined by the q -logarithm. The q -logarithm and its unique inverse function, the q -exponential, are defined as:

where [ · ] + := max {· , 0 } . We define ln 1 = ln , exp 1 = exp , as in the limit q → 1 , the formulas in Eq. (2) approach these functions. Tsallis entropy can be defined by S q ( π ) := p ⟨-π q , ln q π ⟩ , p ∈ R [Suyari and Tsukada, 2005]. We visualize the q -logarithm, q -exponential and Tsallis entropy for different q in Figure 1. As q gets larger, q -logarithm (and hence Tsallis entropy) becomes more flat and q -exponential more steep 1 . Note that exp q is only invertible for x > -1 1 -q .

Tsallis policies have a similar form to softmax, but using the q -exponential instead. Let us provide some intuition for these policies. When p = 1 2 , q = 2 , S 2 ( π ) = 1 2 ⟨ π, 1 -π ⟩ , the optimization problem arg max π ∈ ∆( A ) ⟨ π, Q ⟩ + S 2 ( π ) = arg min π ∈ ∆( A ) | | π -Q | | 2 2 is known to be the Euclidean projection onto the probability simplex. Its solution [ Q -ψ ] + is called the sparsemax [Martins and Astudillo, 2016, Lee et al., 2018] and has sparse support [Duchi et al., 2008, Condat, 2016, Blondel et al., 2020]. ψ : S × A → S is the unique function satisfying 〈 1 , [ Q -ψ ] + 〉 = 1 .

As our first result, we unify the Tsallis entropy regularized policies for all q ∈ R + with the q -exponential, and show that q and τ are interchangeable for controlling the truncation.

=1.0

q=2

q=1

Boltzmann policy q = 50

q=3

q=4

0.5

0.0

0

1

2

3

4

L

Theorem 1. Let Ω( π ) = -S q ( π ) in Eq. (1). Then the regularized optimal policies can be expressed:

where ψ q = ˜ ψ q + 1 1 -q . Additionally, for an arbitrary ( q, τ ) pair with q > 1 , the same truncation effect (support) can be achieved using ( q = 2 , τ q -1 ) .

Theorem 1 characterizes the role played by q : controlling the degree of truncation. We show the truncation effect when q = 2 and q = 50 in Figure 2, confirming that Tsallis policies tend to truncate more as q gets larger. The theorem also highlights that we can set q = 2 and still get more or less truncation using different τ , helping to explain why in our experiments q = 2 is a generally effective choice.

̸

Unfortunately, the threshold ˜ ψ q (and ψ q ) does not have a closed-form solution for q = 1 , 2 , ∞ . Note that q = 1 corresponds to Shannon entropy and q = ∞ to no regularization. However, we can resort to Taylor's expansion to obtain approximate sparsemax policies .

̸

Theorem 2. For q = 1 , ∞ , we can obtain approximate threshold ˆ ψ q ≈ ψ q using Taylor's expansion, and therefore an approximate policy:

K ( s ) is the set of highest-valued actions, satisfying the relation 1 + i Q ( s,a ( i ) ) τ > ∑ i j =1 Q ( s,a ( j ) ) τ , where a ( j ) indicates the action with j th largest action value. The sparsemax policy sets the probabilities of lowest-valued actions to zero: π ( a ( i ) | s ) = 0 , i = z + 1 , . . . , |A| where Q ( s, a ( z ) ) > τ -1 ˆ ψ q ( Q ( s, · ) τ ) > Q ( s, a ( z +1) ) . When q = 2 , ˆ ψ q recovers ψ q .

Proof.

Lee et al. [2020] also used exp q to represent policies but they consider the continuous action setting and do not give any computable threshold. By contrast, Theorem 2 presents an easily computable ˆ ψ q for all q / ∈ { 1 , ∞} .

3.2 Tsallis KL Regularization and Convergence Results

The Tsallis KL divergence is defined as D q KL ( π | | µ ) := 〈 π, -ln q µ π 〉 [Furuichi et al., 2004]. It is a member of f -divergence and can be recovered by choosing f ( t ) = -ln q t . As a divergence penalty,

it is required that q > 0 since f ( t ) should be convex. We further assume that q > 1 to align with standard divergences; i.e. penalize large value of π µ , since for 0 < q < 1 the regularization would penalize µ π instead. In practice, we find that 0 < q < 1 tend to perform poorly. In contrast to KL, Tsallis KL is more mass-covering ; i.e. its value is proportional to the q -th power of the ratio π µ . When q is big, large values of π µ are strongly penalized [Wang et al., 2018]. This behavior of Tsallis KL divergence can also be found in other well-known divergences: the α -divergence [Wang et al., 2018, Belousov and Peters, 2019] coincides with Tsallis KL when α = 2 ; Rényi's divergence also penalizes large policy ratio by raising it to the power q , but inside the logarithm, which is therefore an additive extension of KL [Li and Turner, 2016]. In the limit of q → 1 , Tsallis entropy recovers Shannon entropy and the Tsallis KL divergence recovers the KL divergence. We plot the Tsallis KL divergence behavior in Figure 2.

Now let us turn to formalizing when value iteration under Tsallis regularization converges. The q -logarithm has the following properties: Convexity: ln q π is convex for q ≤ 0 , concave for q > 0 . When q = 0 , both ln q , exp q become linear. Monotonicity: ln q π is monotonically increasing with respect to π . These two properties can be simply verified by checking the first and second order derivative. We prove in Appendix A the following similarity between Shannon entropy (reps. KL) and Tsallis entropy (resp. Tsallis KL). Bounded entropy : we have 0 ≤ H ( π ) ≤ ln |A| ; and ∀ q, 0 ≤ S q ( π ) ≤ ln q |A| . Generalized KL property : ∀ q , D q KL ( π | | µ ) ≥ 0 . D q KL ( π | | µ ) = 0 if and only if π = µ almost everywhere, and D q KL ( π | | µ ) →∞ whenever π ( a | s ) > 0 and µ ( a | s ) = 0 .

However, despite their similarity, a crucial difference is that ln q is non-extensive, which means it is not additive [Tsallis, 1988]. In fact, ln q is only pseudo-additive :

Pseudo-additivity complicates obtaining convergence results for Eq. (1) with q -logarithm regularizers, since the techniques used for Shannon entropy and KL divergence are generally not applicable to their ln q counterparts. Moreover, deriving the optimal policy may be nontrivial. Convergence results have only been established for Tsallis entropy [Lee et al., 2018, Chow et al., 2018].

We know that Eq. (1) with Ω( π ) = D q KL ( π | | µ ) , for any µ , converges for q that make D q KL ( π | | µ ) strictly convex [Geist et al., 2019]. When q = 2 , it is strongly convex, and so also strictly convex, guaranteeing convergence.

Theorem 3. The regularized recursion Eq. (1) with Ω( π ) = D q KL ( π | | · ) when q = 2 converges to the unique regularized optimal policy.

Proof. See Appendix C. It simply involves proving that this regularizer is strongly convex.

3.3 TKL Regularized Policies Do More Than Averaging

We next show that the optimal regularized policy under Tsallis KL regularization does more than uniform averaging. It can be seen as performing a weighted average where the degree of weighting is controlled by q . Consider the recursion

where we dropped the regularization coefficient τ for convenience.

Theorem 4. The greedy policy π k +1 in Equation (6) satisfies

When q = 1 , Eq. (6) reduces to KL regularized recursion and hence Eq. (7) reduces to the KLregularized policy. When q =2 , Eq. (7) becomes:

i.e., Tsallis KL regularized policies average over the history of value estimates as well as computing the interaction between them ∑ k j =2 ∑ k i 1 < ··· <i j Q i 1 . . . Q i j .

Proof. See Appendix D for the full proof. The proof comprises two parts: the first part shows π k +1 ∝ exp q Q 1 . . . exp q Q k , and the second part establishes the more-than-averaging property by two-point equation [Yamano, 2002] and the 2 -q duality [Naudts, 2002, Suyari and Tsukada, 2005] to conclude ( exp q x · exp q y ) q -1 = exp q ( x + y ) q -1 +( q -1) 2 xy .

The form of this policy is harder to intuit, but we can try to understand each component. The first component actually corresponds to a weighted averaging by the property of the exp q :

Eq. (8) is a possible way to expand the summation: the left-hand side of the equation is what one might expect from conventional KL regularization; while the right-hand side shows a weighted scheme such that any estimate Q j is weighted by the summation of estimates before Q j times 1 -q (Note that we can exchange 1 and q , see Appendix A). Weighting down numerator by the sum of components in the demoninator has been analyzed before in the literature of weighted average by robust divergences, e.g., the γ -divergence [Futami et al., 2018, Table 1]. Therefore, we conjecture this functional form helps weighting down the magnitude of excessively large Q k , which can also be controlled by choosing q . In fact, obtaining a weighted average has been an important topic in RL, where many proposed heuristics coincide with weighted averaging [Grau-Moya et al., 2019, Haarnoja et al., 2018, Kitamura et al., 2021].

Now let us consider the second term with q = 2 , therefore the leading ( q -1) j vanishes. The action-value cross-product term can be intuitively understood as further increasing the probability for any actions that have had consistently larger values across iterations. This observation agrees with the mode-covering property of Tsallis KL. However, there is no concrete evidence yet how the average inside q -exponential and the cross-product action values may work jointly to benefit the policy, and their benefits may depend on the task and environments, requiring further categorization and discussion. Empirically, we find that the nonlinearity of Tsallis KL policies bring superior performance to the uniform averaging KL policies on the testbed considered.

4 A Practical Algorithm for Tsallis KL Regularization

In this section we provide a practical algorithm for implementing Tsallis regularization. We first explain why this is not straightforward to simply implement KL-regularized value iteration, and how Munchausen Value Iteration (MVI) overcomes this issue with a clever implicit regularization trick. We then extend this algorithm to q > 1 using a similar approach, though now with some approximation due once again to the difficulties of pseudo-additivity.

4.1 Implicit Regularization With MVI

Even for the standard KL, it is difficult to implement KL-regularized value iteration with function approximation. The difficulty arises from the fact that we cannot exactly obtain π k +1 ∝ π k exp( Q k ) . This policy might not be representable by our function approximator. For q = 1 , one needs to store all past Q k which is computationally infeasible.

An alternative direction has been to construct a different value function iteration scheme, which is equivalent to the original KL regularized value iteration [Azar et al., 2012, Kozuno et al., 2019]. A recent method of this family is Munchausen VI (MVI) [Vieillard et al., 2020b]. MVI implicitly enforces KL regularization using the recursion

We see that Eq. (9) is Eq. (1) with Ω( π ) = -H ( π ) (blue) plus an additional red Munchausen term , with coefficient α . Vieillard et al. [2020b] showed that implicit KL regularization was performed

Cartpole

Cartpole

0.0

2.0

+

0.5

1.5

Rg(Tk+1,Tk)

Ow

1.0

1.0

1.5

q =2.0

0.5

q=3.0

q = 4.0

q=5.0

0.0

2.0

20

40

60

80

100

0

20

40

60

80

100

Iteration

Iteration

under the hood, even though we still have tractable π k +1 ∝ exp ( τ -1 Q k ) :

where Q ′ k +1 := Q k +1 -ατ ln π k +1 is the generalized action value function.

The implementation of this idea uses the fact that ατ ln π k +1 = α ( Q k -M τ Q k ) , where M τ Q k := 1 Z k 〈 exp ( τ -1 Q k ) , Q k 〉 , Z k = 〈 1 , exp ( τ -1 Q k )〉 is the Boltzmann softmax operator. 2 In the original work, computing this advantage term was found to be more stable than directly using the log of the policy. In our extension, we use the same form.

4.2 MVI( q ) For General q

The MVI(q) algorithm is a simple extension of MVI: it replaces the standard exponential in the definition of the advantage with the q -exponential. We can express this action gap as Q k -M q,τ Q k , where M q,τ Q k = 〈 exp q ( Q k τ -ψ q ( Q k τ )) , Q k 〉 . When q = 1 , it recovers Q k - M τ Q k . We summarize this MVI( q ) algorithm in Algorithm B in the Appendix. When q = 1 , we recover MVI. For q = ∞ , we get that M ∞ ,τ Q k is max a Q k ( s, a ) -no regularization-and we recover advantage learning [Baird and Moore, 1999]. Similar to the original MVI algorithm, MVI( q ) enjoys tractable policy expression with π k +1 ∝ exp q ( τ -1 Q k ) .

Unlike MVI, however, MVI( q ) no longer exactly implements the implicit regularization shown in Eq. (10). Below, we go through a similar derivation as MVI, show why there is an approximation and motivate why the above advantage term is a reasonable approximation. In addition to this reasoning, our primary motivation for this extension of MVI to use q > 1 was to inherit the same simple form as MVI as well as because empirically we found it to be effective.

Let us similarly define a generalized action value function Q ′ k +1 = Q k +1 -ατ ln q π k +1 . Using the relationship ln q π k = ln q π k π k +1 -ln q 1 π k +1 -(1 -q ) ln q π k ln q 1 π k +1 , we get

Amidar

1e3

Assault

1e4

Asterix

1e3

BeamRider

2500

MVI(q)

10

25

15

M-VI

1500

6

15

10

Return

500

5

5

Average

Enduro

1e3

Hero

1e3

MsPacman

1e3

SpaceInvaders

4000

20

5

15

2000

10

3

10

5

0

0

0

1

2

Iteration

4

1e7

5

0

1

2

Iteration

4

1e7

5

0

1

2

Iteration

3

4

1e7

5

0

1

2

Iteration

3

4

1e7

5

where we leveraged the fact that -ατ 〈 π k +1 , ln q 1 π k +1 〉 = -ατS q ( π k +1 ) and defined the residual term R q ( π k +1 , π k ) := (1 -q ) ln q 1 π k +1 ln q π k . When q = 2 , it is expected that the residual term remains negligible, but can become larger as q increases. We visualize the trend of the residual R q ( π k +1 , π k ) for q = 2 , 3 , 4 , 5 on the CartPole-v1 environment [Brockman et al., 2016] in Figure 3. Learning consists of 2 . 5 × 10 5 steps, evaluated every 2500 steps (one iteration), averaged over 50 independent runs. It is visible that the magnitude of residual jumps from q = 4 to 5 , while q = 2 remains negligible throughout.

A reasonable approximation, therefore, is to use ln q π k +1 and omit this residual term. Even this approximation, however, has an issue. When the actions are in the support, ln q is the unique inverse function of exp q and ln q π k +1 yields Q k τ -ψ q ( Q k τ ) . However, for actions outside the support, we cannot get the inverse, because many inputs to exp q can result in zero. We could still use Q k τ -ψ q ( Q k τ ) as a sensible choice, and it appropriately does use negative values for the Munchausen term for these zero-probability actions. Empirically, however, we found this to be less effective than using the action gap.

Though the action gap is yet another approximation, there are clear similarities between using Q k τ -ψ q ( Q k τ ) and the action gap Q k - M q,τ Q k . The primary difference is in how the values are centered. We can see ψ q as using a uniform average value of the actions in the support, as characterized in Theorem 2. M q,τ Q k , on the other hand, is a weighted average of action-values.

We plot the difference between Q k - M q,τ Q k and ln q π k +1 in Figure 3, again in Cartpole. The difference stabilizes around -0.5 for most of learning-in other words primarily just shifting by a constant-but in early learning ln q π k +1 is larger, across all q . This difference in magnitude might explain why using the action gap results in more stable learning, though more investigation is needed to truly understand the difference. For the purposes of this initial work, we pursue the use of the action gap, both as itself a natural extension of the current implementation of MVI and from our own experiments suggesting improved stability with this form.

5 Experiments

In this section we investigate the utility of MVI( q ) in the Atari 2600 benchmark [Bellemare et al., 2013]. We test whether this result holds in more challenging environments. Specifically, we compare to standard MVI ( q = 1 ), which was already shown to have competitive performance on Atari [Vieillard et al., 2020b]. We restrict our attention to q = 2 , which was generally effective in other settings and also allows us to contrast to previous work [Lee et al., 2020] that only used entropy regularization with KL regularization. For MVI( q = 2 ), we take the exact same learning setuphyperparameters and architecture-as MVI( q = 1 ) and simply modify the term added to the VI update, as in Algorithm 1.

MVI(q) improvement over MVI

MVI(q) improvement over Tsallis-VI

1000

600

(%)

(%)

Improvement (

400

Improvement (

600

200

200

0

Amidar

Asterix

Assault

Enduro

Space.

Hero

BeamRider

KungFu.

UpNDown

Riverraid

MsPacman

Breakout

Zaxxon

Robotank

Pong

Pitfall

PrivateEye

Chopper.

Seaquest

RoadRun.

Amidar

Assault

Zaxxon

sanbeass

PrivateEye

Alien

apare

Kangaroo

Frostbite

KungFu.

MsPacman

Tutankham

Boxing

s

Enduro

Pong

Gopher

BeamRider

BankHeist

NENDOWn

Auat's

Jamesbond

For the Atari games we implemented MVI( q ), Tsallis-VI and M-VI based on the Quantile Regression DQN [Dabney et al., 2018]. We leverage the optimized Stable-Baselines3 architecture [Raffin et al., 2021] for best performance and average over 3 independent runs following [Vieillard et al., 2020b], though we run 50 million frames instead of 200 million. From Figure 4 it is visible that MVI ( q ) is stable with no wild variance shown, suggesting 3 seeds might be sufficient. We perform grid searches for the algorithmic hyperparameters on two environments Asterix and Seaquest : the latter environment is regarded as a hard exploration environment. MVI ( q ) α : { 0 . 01 , 0 . 1 , 0 . 5 , 0 . 9 , 0 . 99 } ; τ : { 0 . 01 , 0 . 1 , 1 . 0 , 10 , 100 } . Tsallis-VI τ : { 0 . 01 , 0 . 1 , 1 . 0 , 10 , 100 } . For MVI we use the reported hyperparameters in [Vieillard et al., 2020b]. Hyperparameters can be seen from Table 2 and full results are provided in Appendix E.

5.1 Comparing MVI( q ) with q = 1 to q = 2

We provide the overall performance of MVI versus MVI( q = 2 ) in Figure 5. Using q = 2 provides a large improvement in about 5 games, about double the performance in the next 5 games, comparable performance in the next 7 games and then slightly worse performance in 3 games ( PrivateEye , Chopper and Seaquest ). Both PrivateEye and Seaquest are considered harder exploration games, which might explain this discrepancy. The Tsallis policy with q = 2 reduces the support on actions, truncating some probabilities to zero. In general, with a higher q , the resulting policy is greedier, with q = ∞ corresponding to exactly the greedy policy. It is possible that for these harder exploration games, the higher stochasticity in the softmax policy from MVI whre q = 1 promoted more exploration. A natural next step is to consider incorporating more directed exploration approaches, into MVI( q = 2 ), to benefit from the fact that lower-value actions are removed (avoiding taking poor actions) while exploring in a more directed way when needed.

We examine the learning curves for the games where MVI( q ) had the most significant improvement, in Figure 4. Particularly notable is how much more quickly MVI( q ) learned with q = 2 , in addition to plateauing at a higher point. In Hero , MVI( q ) learned a stably across the runs, whereas standard MVI with q = 1 clearly has some failures.

These results are quite surprising. The algorithms are otherwise very similar, with the seemingly small change of using Munchausen term Q k ( s, a ) - M q =2 ,τ Q k instead of Q k ( s, a ) - M q =1 ,τ Q k and using the q -logarithm and q -exponential for the entropy regularization and policy parameterization. Previous work using q = 2 to get the sparsemax with entropy regularization generally harmed performance [Lee et al., 2018, 2020]. It seems that to get the benefits of the generalization to q > 1 , the addition of the KL regularization might be key. We validate this in the next section.

5.2 The Importance of Including KL Regularization

In the policy evaluation step of Eq. (11), if we set α = 0 then we recover Tsallis-VI which uses regularization Ω( π ) = -S q ( π ) in Eq. (1). In other words, we recover the algorithm that incorporates entropy regularization using the q -logarithm and the resulting sparsemax policy. Unlike MVI, Tsallis-

VI has not been comprehensively evaluated on Atari games, so we include results for the larger benchmark set comprising 35 Atari games. We plot the percentage improvement of MVI( q ) over Tsallis-VI in Figure 5.

The improvement from including the Munchausen term ( α > 0 ) is stark. For more than half of the games, MVI( q ) resulted in more than 100% improvement. For the remaining games it was comparable. For 10 games, it provided more than 400% improvement. Looking more specifically at which games there was notable improvement, it seems that exploration may again have played a role. MVI( q ) performs much better on Seaquest and PrivateEye . Both MVI( q ) and Tsallis-VI have policy parameterizations that truncate action support, setting probabilities to zero for some actions. The KL regularization term, however, likely slows this down. It is possible the Tsallis-VI is concentrating too quickly, resulting in insufficient exploration.

6 Conclusion and Discussion

We investigated the use of the more general q -logarithm for entropy regularization and KL regularization, instead of the standard logarithm ( q = 1 ), which gave rise to Tsallis entropy and Tsallis KL regularization. We extended several results previously shown for q = 1 , namely we proved (a) the form of the Tsallis policy can be expressed by q -exponential function; (b) Tsallis KL-regularized policies are weighted average of past action-values; (c) the convergence of value iteration for q = 2 and (d) a relationship between adding a q -logarithm of policy to the action-value update, to provide implicit Tsallis KL regularization and entropy regularization, generalizing the original Munchausen Value Iteration (MVI). We used these results to propose a generalization to MVI, which we call MVI( q ), because for q = 1 we exactly recover MVI. We showed empirically that the generalization to q > 1 can be beneficial, providing notable improvements in the Atari 2600 benchmark.

References

G. Azar, V. Gómez, and H. J. Kappen. Dynamic policy programming. Journal of Machine Learning Research , 13(1):3207-3245, 2012.

Baird and A. Moore. Gradient descent for general reinforcement learning. In Proceedings of the 1998 Conference on Advances in Neural Information Processing Systems II , page 968-974, 1999.

G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research , 47(1):253-279, 2013. ISSN 1076-9757.

Belousov and J. Peters. Entropic regularization of markov decision processes. Entropy , 21(7), 2019.

Blondel, A. F. Martins, and V. Niculae. Learning with fenchel-young losses. Journal of Machine Learning Research , 21(35):1-69, 2020.

Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 , 2016.

Chan, H. Silva, S. Lim, T. Kozuno, A. R. Mahmood, and M. White. Greedification operators for policy optimization: Investigating forward and reverse kl divergences. Journal of Machine Learning Research , 23(253):1-79, 2022.

Chen, Y. Peng, and M. Zhang. Effective exploration for deep reinforcement learning via bootstrapped q-ensembles under tsallis entropy regularization. arXiv:abs/1809.00403 , 2018. URL http://arxiv.org/abs/1809.00403 .

Chow, O. Nachum, and M. Ghavamzadeh. Path consistency learning in Tsallis entropy regularized MDPs. In International Conference on Machine Learning , pages 979-988, 2018.

Condat. Fast projection onto the simplex and the l1 ball. Mathematical Programming , 158: 575-585, 2016.

M. Cover and J. A. Thomas. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing) . Wiley-Interscience, USA, 2006.

Dabney, M. Rowland, M. Bellemare, and R. Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, pages 2892-2899, 2018.

Duchi, S. Shalev-Shwartz, Y. Singer, and T. Chandra. Efficient projections onto the l1-ball for learning in high dimensions. In Proceedings of the 25th International Conference on Machine Learning , page 272-279, 2008.

Furuichi, K. Yanagi, and K. Kuriyama. Fundamental properties of tsallis relative entropy. Journal of Mathematical Physics , 45(12):4868-4877, 2004.

Futami, I. Sato, and M. Sugiyama. Variational inference based on robust divergences. In Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics , volume 84, pages 813-822, 2018.

Geist, B. Scherrer, and O. Pietquin. A theory of regularized Markov decission processes. In 36th International Conference on Machine Learning , volume 97, pages 2160-2169, 2019.

K. S. Ghasemipour, R. S. Zemel, and S. S. Gu. A divergence minimization perspective on imitation learning methods. In Conference on Robot Learning , pages 1-19, 2019.

Grau-Moya, F. Leibfried, and P. Vrancx. Soft q-learning with mutual-information regularization. In International Conference on Learning Representations , pages 1-13, 2019.

Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning , pages 1861-1870, 2018.

Hiriart-Urruty and C. Lemaréchal. Fundamentals of Convex Analysis . Grundlehren Text Editions. Springer Berlin Heidelberg, 2004.

Ke, S. Choudhury, M. Barnes, W. Sun, G. Lee, and S. Srinivasa. Imitation learning as f -divergence minimization, 2019. URL https://arxiv.org/abs/1905.12888 .

Kitamura, L. Zhu, and T. Matsubara. Geometric value iteration: Dynamic error-aware kl regularization for reinforcement learning. In Proceedings of The 13th Asian Conference on Machine Learning , volume 157, pages 918-931, 2021.

Kozuno, E. Uchibe, and K. Doya. Theoretical analysis of efficiency and robustness of softmax and gap-increasing operators in reinforcement learning. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics , volume 89, pages 2995-3003, 2019.

Kozuno, W. Yang, N. Vieillard, T. Kitamura, Y. Tang, J. Mei, P. Ménard, M. G. Azar, M. Valko, R. Munos, O. Pietquin, M. Geist, and C. Szepesvári. Kl-entropy-regularized rl with a generative model is minimax optimal, 2022. URL https://arxiv.org/abs/2205.14211 .

Lee, S. Choi, and S. Oh. Sparse markov decision processes with causal sparse tsallis entropy regularization for reinforcement learning. IEEE Robotics and Automation Letters , 3:1466-1473, 2018.

Lee, S. Kim, S. Lim, S. Choi, M. Hong, J. I. Kim, Y. Park, and S. Oh. Generalized tsallis entropy reinforcement learning and its application to soft mobile robots. In Robotics: Science and Systems XVI , pages 1-10, 2020.

Li and R. E. Turner. Rényi divergence variational inference. In Advances in Neural Information Processing Systems , volume 29, 2016.

F. T. Martins and R. F. Astudillo. From softmax to sparsemax: A sparse model of attention and multi-label classification. In Proceedings of the 33rd International Conference on Machine Learning , page 1614-1623, 2016.

Nachum and B. Dai. Reinforcement learning via fenchel-rockafellar duality. 2020. URL http://arxiv.org/abs/2001.01866 .

Nachum, B. Dai, I. Kostrikov, Y. Chow, L. Li, and D. Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv:1912.02074 , 2019.

Naudts. Deformed exponentials and logarithms in generalized thermostatistics. Physica A-statistical Mechanics and Its Applications , 316:323-334, 2002.

Nowozin, B. Cseke, and R. Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Advances in Neural Information Processing Systems , volume 29, pages 1-9, 2016.

Prehl, C. Essex, and K. H. Hoffmann. Tsallis relative entropy and anomalous diffusion. Entropy , 14(4):701-716, 2012.

Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research , 22 (268):1-8, 2021.

Sason and S. Verdú. f-divergence inequalities. IEEE Transactions on Information Theory , 62: 5973-6006, 2016.

Suyari and M. Tsukada. Law of error in tsallis statistics. IEEE Transactions on Information Theory , 51(2):753-757, 2005.

Suyari, H. Matsuzoe, and A. M. Scarfone. Advantages of q-logarithm representation over qexponential representation from the sense of scale and shift on nonlinear systems. The European Physical Journal Special Topics , 229(5):773-785, 2020.

Tsallis. Possible generalization of boltzmann-gibbs statistics. Journal of Statistical Physics , 52: 479-487, 1988.

Tsallis. Introduction to Nonextensive Statistical Mechanics: Approaching a Complex World . Springer New York, 2009. ISBN 9780387853581.

Vieillard, T. Kozuno, B. Scherrer, O. Pietquin, R. Munos, and M. Geist. Leverage the average: an analysis of regularization in rl. In Advances in Neural Information Processing Systems 33 , pages 1-12, 2020a.

Vieillard, O. Pietquin, and M. Geist. Munchausen reinforcement learning. In Advances in Neural Information Processing Systems 33 , pages 1-11. 2020b.

Wan, D. Li, and N. Hovakimyan. f-divergence variational inference. In Advances in Neural Information Processing Systems , volume 33, pages 17370-17379, 2020.

Wang, H. Liu, and Q. Liu. Variational inference with tail-adaptive f-divergence. In Proceedings of the 32nd International Conference on Neural Information Processing Systems , NIPS'18, page 5742-5752, 2018.

Yamano. Some properties of q-logarithm and q-exponential functions in tsallis statistics. Physica A: Statistical Mechanics and its Applications , 305(3):486-496, 2002.

Yu, Y. Song, J. Song, and S. Ermon. Training deep energy-based models with f-divergence minimization. In Proceedings of the 37th International Conference on Machine Learning , ICML'20, pages 1-11, 2020.

Zhang, B. Dai, L. Li, and D. Schuurmans. Gendice: Generalized offline estimation of stationary values. In International Conference on Learning Representations , 2020.

A Basic facts of Tsallis KL divergence

We present some basic facts about q -logarithm and Tsallis KL divergence.

We begin by introducing the 2 -q duality for Tsallis statistics. Recall that the q -logarithm and Tsallis entropy defined in the main paper are:

In the RL literature, another definition q ∗ = 2 -q is more often used [Lee et al., 2020]. This is called the 2 -q duality [Naudts, 2002, Suyari and Tsukada, 2005], which refers to that the Tsallis entropy can be equivalently defined as:

By the duality we can show [Suyari and Tsukada, 2005, Eq.(12)]:

i.e. the duality between logarithms ln q ∗ x and ln q x allows us to define Tsallis entropy by an alternative notation q ∗ that eventually reaches to the same functional form.

We now come to examine Tsallis KL divergence (or Tsallis relative entropy) defined in another form: D q KL ( π | | µ ) = 〈 π, ln q ∗ π µ 〉 [Prehl et al., 2012]. In the main paper we used the definition D q KL ( π | | µ ) = 〈 π, -ln q µ π 〉 [Furuichi et al., 2004]. We show they are equivalent by the same logic:

The equivalence allows us to work with whichever of ln q and ln q ∗ that makes the proof easier to work out the following useful properties of Tsallis KL divergence:

-Nonnegativity D q KL ( π | | µ ) ≥ 0 : since the function -ln q π is convex, by Jensen's inequality

-Conditions of D q KL ( π | | µ ) = 0 : directly from the above, in Jensen's inequality the equality holds only when µ π = 1 almost everywhere, i.e. D q KL ( π | | µ ) = 0 implies µ = π almost everywhere.

-Conditions of D q KL ( π | | µ ) = ∞ : To better align with the standard KL divergence, let us work with ln q ∗ , following [Cover and Thomas, 2006], let us define

We conclude that D q KL ( π | | µ ) = ∞ whenever π > 0 and µ = 0 .

-Bounded entropy ∀ q, 0 ≤ S q ( π ) ≤ ln q |A| : let µ = 1 |A| , by the nonnegativity of Tsallis KL divergence:

Notice that ⟨ 1 ,π q ⟩-1 q -1 = 〈 π q , 1 -π 1 -q 1 -q 〉 = ⟨ π, ln q π ⟩ = -S q ( π ) and 1 |A| q -1 -1 q -1 = ln q |A| , we conclude S q ( π ) ≤ ln q |A| .

B Proof of Theorem 1 and 2

We structure this section as the following three parts:

Tsallis entropy regularized policy has general expression for all q . Moreover, q and τ are interchangeable for controlling the truncation (Theorem 1).

The policies can be expressed by q -exponential (Theorem 1).

We present a computable approximate threshold ˆ ψ q (Theorem 2).

General expression for Tsallis entropy regularized policy. The original definition of Tsallis entropy is S q ∗ ( π ( ·| s )) = p q ∗ -1 ( 1 -∑ a π q ∗ ( a | s ) ) , q ∗ ∈ R , p ∈ R + . Note that similar to Appendix A, we can choose whichever convenient of q and q ∗ , since the domain of the entropic index is R . To obtain the Tsallis entropy-regularized policies we follow [Chen et al., 2018]. The derivation begins with assuming an actor-critic framework where the policy network is parametrized by w . It is well-known that the parameters should be updated towards the direction specified by the policy gradient theorem:

Recall that H ( π ) denotes the Shannon entropy and τ is the coefficient. λ ( s ) are the Lagrange multipliers for the constraint ⟨ 1 , π ⟩ = 1 . In the Tsallis entropy framework, we replace H ( π ) with S q ∗ ( π ) . We can assume p = 1 q ∗ to ease derivation, which is the case for sparsemax.

We can now explicitly write the optimal condition for the policy network parameters:

where we leveraged ∂S q ∗ ( π ) ∂w = 1 q ∗ -1 〈 1 , π q ∗ ∂ ln π ∂w 〉 in the second step and absorbed terms into the expectation in the last step. ˜ ψ q ( s ) denotes the adjusted Lagrange multipliers by taking λ ( s ) inside the expectation and modifying it according to the discounted stationary distribution.

Now it suffices to verify either ∂ ln π ∂w = 0 or

where we changed the entropic index from q ∗ to q . Clearly, the root does not affect truncation. Consider the pair ( q ∗ = 50 , τ ) , then the same truncation effect can be achieved by choosing ( q ∗ = 2 , τ 50 -1 ) . The same goes for q . Therefore, we conclude that q and τ are interchangeable for the truncation, and we should stick to the analytic choice q ∗ = 2( q = 0) .

Tsallis policies can be expressed by q -exponential. Given Eq. (15), by adding and subtracting 1 , we have:

where we defined ˆ ψ q = ˜ ψ q + 1 1 -q . Note that this expression is general for all q , but whether π ∗ has closed-form expression depends on the solvability of ˜ ψ q .

Let us consider the extreme case q = ∞ . It is clear that lim q →∞ 1 1 -q → 0 . Therefore, for any x > 0 we must have x 1 1 -q → 1 ; i.e., there is only one action with probability 1, with all others being 0. This conclusion agrees with the fact that S q ( π ) → 0 as q →∞ : hence the regularized policy degenerates to arg max .

A computable Normalization Function. The constraint ∑ a ∈ K ( s ) π ∗ ( a | s ) = 1 is exploited to obtain the threshold ψ for the sparsemax [Lee et al., 2018, Chow et al., 2018]. Unfortunately, this is only possible when the root vanishes, since otherwise the constraint yields a summation of radicals. Nonetheless, we can resort to first-order Taylor's expansion for deriving an approximate policy. Following [Chen et al., 2018], let us expand Eq. (15) by the first order Taylor's expansion

f ′ ( x ) = 1 1 -q x q 1 -q . So that the unnormalized approximate policy has

Therefore it is clear as q →∞ , ˜ π ∗ ( a | s ) → 1 . This concords well with the limit case where π ∗ ( a | s ) degenerates to arg max . With Eq. (16), we can solve for the approximate normalization by the constraint ∑ a ∈ K ( s ) π ∗ ( a | s ) = 1 :

In order for an action to be in K ( s ) , it has to satisfy Q π ( s, · ) τ > ∑ a ∈ K ( s ) Qπ ( s, · ) τ -1 | K ( s ) | + 1 -1 1 -q . Therefore, the condition of K ( s ) satisfies:

Therefore, we see the approximate threshold ˆ ψ q = ˜ ψ q +1 . When q = 0 or q ∗ = 2 , ˆ ψ q recovers ψ and hence ˜ π ∗ recovers the exact sparsemax policy.

C Proof of convergence of Ω( π ) = D q KL ( π | | · ) when q = 2

Let us work with ln q ∗ from Appendix A and define | |·| | p as the l p -norm. The convergence proof for Ω( π ) = D q KL ( π | | · ) when q = 2 comes from that Ω( π ) is strongly convex in π :

Similarly, the negative Tsallis sparse entropy -S 2 ( π ) is also strongly convex. Then the propositions of [Geist et al., 2019] can be applied, which we restate in the following:

Lemma 1 ([Geist et al., 2019]) . Define regularized value functions as:

If Ω( π ) is strongly convex, let Ω ∗ ( Q ) = max π ⟨ π, Q ⟩ -Ω( π ) denote the Legendre-Fenchel transform of Ω( π ) , then

Algorithm 1: MVI( q )

Input: number of iterations T , entropy coefficient τ , TKL coefficient α Initialize Q 0 , π 0 arbitrarily Let {|A|} = { 1 , 2 , . . . , |A|} for k = 1 , 2 , . . . , T do # Policy Improvement for ( s, a ) ∈ ( S , A ) do Sort Q k ( s, a (1) ) > · · · > Q k ( s, a ( |A| ) ) Find K ( s ) = max { i ∈ {|A|} ∣ ∣ 1 + i Q k ( s,a ( i ) ) τ > ∑ i j =1 Q k ( s,a ( j ) ) τ + i ( 1 -1 1 -q )} Compute ˆ ψ q ( Q k ( s, · ) τ ) = ∑ a ∈ K ( s ) Qk ( s,a ) τ -1 | K ( s ) | +1 # Normalize when q = 2 π k +1 ( a | s ) ∝ exp q ( Q k ( s,a ) τ -ˆ ψ q ( Q k ( s, · ) τ )) end for # Policy Evaluation for ( s, a, s ′ ) ∈ ( S , A ) do Q k +1 ( s, a ) = r ( s, a ) + ατ ( Q k ( s, a ) -M q,τ Q k ( s )) + γ ∑ b ∈A π k +1 ( b | s ′ ) ( Q k ( s ′ , b ) -τ ln q π k +1 ( b | s ′ )) end for end for

̸

∇ Ω ∗ is Lipschitz and is the unique maximizer of arg max π ⟨ π, Q ⟩ -Ω( π ) .

T π, Ω is a γ -contraction in the supremum norm, i.e. | | T π, Ω V 1 -T π, Ω V 2 | | ∞ ≤ γ | | V 1 -V 2 | | ∞ . Further, it has a unique fixed point V π, Ω .

The policy π ∗ , Ω = arg max π ⟨ π, Q ∗ , Ω ⟩ -Ω( π ) is the unique optimal regularized policy.

Note that in the main paper we dropped the subscript Ω for both the regularized optimal policy and action value function to lighten notations. It is now clear that Eq. (6) indeed converges for entropic indices that make D q KL ( π | | · ) strongly convex. But we mostly consider the case q = 2 .

D Derivation of the Tsallis KL Policy

This section contains the proof for the Tsallis KL-regularized policy (7). Section D.1 shows that a Tsallis KL policy can also be expressed by a series of multiplications of exp q ( Q ) ; while Section D.2 shows its more-than-averaging property.

D.1 Tsallis KL Policies are Similar to KL

We extend the proof and use the same notations from [Lee et al., 2020, Appendix D] to derive the Tsallis KL regularized policy. Again let us work with ln q ∗ from Appendix A. Define state visitation as ρ π ( s ) = E π [ ∑ ∞ t =0 ✶ ( s t = s )] and state-action visitaion ρ π ( s, a ) = E π [ ∑ ∞ t =0 ✶ ( s t = s, a t = a )] . The core of the proof resides in establishing the one-to-one correspondence between the policy and the induced state-action visitation ρ π . For example, Tsallis entropy is written as

This unique correspondence allows us to replace the optimization variable from π to ρ π . Indeed, one can always restore the policy by π ( a | s ) := ρ π ( s,a ) ∑ a ′ ρ π ( s,a ′ ) .

Let us write Tsallis KL divergence as D q ∗ KL ( π | | µ ) = D q ∗ KL ( ρ | | ν ) = ∑ s,a ρ ( s, a ) ln q ∗ ρ ( s,a ) ∑ a ′ ν ( s,a ′ ) ν ( s,a ) ∑ a ′ ρ ( s,a ′ ) by replacing the policies π, µ with their state-action visita-

tion ρ, ν . One can then convert the Tsallis MDP problem into the following problem:

where d ( s ) is the initial state distribution. Eq. (18) is known as the Bellman Flow Constraints [Lee et al., 2020, Prop. 5] and is concave in ρ since the first term is linear and the second term is concave in ρ . Then the primal and dual solutions satisfy KKT conditions sufficiently and necessarily. Following [Lee et al., 2020, Appendix D.2], we define the Lagrangian objective as

where λ ( s, a ) and ζ ( s ) are dual variables for nonnegativity and Bellman flow constraints. The KKT conditions are:

The dual variable ζ ∗ ( s ) can be shown to equal to the optimal state value function V ∗ ( s ) following [Lee et al., 2020], and λ ∗ ( s, a ) = 0 whenever ρ ∗ ( s, a ) > 0 .

By noticing that x q ∗ -1 = ( q ∗ -1) ln q ∗ x + 1 , we can show that -∂D q ∗ KL ( ρ ∗ | | ν ) ∂ρ ( s,a ) = -q ∗ ln q ∗ ρ ∗ ( s,a ) ∑ a ′ ν ( s,a ′ ) ν ( s,a ) ∑ a ′ ρ ∗ ( s,a ′ ) -1 + ∑ a ( ρ ∗ ( s,a ) ∑ a ′ ρ ∗ ( s,a ′ ) ) q ∗ ( ∑ a ′ ν ( s,a ) ν ( s,a ) ) q ∗ -1 . Substituting ζ ∗ ( s ) = V ∗ ( s ) , π ∗ ( a | s ) = ρ ∗ ( s,a ) ∑ a ′ ρ ∗ ( s,a ) , µ ∗ ( a | s ) = ν ∗ ( s,a ) ∑ a ′ ν ∗ ( s,a ) into the above KKT condition and leverage the equality Q ∗ ( s, a ) = r ( s, a ) + E s ′ ∼ P [ γζ ∗ ( s ′ )] we have:

By comparing it to the maximum Tsallis entropy policy [Lee et al., 2020, Eq.(49)] we see the only difference lies in the baseline term µ ( a | s ) -( q ∗ -1) , which is expected since we are exploiting Tsallis KL regularization. Let us define the normalization function as

then we can write the policy as

In a way similar to KL regularized policies, at k + 1 -th update, take π ∗ = π k +1 , µ = π k and Q ∗ = Q k , we write π k +1 ∝ π k exp q Q k since the normalization function does not depend on actions. We ignored the scaling constant q ∗ and regularization coefficient. Hence one can now expand Tsallis KL policies as:

which proved the first part of Eq. (7).

D.2 Tsallis KL Policies Do More than Average

We now show the second part of Eq. (7), which stated that the Tsallis KL policies do more than average. This follows from the following lemma:

Lemma 2 (Eq. (25) of [Yamano, 2002]) .

However, the mismatch between the base q and the exponent 1 -q is inconvenient. We exploit the q = 2 -q ∗ duality to show this property holds for q ∗ as well:

Now since we proved the two-point property for q ∗ , by the same induction steps in [Yamano, 2002, Eq. (25)] we conclude the proof. The weighted average part Eq. (8) comes immediately from [Suyari et al., 2020, Eq.(18)].

E Implementation Details

We list the hyperparameters for Gym environments in Table 1. The epsilon threshold is fixed at 0 . 01 from the beginning of learning. FC n refers to the fully connected layer with n activation units.

The Q-network uses 3 convolutional layers. The epsilon greedy threshold is initialized at 1.0 and gradually decays to 0.01 at the end of first 10% of learning. We run the algorithms with the swept hyperparameters for full 5 × 10 7 steps on the selected two Atari environments to pick the best hyperparameters.

We show in Figure 6 the performance of MVI ( q ) on Cartpole-v1 and Acrobot-v1, and the full learning curves of MVI( q ) on the Atari games in Figure 7. Figures 8 and 9 show the full learning curves of Tsallis-VI.

CartPole

Acrobot-v1

500

MVI(q)

30

Average Return

M-VI S2(π)

20

Average Return

10

250

0

-10

-20

-30

0

1

2

Iteration

3

4

1e5

5

1.0

2.0

3.0

4.0

5.0

10.0

20.0

inf

entropicindexq

MVI(q)

一

M-VI

Assault

Asterix

Amidar

BeamRider

10000

250000

2500

17500

8000

200000

2000

14000

6000

150000

1500

10500

4000

100000

1000

7000

2000

50000

500

3500

0

0

0

2000

Breakout

8000

ChopperCommand

Enduro

1600

1200

500

400

300

6400

4800

4000

2400

3200

800

200

3200

1600

400

100

1600

800

Return

25000

0

Hero

50000

0

KungFuMaster

6000

0

MsPacman

200

Pitfall

19000

40000

4800

-80

AverageR

13000

7000

20000

30000

3600

2400

y

-360

640

1000

10000

1200

920

-5000

Pong

0

PrivateEye

U

Riverraid

1200

Robotank

20

2500

0000

70

12

1900

6000

56

4

1300

2000

42

-4

700

8000

28

12

-20

-500

100 4w

4000

14

Seaquest

Spacelnvaders

UpNDown

12500

Zaxxon

50000

40000

17500

14000

25000

20000

10000

30000

10500

15000

7500

20000

7000

10000

5000

10000

3500

5000

2500

0

0

0

Iteration

1e7

Iteration

3

1e7

Iteration

1e7

Iteration

Alien

Amidar

Asteroids

Assault

2500

2250

10000

8000

2000

1850

8000

6400

1500

1450

6000

4800

3200

1000

1050

4000

1600

500

650

2000

0

250

0

Asterix

4.0

1e6

Atlantis

BankHeist

BeamRider

250000

3.2

1000

17500

150000

200000

2.4

1.6

800

600

10500

14000

100000

0.8

400

7000

200

50000

0.0

0OSE

0

Return

2000

0

Boxing

Breakout

0

Centipede

100

10000

Average

1600

1200

80

60

40

500

400

300

5200

7600

800

20

200

2800

400

100

400

0

ChopperCommand

Enduro

0

Frostbite

-2000

Gopher

1160

1400

3200

4000

4800

6000

50000

40000

920

2400

3600

30000

680

1600

2400

20000

440

800

1200

10000

200

0

0

0

Hero

Jamesbond

Kangaroo

Krull

8000

10000

20000

14000

16000

6400

11200

8400

12000

4800

8400

6800

5200

8000

3200

5600

3600

4000

1600

2800

2000

0

0

0

Iteration

le75

Iteration

E

1e7

Iteration

1e7

Iteration

le7

MVI(q)

Tsallis-VI

KungFuMaster

6000

MsPacman

Pitfall

PrivateEye

40000

4800

200

1000

32000

-80

700

24000

3600

-360

400

16000

2400

-640

100

8000

1200

-920

-200

0

1200

500

Pong

Riverraid

RoadRunner

Robotank

20

20000

60000

70

12

16000

48000

56

4

12000

36000

42

-4

8000

24000

28

12

4000

12000

14

-20

Seaquest

n

Spacelnvaders

StarGunner

0

Tutankham

8000

17500

140000

250

6400

14000

112000

190

4800

10500

84000

130

3200

7000

56000

70

1600

3500

28000

10

0

0

0

-50

1e7

UpNDown

14000

Zaxxon

iteration

le7

40000

10800

32000

7600

24000

16000

4400

8000

1200

0

-2000

Iteration

3

1e7

Iteration

3

Peer reviews:
comments: SUMMARY:
This paper studies the Tsallis regularized MDPs, and proposes a practical algorithm for Tsallis KL divergence based on Munchausen RL. The experiments show that the resulting algorithm MVI($q$) performs notably better than its counterpart MVI.

STRENGTHS:
1. The paper is well-written with sufficient introduction of background
2. Properties of Tsallis policies are sufficiently studied in the paper.

WEAKNESSES:
1. $\exp_q Q_2$ in Eq.(8) should be removed. In fact, since the permutation is changeable, no weighted average is performed in Eq.(8).
2. There is no clear theoretical justification nor intuition for why Tsallis regularization is beneficial for regularized value iteration

QUESTIONS:
1. Theorem 4 seems to suggest that the Tsallis KL regularized policy focuses more on the Q values at early iterations based on the second term in Eq.(7). Is this desirable?
2. Why would MVI($q$) provide notable gains while Tsallis value iteration provides no benefits?

LIMITATIONS:
The paper only considers $q>1$ for Tsallis regularization, but does not explore the case of $q<1$
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-06-20

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a principle way of generalizing KL-divergence regularized RL into Tsallis KL regularized RL. There have been studies that replaced Shannon entropy with Tsallis entropy to obtain sparsemax policies, but they had limited success. On the other hand, this paper extends Munchausen value iteration to get MVI(q). The extension is not straightforward due to the pseudo-additivity of $\\ln_q$, but the paper manages to do it with approximation, and empirically show that the apprixmation error remains negligible for small $q$. In the experiments, the paper achieves large performance gain across 35 Atari games, where the authors conjecture that the improvement is due to the change in how MVI(q) do the exploration.\n\nSTRENGTHS:\n- Clear theoretical foundations for establishing the MVI(q) algorithm\n- The paper is written clearly and easy to follow\n- Proposed algorithm shows notable performance improvement over existing baselines\n\nWEAKNESSES:\n- The main characteristics/advantages of the proposed algorithm is less explained. While the authors assumed that the Tsallis policy can be better in exploitation allowing the policy to exploit high probability actions, It is also possible to control the regularization coefficient (i.e. temperature) to get similar effect, and it is hard to know the difference between these two options from the paper. The paper overall explains very well about the derivation of the algorithm, but it is hard to understand how it works and in what aspects it is better. The paper may benefit from including case scenarios on toy problems when MVI(q) works better than other algorithms.\n\nQUESTIONS:\n- The paper only explains in a way that MVI(q) does less exploration compared to MVI, and it better explores compared to Tsallis-VI. What happens if we control the regularization coefficient to negate thses effects (we can try adding shannon entropy regularization on top of Tsallis-VI if it concentrate to quickly)? If these effects are not able to be negated, in what aspects MVI(q) expores/exploits better than the other algorithms?\n- How sensitive is MVI(q) to the regularization coefficient $\\alpha$ compared to existing algorithms?\n\nLIMITATIONS:\nThe authors have not addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the idea of Tsallis KL divergence for regularizing RL algorithms. They first introduce the Tsallis entropy based regularization formally whilst intuiting the how the q exponential and q logarithm, as defined by Tsallis 1998, have a truncation effect on the divergence. They also formalise this idea of truncation in Theorem 1. They also provide a provably tractable approximation of the *threshold function*. The authors then go on to very interestingly analyze Tsallis KL regularized policy and assert that \"they average over history of value estimates\". They finally introduce an algorithm similar to Munchausen Value Iteration (Vieillard et al., 2020b) which uses the Tsallis KL regularization instead of vanilla KL. Their empirical results show significant gains in a lot of environments.\n\nSTRENGTHS:\nThe paper is largely well written. The manner in which the authors build on previous work in addition to explaining various concepts so well, section after section, is impressive. I liked how their theories and their empirical applications associated with Tsallis KL are explained via illustrations (figures 1,2,3). The empirical results suggest a strong improvement over MVI with numerous Atari environments. Figure 5 (left) shows that MVI(q) has significant improvements over MVI in various environments.\n\nWEAKNESSES:\nI see no major weaknesses. I have a couple of comments:\n1. For section 3.3: is this average of histories a feature of Tsallis KL or is it to be expected of other KL regularization as well? Please let me know if this is mentioned somewhere (maybe line 183-184?). I would be interested in knowing how come this is unique for Tsallis KL case.\n2. Section 4.2: you empirically choose to \"omit the residual term\" and judging from Section 5 this works well in practice. Despite this I would be interested in seeing the return on the cartpole environment as iterations increase. How is the the residual calculated or estimated for Figure 3? Please feel free to point me to the appendix or elsewhere in case I might have missed this.\n\nMinor issues and fixes:\n1. Line 77-78: citep for Sason and Verdu [2016]\n2. Line 102: the use of the constant $p$ is not defined or explained while being introduced\n3. How many seeds are the results of Figures 4 and 5 averaged over?\n4. I would expect some discussion of the computational overhead especially in terms of GPU compute time MVI(q) adds to the algorithm.\n5. For the sake of completeness, it would be beneficial to report all the hyper-params used by you even if you are porting them from past work. Additionally a detailed algorithm would also be a helpful addition in the Appendix.\n\nQUESTIONS:\nPlease see the weaknesses section.\n\nLIMITATIONS:\nI do not see any major limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPreface: this paper was assigned to me as an emergency review paper, so I had less time to do an in-depth review.\n\nThe paper tried to extend Munchausen Reinforcement learning by replacing the commonly used KL divergence with a more generalised form, i.e., Tsallis KL, and empirically demonstrated its benefits under various simulation experiments. The authors conduct theoretical proves to motivate their intuitive extension. Results showed that adding the Tsallis KL can bring impressive improvements over the simplest M-RL algorithm on game simulations. \n\n=======post-rebuttal=========\n\nBased on the clarifications and new experimental results, I raised my score to 6.\n\nSTRENGTHS:\nThe paper is generally well-motivated and well-written at the beginning, with a clear introduction to new concepts such as the Tsallis KL Divergence and its intuitive effect visualisation as in Figure 2. It extended the standard KL-based MVI to a more generalised form MVI(q), supported by theoretical proof of the feasibility of using Tsallis KL instead of KL. Further discussion had been made to explore how Tsallis KL reweights the current policy, in the analogy of other well-recognised algorithms.\n\nWEAKNESSES:\n1.Contribution. The main contribution of this paper is the extension of Tsallis KL for M-RL and provided necessary math evidence of why can, which I sincerely appreciate. However, the authors fail to address why reweighting the policy distribution by the Tsallis q matters under the hood. From a critical point of view, this work extends an established work 3 years ago by replacing the policy regularisation function to a more variable form where a new hyper-parameter is introduced. Although the paper is self-contained, the contribution to RL community seems to be minor. \n\n2.Potentially misleading and inadequate experiment result. The experiments only showed MVI(q) (i.e., the proposed method) with MVI and Tsallis-VI. The authors would be appreciated if they are able to include more baselines, especially those which considers policy regularisation, such as SAC, TRPO, etc. Moreover, the experiments were not sufficiently standardized. The authors may consider compare their methods with commonly acceptable benchmarks, such as Atari-57, or Mojuco. I also have some concern about Figure 5. The performance is obtained by computing “Improvement over Tsallis-VI on Atari environments, normalized with Tsallis-VI scores”, which, in an extreme case, look exaggerated when the Tsallis-VI scores are small enough. An easy fix is to compare MVI(q) to existing benchmark.\n \n3.Lack of deeper explanation of experiments. It is yet clear to me how MVI(q) boosts performance. Figure 3 somehow tried to discuss it but still insufficient. The authors brought up many guesses in the paper, such as line 179 and line 272. However, they did not dig into any of those.\n\nQUESTIONS:\nWill the choice of q affect the performance significantly? Is it possible that some q choices are more suitable for some particular environments? If so, can you provide an empirical study of how to choose q adapting to the characteristic of task or environment?\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-26"}