Resmax: An Alternative Soft-Greedy Operator for Reinforcement Learning
Authors: Erfan Miahi
Revan MacQueen
Alex Ayoub
Abbas Masoumzadeh
Martha White
Venue: Accepted by TMLR ()

Abstract: Soft-greedy operators, namely $\varepsilon$-greedy and softmax, remain a common choice to induce a basic level of exploration for action-value methods in reinforcement learning. These operators, however, have a few critical limitations. In this work, we investigate a simple soft-greedy operator, which we call resmax, that takes actions proportionally to their max action gap: the residual to the estimated maximal value. It is simple to use and ensures coverage of the state-space like $\varepsilon$-greedy, but focuses exploration more on potentially promising actions like softmax. Further, it does not concentrate probability as quickly as softmax, and so better avoids overemphasizing sub-optimal actions that appear high-valued during learning. Additionally, we prove it is a non-expansion for any fixed exploration hyperparameter, unlike the softmax policy which requires a state-action specific temperature to obtain a non-expansion (called mellowmax). We empirically validate that resmax is comparable to or outperforms $\varepsilon$-greedy and softmax across a variety of environments in tabular and deep RL.

Published in Transactions on Machine Learning Research (12/2023)
Resmax: An Alternative Soft-Greedy Operator for Reinforce- ment Learning
Erfan Miahi
miahi@hallerta.ca
Deportment of Computing Science University of Alberta
Revan MacQueen
revan@halberta.car
Department of Computing Science University of Aberta/Amis
Alex Ayoub
aagozb@uallerto..cm
Department of Computing Science University of Alberta
Abbas Masoumzadeh University of Alberta
masoumaa@hullerta..om
Martha White
whitem
Deportment of Computing Science University of Alberta
Reviewed OpenReview: https: opwnrevviee....ee.eeemmmm net/ formm? www.arrr60ttddddeooeeddddd--o-/
Abstract
Soft-greedy operators, namely e-greedy and softmax, remain a common choice to induce basic level of exploration for action-value methods in reinforcement learning. These operators, however, have a few critical limitations. this work, we investigate a simple soft-groody operator, which we call resmax, that takes actions proportionally to their max action gap: the residual to the estimated maximal value. It is simple to use and ensures coverage of the state-space like e-greody, but focuses exploration like softmax. Further, it does not concentrate probability as quickly softmax, and so better avoids overemphasizing sub-optimal actions that appear high-valued during learning. Additionally, we prove it is a non-expansion for any fixed exploration hyperparameter, unlike the softmax policy which requires statc-action specific temperature to obtain a non-expansion (called mellowmax). We empirically validate that resmax is comparable to or outperforms e-greedy and softmax across variety of environments tabular and deep RL.
on potentially promising actions
1 Introduction
Many value-based methods in reinforcement learning rely on soft greedy operators, such as softmar (Luce, 1959) and E-greedy (Watkins, 1989). These operators play two roles: for soft- coft-greedification within the Bellman update when bootstrapping off values in the next state (target policy) and to encourage exploration (in the behavior policy). Soft-greediffcation within the Bellman update facilitates learning a stochastic soft-optimal policy and can help mitigate the overestimate bias in Q-learning (Song et al., 2019). This changes the target policy being learned. For the second role, regardless whether a Q-learning update is used or soft-greody update, the soft-greedy operator can be used take actions. Such an approach is actually complementary directed exploration approaches, like those that learn optimistic values, because it can provide small amount of additional exploration and robustness to estimation error in the optimistic
amount of

---

Published in Transactions on Machine Learning Research (12/2023)
values. And for some environments, where only small amount of exploration is needed, these soft-greedy operators for the behavior provide a sulficient level of exploration.
The two most common soft-greedification operators e-greedy and softmax have several limitations. e- greedy explores in an undirected way. Regardless of the agent's estimates of the value of an action, the agent's 's exploration step is uniformly random. Softmax a Boltzmann policy on the action-values is more directed, in that it samples proportionally to the exponentials of the value of the actions. However, softmax is notoriously difficult to tune and suffers from concentrating too quickly due to the use of this exponential. This concentration overemphasizes actions that appear high-valued under current estimates but are actually suboptimal. This overemphasis issuc can cause softmax to settle on a suboptimal policy, as we reaffirm in our experiments. Further, the softmax is not sound when used within the Bellman update: it does not guarantee that the Bellman update is a contraction (Asadi & Littman, 2017), which is needed to ensure convergence under Sarsa or dynamic programming updates (Littman & Szepesvári, 1996). The greedy or e-groody operators, on the other hand, are both non-erpansions and so ensure convergence: when used inside the Bellman update, with a discount. less than 1 or under proper policies, the Bellman update is contraction.
There has been work improving these soft-greedy operators for value-based techniques. Several works have addressed the non-expansion issue for softmax (Asadi & Littman, 2017; Cesa- Bosa-Bianchi et al., 2017; Pan et al.). One of these works proposes a soft-greedy operator called mellowmar (Asadi & Littman, 2017) that is guaranteed to be a non-expansion when it is used within the Bellman update. The operator involves optimizing the exploration parameter in softmax per state and has been shown to improve stability when it is used for exploration (Kim et al., 2019). However, mellowmax is not a simple heuristic to use, as it requires solving a root finding problem to compute the policy for decision making. To solve this problem they used Brent's 's method, which is computationally complex (Wilkins & Gu, 2013). An approach called ralfe-difference exploration (Tokic, 2010; Tokic & Palm, 2011) adapts the exploration parameter and the temperature for softmax over time, using the difference in the softmax of the values before and after learning. A later empirical study, however, highlighted that this approach docs not perform consistently (Gimelfarb et al., 2020).
In this work, we consider new soft-greedy operator, which we call resmax, based on a probability matching scheme originally developed in the contextual bandit setting (Abe & Long, 1999; Foster & Rakhlin, 2020). This technique is similar to softmax in the sense that il assigns distinct probabilities to actions based on the estimated action-values. However, unlike softmax, the probability for taking each action is determined using its max action gap: the difference between the approximated value of the greedy action and the given action. The policy is inversely proportional to this max action gap, and avoids the use of the exponential that causes softmax to overemphasize actions. We theoretically show that it ensures a minimal probability on each action, regardless of the action-values, ensuring all actions are explored. We prove that it is a non-expansion, and combines well with generalized value iteration algorithms. We additionally provide an empirical study, across variety of hard and easy exploration problems, with tabular and deep function approximation. We find that, in tabular experiments, resmax outperforms both softmax and e-gready-ospecially when softmax suffers from overestimation and performs similarly to mellowmax, but with dramatically lower run time. In the deep RL experiments, resmax is comparable to the baselines.
Remark: It worth noting that parallel work in the policy gradient litcrature examines different policy parameterizations, that resemble some of these operators used for value- based methods. It has been noted that the softmax policy parameterization can concentrate too quickly (Mei et al., 2020), motivating the introduction of another parameterization called the escort transform. Other work, particularly those analyzing convergence properties, examines categorical policy parameterizations (Zhan et al., 2021). This policy gradient work does not directly apply here, because we do not learn a parameterized policy. Instead, we use these operators to obtain a policy directly from learned action-values. Policy gradient methods often learn action preferences, which are not the same as action-values, resulting in notably different properties. For example, action preferences can concentrate very quickly, making entropy regularization more critical in policy gradient methods. Further, the non-expansion requirement is unnecessary for policy gradient methods; in fact, we highlight that an extension of the escort transform to this value-based setting does not have the non-expansion property.
2

---

Published in Transactions on Machine Learning Research (12/2023)
2 Background
We model the environment as a discounted Markov Decision Process (MDP) R. where S is the set of states; A the set of actions; R the set of possible rewards; E [0, 1] the discount factor; and P:S. x RxSx x A [0, 1] the dynamics function. In a given state s, the agent takes action and transitions to state s' and receives reward r according to probability P(s',r(s,a).
The agent's goal to learn a policy π. A [0, 1] that maximizes its discounted cumulative reward. The action-value function for each state-action pair under the policy is
4n (8 a) ET
Σ 7 k=
= A =
(1)
The agent attempls to approximate ge*, the action-value function for the optimal policy x², with an approximate action-value function q that is parameterized by 6. Many reinforcement learning algorithms compute this approximation iteratively, using either Q-lcarning Expected Sarsa. Both methods use update to the paramcter vector 0 of the form
Oc+1 f O2 + ad: Va(s,
but with different TD errors 5,. Q-learning uses oN = r ymax (s',''' a', 02) - q(s, a, 0z). Expected Sarsa uses 6, = rr+ + (1a))ss [q(s', a', 0.)] - ((*, a, f2) where the target policy π is typically a soft-greedy policy. When has an entry for each s, a, pair, we call this the tabular setting. Going forward, we will use q(s, a) in our descriptions of operators and policies; however, one can easily substitute it for g(s, a, in non-labular settings.
To learn these action-values, the agent needs to explore. One simple strategy to promote exploratory behavior is to use soft-greedy policies, either just for the behavior (as in Q-learning) or for both the behavior and target (as in Expected Sarsa). Two common soft- greddy policies are the Boltzmann softmax policy and e-greedy, definod as
(a 8)
(2)
Σ eq(a,0")=-1
d'EAA
1 1- [aj
if ungmax,g(s,c0
Teg(a 8) =
(3)
otherwise
with an exploration parameter 7 > and E [0, 1]. With lower values r, softmax is greedier; with higher values of r, softmax action selection becomes more equiprobable. Similarly, for 0, the e-greedy policy is perfectly greedy, becoming increasing random with higher e; e 1.0, the policy becomes uniform random with equiprobable action selection. Thus, these methods share two essential charactcristies: randomness in action ion to guarantee the coverage of the whole state space and hyperparameter to vary the degree to which the greedy action is chosen.
Though widely used, softmax and s-greedy suffer from several flaws. One of the major problems of e-greedy is that it ignores the estimates of action-values and assigns a uniform probability to each non-greedy action. This undirected exploration results wasting time taking actions that the agent might already know are vastly suboptimal. Further, to explore specific parts of the environment, it might need to chain sequence of random actions, which might be very low probability. Consider, for instance, the RiverSwim environment (Strehl & Littman, 2008), where agent receives a minor positive reward if it takes the left action in the initial state, but a much larger reward at the far right of the environment that is harder to reach. To receive this high reward, the agent might need to take several exploratory actions in a row, and so typically gets stuck learning to go left.
Softmax more directed, in that it assigns the probability of each action corresponding to its action-value. However, if the temperature is not set carefully, softmax will assign excessively disproportionate probability
3

---

Published in Transactions on Machine Learning Research (12/2023)
to the greedy action because this probability is based on exponentiating action values. Consequently, softmax often overemphasizes actions that currently have high value estimates, at the expense of exploring other actions. We provide an illustative example in the next section, explaining this issue further.
Furthermore, when using the softmax operator in the Expected Sarsa update, this operation is not guaranteed to be non-expansion (Littman, 1996; Littman & Szepesvári, 1996). The softmax operator inside the Bellman update, sm(g(s', ,), = (- a')] = -a Tam (a'ls')g(( s')g a'). Generalized Value Iteration (GVI) algorithms, such Expected Sarsa, require this operator to be non-expansion, to guarantee converge to unique fixed point (Littman & Szepesvári, 1996; Asadi & Littman, 2017). Without the non-expansion property, an operator may converge to multiple fixed points, even in the tabular setting; softmax does not. have this property.
The mellowmax operator (Asadi Littman, 2017)
1 Σ imd
(x) = w log
(4)
was designed to fix this non-expansion issuc! Mellowmax behaves as a quasi-arithmetic mean, with a parameter w controlling how much mellowmax behaves like an average or a max. The correspouding mellowmax policy Tmm can be found by solving a convex optimization. First, using a solver?, one obtains a value 3 such that:
e 8a(x,a) Bqlm.a))Bumm(atte g(s. a) - mm =
(5)
AEA
This B is state-specific softmax temperature to provide the resulting (maximum entropy) mellowmax policy:
Tmm (als) = En'e Enresellaaay eals,a')
(6)
Since 3 needs to be computed al each state, mellowmnx needs considerably more compute than soflmax, resmax or s-greedy. The need for root finding also increases the complexity of implementing mellowmax as soft-greedy operator; in our implementation, we had to correct for numerical instability and iteratively increase the bounds of the root finder to make sure B always exists without having bounds set too wide (and thus affecting runtime).
3 The Resmax Operator
Our goal is to obtain the benefits of all three of these operators: (a) use the information provided by action-values (like softmax and mellowmax), (b) avoid the overemphasis of softmax (like e-greedy) and (c) be easy and computationally efficient to use (unlike mellowmax). For this purpose, we propose resmax. Let fmax (s) = max, ((s, a) be the value of any greedy action state 8, and G(s) (b E A g(s b) gmax be the set of greedy actions. The resmax policy Tem is defined
1
a # G(s): T (a [c]
(7)
A + 17 1 (Amax q(s, a))
b E G(s): Tmm [b|s)
=
G(s)] (1 Σ Tmm $)
afG(+)
where the exploration parameter 1) > 0 can be thought of as the exploration pressure, Larger values push towards exploration whereas low values result in more exploitation- are greedier. When n + oo the policy is uniformly random and when n - the policy is greedy with respect to q(s,)).
1 We write mellowmax with w/// Asadi Littman (2017), that increases, mellowmax behaves greedily similarly e e-greedy and softmax.
We brentq, provided in scipy (Virtanen al., 2020).
4

---

Published in Transactions on Machine Learning Research (12/2023)
1.0
17(a1)
0.8
0.6
4
10
q(a1)
((22)
0.125
0.5
2 8
softmax
resmax
Figure 1: Contrasting policies produced using the softmax and resmax, for different exploration parameters, with ((02) = -5 and ((11) E 5-,,5]. As the max action gap increases, softmax much more quickly concentrates probability on action a1 than resmax.
In words, the resmax policy first assigns probability to every action inversely proportionally to it's action-values 1/([A| + + (9max - q(s, a))) is biggest when q(s, a) = mmax: equaling 1/([,)),, with less probability assigned to an action as q(s.a, a) gets further from 9max. Then the remaining probability is evenly distributed amongst the maximal (groedy) actions. This mimics e-greody, where all actions are assigned small probability e/ AA] and remaining probability distributed among maximal actions. Resmax, however, assigns small probabilities for each non-greedy action proportional to the action-volues, rather than uniformly.
Consider two extreme cascs for intuition. If all actions are equal, then they are all assigned 1/|A|. If one action is maximal, then all others actions are assigned some probability smaller than 1/|AI, and the remaining probability assigned to will be higher than 1/|.A. If the action gap is big, g(s, b) >> q(s, a) and = 1, for example, then Tym (b]s) is almost and the probability on the remaining actions is near zero.
This operator efficient to compute and easy Q-learning DQN, the agent simply samples actions from the resmax poliey in Equation 7. To use it within the Bellman update, for noft-greedification,
for exploration soft-greedification. To use with
can simply use the corresponding resmax operator
rm (g(
Σ aEA
Trm \ s)g(s,a)
(8)
((,00) a)
Σ aEA
(gmax a)) 9max
A 1 (ymax g(s, a))
As goes to 0, this operator becomes the max operator (proven in Appendix A).
To better understand resmax, we contrast
softmax Figure 1. We particularly highlight that softmax overemphasizes actions with high approximates values, in comparison to resmax. We consider a sctting actions dı and 02, and visualize the policies for state for different max action gaps
where there between the approximate values, q(a1) q(a2), where g(a:)) = q(š, a.). We set g(a2) -5 and then vary g(az) E 5]. For most choices of the temperature т, if q(a1) is sufficiently larger than ((02), then the outputs of the softmax policy for aj will be near 1. Thus, it may take a very large number of steps to finally choose action 02 and explore this other action. Under resmax, the probability assigned to the greedy action is much less skewed towards 1, particularly for smaller gaps between the values in the actions.
In the next two sections, we motivate that resmax satisfies the other two requirements, in addition ease-of-use: (1) it avoids the overemphasis in softmax and encourages exploration and (2) is non-expansion.

---

Published in Transactions on Machine Learning Research (12/2023)
4 Resmax Encourages Exploration
In this section, we first confirm that resmax is guaranteed to provide sufficient exploration, by maintaining non-noglible probability on all actions. Then we invostigate resmax in two hard exploration problems, with misleading rewards, in comparison to softmax and mellowmax.
4.1 State-Action Space Coverage
Exploration strategies should satisfy certain fundamental properties make sure that algorithms such as Q-learning and Sarsa converge the optimal value (Singh al., 2000; Watkins & Dayan, 1992). A key property is that cach stato-action pair should be visited infinitely many times during continual learning. To show that resmax satisfies this properly, we prove that the probability of laking all of the actions will be higher than zero during learning for any bounded action-values. This result is straightforward to show, but needed for completeness to ensure we do not lose this useful property of s-greedy and softmax. Property 4.1. Assume there exists Thound > 0 such that Vs, a, (a(8, a)l s Thound. The probability of taking any non-greedy action a and a greedy action satisfies
1 A
0 <
I TTm (a 18)
S 7m (b [b1s)<1
A + 2m
Thound
Proof. First, we determine the upper-bound and lower-bound for non-greedy actions a. By analyzing Equation 7, it is clear that its lowest-value will be obtained only when the difference between {ss and g(s,0r a) is at its highest. We therefore get the following lower bound on Trm ()
Tem (a () = |A| 17 (q(s, b) - {(s,a)
| 11 (Toound (hound))
>0
|A| + 211-21 Thound
Furthermore, Frm |)) is bounded above by 1/|A], because ((s,b) b) q(s, in the denominator. The probability of greedy actions b are lowest when all actions greedy, having probability 1/|A]. ]
4.2 An Illustrative Example of Overemphasis
A serious issue with using softmax for exploration comes from the fact that it uses exponents within its formulation, which can assign an overly disproportionate probability to current greedy action under the approximation action values. We demonstrate the severity of the softmax overemphasis problem in the HardSquare MDP (Figure 2a). The agent starts in the state and 82 with an equal probability. The agent can stay in the start states for a reward of 10*, or move to 83 and $4, where it can receive a larger reward of x 104. In expectation, $3 and 81 are better, but the high reward in the initial states 81 and $2 can mislead the agent. For the experiment, we use tabular Q-lcarning. We initialize q(s, a) + O, V(s, E S A. The results, depicted in Figure 2b, are obtained by averaging over 30 runs and selecting the best performance across three different step sizes.
We can see that the softmax policy stays in the initial states and 82 duc to its tendency to overemphasize the actions that stay in the start states On the contrary, resmax can escape from the initial states and successfully explore towards the optimal solution. We include mellowmax in our results for Hardsquare and note that performs worse than resmax for best choices of hyperparameters. Due to the optimization procedure inherent in mellowmax, its runtime on average 28x longer than resmax.
4.3 More Experiments in Classic Hard Exploration Environment
We further investigate the exploration properties of these operators in more well-known environment typically used to test exploration, called RiverSwim (Strehl & Littman, 2008). This environment consists

---

Published in Transactions on Machine Learning Research (12/2023)
of states arranged in line, of which the agent starts in the leftmost. This state has small reward but the mosh state may give a comparatively large reward. In order to reach this state the agent must. traverse the MDP from left to right, fighting against a current which causes moving right to often fail. We use RiverSwim with a fixed horizon, meaning that after 20 actions the agent will be returned to the start stale, in effect making the environment more difficult. See Figure for the full specification. All algorithms are run for 800,000 time steps.
-104 b, 0.25,
b, 0.25,
a,1.00 10a
-10
104
le9
a, 1.00. 4
1.8
1.000, 0
1.6
S
S
Total Return
1.4
c, 1.00, 0
b, 6.0.75, 0
b,1.00,0
6,0.76,0
1.2
b. 0,1.00,
1.0
c, 1.00,
0.8
mellowmax
S3
S4
softmax
0.6
resmax
c, 1.000,00
0.4
a, 1.00, 10
a, 1.00, x 10
2 12
2
2 2-
23 2 2 2 23
n, T, W
(a) HardSquare MDP
(b) Performance on HardSquare
Figure (a) The HardSquare MDP: a simple MDP with four states, three actions and y 0.95. Each edge shows an stochastic transition labeled by action, probability, and reward respectively. The agent starts in states and $2 with equal probability. (b) Performance of resmax, mellowmax and softmax HardSquare. The means and standard errors are shown for 30 runs for each parameter setting. The x-axis is the
choices for each operator. Softmax gets stuck in states $1 and 82, whereas resmax can escape from the initial states and successfully explore towards higher value states. Results are shown for 15, --14 ,...,.25) 2
n,, w E (2 2 ,..,, 255 and T E {2-15,2
0.4
0.6
0.6
0.6
0.6
Q
0.6
0.35
0.35
0.35
0.35
1 0.005
82
86
0.6, r
0.05
0.05
0.05
0.05
A
0.4
(a)
RiverSwim
0.4
0.6
0.6
0.6
0.6
0.6
0.35
0.35
0.35
0.35
1,r (0.005, 1) <+
82
33
85
86
0.6, ~,6,))
0.05
0.05
0.05
0.05
0.4
1
(b) Stochastic-reward RiverSwim
Figure Diagram of RiverSwim and stochastic-rewait RiverSwim. Dotted lines and solid lines show the transitions and probabilities for the left and right actions, respectively. Diagram adapted from Osband et al. (2013).

---

Published in Transactions on Machine Learning Research (12/2023)
1e6
softmax
Frequency of State
0.6
1.0
resmax mellowmax E-greedy
Total Return
0.4
Softmax
0.5
0.2
Resmax
0.0
0.0
2
2
10000 20000 30000 40000 Episode
E, n, T, w
(a) Performance on RiverSwim
(b) State visitation on RiverSwim
Figure 4: (a) Performance across hyperparamcters for RiverSwim. The means and bootstrapped 95% confidence intervals are shown for 30 runs for each parameter setting. The x-axis is plotted with a log scale, and shows the value of exploration parameters. We show results for E E .0.0.1...., 1} T E (2-9,2 2,25 ...., 22), 7 (2-12, 2-11 .....2% 20) and w E (2-12, 2-11 ...., 2"} (b) The state visitation frequency per episode for the initial state (light curve) and final state (dark curve) of RiverSwim for best hyperparameters (n 2 2- 1,,, 2 2- Resmax starts with slightly higher frequency in the initial low reward state, but once resmax encounters the final high reward state it pushes towards this state more than softmax since it avoids overemphasis of the low reward actions in the initial state. Note that mellowmax and desllon-greedy are left out to mainly focus on comparing state visitation of resmax to sax. But, as expected from the similar performance of mellowmax to resmax in these experiments, their state visitation similar.
Runtime (s)
103
Mellowmaz
Softmax
Resmax
E-greedy
Figure 5: Logarithmic-scale runtime of Expected-Sarsa for each operator, averaged over all parameters across both RiverSwim variants. We show bootstrapped 95% confidence intervals with error bars.
We ran resmax, softmax, mellowmax and 6-greedy coupled with Expected Sarsa on RiverSwim across a broad range of hyperparameters (n, т, w and e). The hyperparameter ranges were chosen match the technique. For example softmax requires relatively larger values of 7 due its exponents. Results are shown in Figure 4a, where measure algorithm performance by the total online return across these hyperparameters.
We find that resmax achieves higher total return under best hyperparameter settings than softmax, indicating it is more capable of escaping the pull of the left stale. e-greedy fails to explore at all, and its total return is hardly visible for any values of E. Figure 4b confirms that softmax lingers in the initial leftmost state longer than resmax does and ultimately spends less time in the high reward rightmost state. Mellowmax performs very similarly resmax across hyperparamter settings; however, due to the call a root finder within mellowmax, takes much longer in terms of wall time (shown in Figure 5).
8

---

Published in Transactions on Machine Learning Research (12/2023)
1e6
1.5
softmax
Prob. Optimal Action
0.9
resmax
Total Return
1.0
mellowmax E-greedy
0.8
Resmax
0.7
0.5
0.6
Softmax
0.0
0.5
2
2
0
10000
20000 Episode
30000
40000
E, n, T, w
(a) Performance across hyperparameters
(b) Policy in initial state
Figure 6: (a) Performance across hyperparameters on Stochastic-reward RiverSwim. The mcans and bootstrapped 95% confidence intervals are shown for 30 runs for each parameter setting. The x-axis is plotted with log scale, and shows the value of exploration parameters. We show results for the same hyperparameter ranges as Figure 4a. (b) The probability of selecting the optimal action (right) from the initial state during the first step in the episode in Stochastic-reward RiverSwim. Results for both figures are averaged over 100 runs and for optimal choices of hyperparameters in Stochastic-reward RiverSwim: 1j 2-3 =
Stochasticity in rewards can help alleviate some of the misleading reward problem. Uncertainty in the rewards prevent overeexploitatio of sub-optimal actions and help drive exploration. We replaced the deterministic positive rewards in RiverSwim with rewards drawn from Gaussian distribution with the mean being the original reward and variance of 1. Results are shown in Figure 6a. All four techniques see a reduction in hyperparameter sensitivity, but resmax, mellowmax and e-greedy increase their total return under optimal parameter scttings, whereas softmax does not. With stochastic rewards, softmax still assigns too great a weight on sub-optimal actions, as shown in Figure 6b. Softmax learns take the optimal action in the initial state at slower rate than resmax, indicating that softmax is concentrating too heavily misleading rewards.
5 Resmax Is Non-Expansia
The second key property of resmax is that it is a non-expansion. Figure 7a shows a simple two state MDP (taken from Asadi & Littman (2017)) where the action-values of softmax may not converge, as shown Figure 7b. Resmax, since resmax indeed converges under setting of 7 that provides similar exploration to softmax.
non-expansion, converges for all settings of n. We show in Figure that
We now show that the resmax operator is non-expansion. Property 5.1. The resmax operator non-expansion: for any two vectors ā. B.A. and nj 2 0 we have,
(m)(E,n) rm(j, 7)) S max lx: - Vil
xE(A)
Proof. Let us rewrite the resmax operator, using & = ||
rm((,n)
Σ
+ ||00100
Σ
A + 18;
A + 18,
KE(A]
iE(A)
5;
=||0|lec
Σ
||0||00
(10)
A
L5; 7
An
|E[A
SE[A]
To show that resmax is non-expansion, we use Theorem 1 of Paulavičius & Žilinskas (2006) which states the following. For Lipschitz function f(x), Rdd R,
(5(z) f(w) < Ln |xc yl1oo
9

---

Published in Transactions on Machine Learning Research (12/2023)
2.0
Qls a) O(5 b)
2.0
Q(s a)
Q b)
Action-valus
1.5
Action-value
1.5
(a, 0.66, 0.122)
(a, 0.34, 0.122)
1.0
82
0.5
0.5
(b. 0.01, 0.033)
0.0
0.0
(b, 0.99, 0.033)
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Iteration
1e7
Iteration
1e7
(a) Simple 2 state MDP
(b) Softmax
(c) Resmax
Figure 7: The non-expansion property is important for convergence under GVI. On the MDP shown in (a), if we use softmax in the Expected Sarsa update (with т 1/16.55), the action values do not converge, shown in (b). We can see that resmax does converge in (c) with = 0.000085 (chosen to induce similar level of exploration to softmax). We use step size a at time t to be 1/(L+/100, 0000 + 1), which meets the usual conditions for stochastic approximation to converge (Sution & Barto, 2018). The tuples in (a) are of the form: (action, probability of transition given action, reward). Results are smoothed with a window size of 10, as in Asadi & Littman (2017).
where = sup([VS(<)) x D} is the Lipschitz constant, D is compact set, and Vf(x)) (Of Of is the gradient of the function f(x). Therefore, we need show that the & norm of the gradient of resmax is less than equal to 1, namely L1 S.
The resmax operator is differentiable, so can apply this theorem. We characterize the partial derivatives and show that their & norm is bounded by 1. Notice first that ب |x]|| = 0 if æj < || or if more than one entry in maximal. Infinitesinally changing 2j in these two cases does not change || This partial derivative is only non-zero if æy is the unique max; changing it changes the max norm lincarly, meaning the partial derivative is 1. Therefore, for where Iÿ is the unique max,
An2
rm (2, n) da,
Σ
(11)
(Ar) + 6.)2
iE(A),is
and for other indices (or x where there are two or more maximal elements), we have
0 mm = (An + 6,)2
Anj
(12)
Notice now that
Arp
Anj²
An2
(13)
(An + 8,)2
A2 + 2Ano; + 52
A
where the inequality holds because all the terms in the denominator are positive. The l norm corresponds to summing up all the partial derivatives.
Case 1 (ar with two or more maximal elements): All partial derivatives are of the form in Equation 12, bounded above by 1/A as per Equation 13, so the & norm is bounded above by ΣiE(A) =/A= 1.
Case 2 (or has a unique maximal element rx): By Equation 13, we know that the partial derivative for k (in Equation 11) is positive. Therefore, the & norm corresponds to summing up all these nonnegative partial derivatives
An (An + 8,)2
An2 (An + 6,)2
Σ
Σ
i
(E(A)
All these arguments were true for any G, therefore, La and resmax is nonexpansion.
10

---

Published in Transactions on Machine Learning Research (12/2023)
This result also lets us show that the sequence of policies generated by approximate policy iteration under the resmax operator converges to unique limiting policy regardless of the choice of the initial policy This follows from Theorem 1 of Perkins & Precup (2002), which simply requires the operator be Lipschitz continuous with an c-soft policy (I is c-soft if (a]s)) > for Va E A). As shown in Section 4.1, the resmax policy e-soft. By the non-expansion properly of resmax, we can also say the resmax operalor Lipschitz continuous (Asadi Littman, 2017).
A natural question is if we could have obtained non-expansion result with different variant of resmax. For example, alternative is to take the squared max action gap, more generally p-norm: (xy 24)". Somewhat surprisingly, we cannot do so; we prove that for p > 1, we would no longer have a non-expansion (see Appendix B.1). This further motivates the particular form we chose for resmax.
Finally, our strategy to prove that resmax an non-expansion was simple: bound the -orom of the gradient, with respect the inputted action-valucs. more complex, specialized approach was used for mellowmax (Asadi & Litlman, 2017). Though our stralegy is straightforward, it the first time it has been used for these operators, and should facilitate analyzing other new operators. We use this strategy to show another potential operator, based on the escort transform (Mei et al., 2020), does not have the non-expansion property (see Appendix B.2).
6 Resmax in Deep RL
Many balued-based RL algorithms with function approximation use s-greedy as default method add a degree of exploration. Resmax, in order to be considered general-purpose method, should also scale this sctting. To show this, conduct experiments in the deep RL setting (1890 experiments) across both casy and hard exploration Atari 2600 environments (Bellemare et al., 2013). Ranges for exploration parameters are chosen to ensure that the peaks for each approach resides in the chosen range. Further details on our implementation and computational infrastructure are in Appondix C.
We first study the utility of resmax compared to the baselines of mellowmax, softmax and e-greedy on easy exploration Atari environments, namely Asterix and Breakout, chosen from human-optimal easy exploration environments as categorized in Bellemare et al. (2016). We begin by analyzing the sensitivity of both algorithms to their exploration parameters. The sensitivity plots for these experiments are shown in Figure 8, We see that softmax performs better with higher exploration parameters compared to resmax. This observation aligns with our intuition, considering the overemphasis property of softmax. In both environments, we can observe that the e-greedy approach yields better results with a lower level of exploration, specifically with an
Breakout
Asterix
1e2
8 1e2
Accumul lated Returns
Accumulated lted Returns
6
2-
softmax
1
mellowmax
Greedy Baseline
-220
14
-
2-2
2-20
14
2-8
--2
7, T, w
77 T, w
Figure 8: Sensitivity curves showing the performance across hyperparameters. Results shown are averagod over 10 runs for each parameter setting and the shaded region represents the standard error. The x-axis of sensitivity plots plotted with log scale, and shows the value of n and т, which swept over (29 2- 1 ..... 2-20 2- 21). e-greedy is the best rest-performing instance selected from values of {0.01, 0.1, 0.2, 0.3, 0.5).
11

---

Published in Transactions on Machine Learning Research (12/2023)
Breakout
Asterix
lel
le1
2.0
1.0
Average Retum
Average Retum
0.8
1.5
0.6
1.0
greedyy: 0.01
0.4
softmax: 2
0.2
mellowmax: 2-8
0.5
resmax: 2-12
0.0
10
20
30
40
50
0
10
20
30
40
50
Step (10')
Step (10*)
Figure 9: Learning-curves presenting the best, performance across the selected exploration parameters. Results shown are averaged over 10 runs for each paramcter sctting and the shaded region represents the standard error.
Freeway
Pitfall
Gravitar
1e3
1e3
1e1
Accumulate aaddd Returss
1.5
Accumulated Returns
Accumulated Returns
1.6
-1-
1.0-
1.4
-2
resmax
1.2
softmax
mellowmax
Greedy Baseline
1.0
2-
2
2
2
-50
2
2
2
2-20
2
-
2
n) T,
n. T. w
9. T. W
Venture
PrivateEye
1e1
1e2
Accumulated Returns
Accumulated Returns
2.0
-2
1.0
0.5
-6-
0.0
2
20
2-14
2
2
20
8
2
W
Figure 10: Sensitivity curves showing the performance across hyperparameters. Results shown av- eraged over 10 runs for each parameter setting and the shaded region represents the standard error. The x-axis of sensitivity plots is plotted with log scale, and shows the value of and т, which are swept over {20, 2-4 2 2-...,.20... 2,,224, 2-22.. e-greedy is the best performing instance selected from e values of (0.01, 0.3, 0.4.0.5..
value of 0.01, indicating that these environments embrace more greedy algorithms, finding consistent with Laidlaw et al. (2023).
The learning curves for the lop-performing hyperparameter configurations of each algorithm are illustrated in Figure 9. In general, resmax exhibits superior performance or remains competitive when compared to softmax and e-greody, while demonstrating similar performance to mellowmax. Specifically, in the Asterix environment, resmax competes well with softmax and mellowmax, surpassing e-greedy. However, on Breakout, resmax out performs softmax, while performing similarly to e-greedy and mellowmax.
12

---

Published in Transactions on Machine Learning Research (12/2023)
Freeway
1e1
Pitfall
Gravitar
lel
Average Return
3.5
Average Return
Average Return
-2
3.0
-4
e-greedy: 0.3
2.5
softmax: 2
mellowmax: 2-1
2
2.0
20
10
20
30
40
50
10
20
30
40
50
Step (10*)
Step (109)
Step
(109)
PrivateEye
Venture
le1
1.25
Average Return
1.00
Average Return
-2
0.75
-4
0.50
0.25
0.00
20 30
50
10
20
30
40
50
Step (109)
Step (10?)
Figure 11: Learning-curves presenting the best performance across the selected exploration parameters. The results shown are averaged over 10 runs for each parameter setting and the shaded region represents the standard error.
Next, we analyze the efficacy of resmax with respect to the same baselines in five sparse-reward Atari environments that are hard to explore: Freeway, Pitfall, Gravitar, Venture, and PrivateEye. The sensitivity plots are presented Figure 10. Opposite the casy exploration results, resmax, mellowmax, and softmax tend to do better with lower exploration parameters, except in Gravitar and Pit.fall, showing that they require more exploration in these environments.
The learning curves for the top-performing instances in these enviromments are depicted in Figure 11. As observed, resmax outperforms both softmax and e-greedy in two the environments: Freeway and Venture. Additionally, it cxhibits competitive performance in the remaining three environments, with a slightly lower return in Gravitar compared to softmax. When compared to mellowmax, resmax performs similarly in all environments, except in Venture where it slightly outperforms mellowmax.
These findings indicate that resmax is promising alternative to e-greedy and softmax for promoting exploration. A particularly striking result is how much more effective both resmax and softmax are than s-greedy, both in Atari and in earlier results. If nothing else, these results suggest that we should consider resmax and softmax more often in Atari experiments. Further, resmax typically quite similar to softmax, with a few instances where is notably better.
7 Conclusion
Soft-greedy operators, including e-greedy, softmax and resmax, continue to play important role in reinforcement learning. They serve dual purposes: they function soft-greedification within the Bellman update and to induce basic level of exploration. Hence, they are valuable for both on-policy and off-policy learning. Their simplicity is also strength: they can be easily used with (deep) function approximation and are complementary to more focused exploration techniques. We propose a new soft-greedy operator, called resmax. Unlike softmax, resmax a non-expansion, regardless of the choice of exploration paramcter, and thus suitable for use with a Bellman update. Moreover, resmax ensures state-action space coverage and it avoids softmax's fundamental issue of overemphasis. Our empirical results show that resmax encourages
13

---

Published in Transactions on Machine Learning Research (12/2023)
exploration than softmax, since it does not overemphasize and also explores more efficiently than e-greedy. Resmax also has the non-expansion property of mellowmax with fraction of the computation required.
This paper proposes resmax in its simplest form, so there are many avenues for future research. As resmax is a new operator to reinforcement learning, it deserves further benchmarking and experimentation in both simple and complex environments order to shed more light on when this operator useful. Another natural direction for future work is explore adaptation and normalization techniques which are suitable for resmax. Just like decay schedules for schedules adaption schemes for resmax could allow more exploration in early layer, and allow greedier policies later.
References
Naoki Abe and Philip M Long- Associative reinforcement learning using linear probabilistic concepts. Tn International Conference on Machine Learning, 1999.
Kavosh Asadi and Michael L. Littman. An alternative softmax operator for reinforcement learning. In International Conference on Machine Learning, 2017.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In International Conference on Neural Information Processing Systems, 2016.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research,
2013.
Nicolò Cesa-Bianchi, Claudio Gentile, Gabor Lugosi, and Gergely Neu. Boltzmann exploration done right. In International Conference Neural Information Processing Systems, 2017.
Dylan Foster and Alexander Rakhlin. Beyond UCB: Optimal and efficient contextual bandits with regression oracles. In International Conference on Machine Learning, 2020.
Michael Gimelfarb, Scott Sanner, and Chi-Guhn Lee. Epsilon- BMC: A Bayesian ensemble approach to opsilon-greedy exploration in model-free reinforcement learning. Uncertainty in Artificial Intelligence, 2020.
Seungchan Kim, Kavosh Asadi, Michael L. Littman, and George Konidaris. Deepmellow: removing the need for a target network in deep Q-learning. In International Joint Conference on Artificial Intelligence, 2019.
Cassidy Laidlaw, Stuart Russell, and Anea Dragan. Bridging rl theory and practice with the effective horizon. In International Conference on Neural Information Processing Systems, 2023.
Michael L. Littman. Algorithms for sequential decision making. Brown University, 1996.
Michael L. Littman and Csaba Szepesvári. A generalized reinforeement-learning model: Convergence and applications. In International Conference on Machine Learning, 1996.
R Duncan Luce. Individual choice behavior: A theoretical analysis. 1959.
Jincheng Mei, Chenjun Xiao, Bo Dai, Lihong Li, Csaba Szepesvári, and Dale Schuurmans. Escaping the gravitational pull of softmax. International Conference on Neural Information Processing Systems, 2020.
Tan Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient. reinforcement learning via posterior sampling. In International Conference on Newral Information Processing Systems, 2013.
Ling Pan, Qingpeng Cai, Qi Meng, Wei Chen, and Longbo Huang. Reinforcement learning with dynamic boltzmann softmax updates. In International Joint Conference on Artificial Intelligence.
Remigijus Paulavičius and Julius Žilinskas. Analysis of different norms and corresponding lipschitz constants for global optimization. Technological and Economic Development of Economy, 12:301-306, 12 2006. doi: 10.3846/1338 11166/19916999./// 2006.9637758.
14

---

Published in Transactions on Machine Learning Research (12/2023)
Theodore J. Perkins and Doina Precup. A convergent form of approximate policy iteration. In International Conference on Neural Information Processing Systems, 2002.
Satinder Singh, Tommi Jaakkola, Michael L. Littman, and Csaba Szepesvári. Convergence results for
single-step on-policy reinforcement-earminn algorithms. Machine learning, 38(3):887-30) 308, 2000.
Zhao Song, Ron Parr, and Lawrence Carin. Revisiting the softmax bellman operator: New benefits and new perspective. In International Conference on Machine Learning, 2019.
Alexander L. Strehl and Michael L.. Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences, 74(8))1309 1331, 2008.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Michel Tokie. Adaptive e-greedy exploration in reinforcement. learning based on value differences. Annual Conference on Artificial Intelligence, 2010.
Michel Tokic and Günther Palm. Value-difference based exploration: Adaptive control between epsilon-greedy and softmax. In German Conference on Aduances in Artificial Intelligence, 2011.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, J Carey, Ihhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261 272, 2020.
Christopher J. C. H. Watkins. Learning from delayed rewards. King's College, Cambridge, 1989.
Christopher J. C. H. Watkins and Peter Dayan. Technical note Q-learning. Machine Learning, 8:279 292, 1992.
Gautam Wilkins and Ming Gu. A modified brent's method for finding zeros of functions. Mathematik, 123(1):177 188, 2013.
Numerische
Wenhao Zhan, Shicong Cen, Baihe Huang, Yuxin Chen, Jason D Lee, and Yuejic Chi. Policy mirror descent for regularized reinforcement learning: A generalized framework with linear convergence. arXiv preprint 1rXiu:2205.11006 2021.
15

---

Published in Transactions on Machine Learning Research (12/2023)
A Resmax Does Maximization
It can be easily shown that the Expected Sarsa update of resmax, represented in Equation 8, can do maximization (i.e, give the highest probability to the greedy action) when 1) goes to O. We first show how Ta s) for non-greedy actions will change when this happens:
lim Tem (a 8) = lim 7
7-0 |Ai + 17 b) {(s, 0
This equality will hold as long as (),,b) + g(8.a)) a).
Considering this, we can derive Equation 8 when goes to infinity as follow:
lim mm((ss,),,) = lim Σ agG(s)
Tum (a s)q(s,
7-40
+ (1
Σ
Tym 8))
agG(s)
0 (1 0)qmax qmax
Since Imax is the action-value of the greedy action, resmax can do maximization. It is also interesting to note that when 7) goes to oc, the gencrated policy will be equiprobable thus Expected Sarsa update of resmax will average all the action-values. So, this operator can make a balance between q-learning update and update with oquiprobable policy by tuning the value of 17, like softmax and mellowmax.
B More Non-Expansion Results
Here, we first show that more general form of resmax operator is not a non-expansion, except when its form is equal to resmax. Then, show that the escort transform operator recently presented in the policy gradient literature is not non-expansion.
B.1 Is the General Form of Resmax Non-Expansion?
A general form of resmax operator with max action gap, v, replaced by a gap emphasis function g(v) up for E I is defined follows
rm(Z,n)
Σ
d+ d+ (20 2. tTg
Σ
Σ
Zj [
+xj
d+ (xx - 22)0 ()(()))((s)
+
x, x.)
(E()))
(14)
where G(s) = { max; x; = Ji} and x; = max; Zi.
Now we want to show that if p > resmax with gap emphasis function g(v) = uP is no longer a non-expansion.
Let n 1, = 2, and 81 = 12 21 2 We take the derivative of resmax with respect x1 and get
rm E = 1) -po- + 8
+
2+ (1- p)0p
dx
82+
(2 61)
Now we compute Örm(A, 17 0x2
i Orz
rm 11 1) (p 1)0p -
(p + 3)0p + 8:P +
(2 + 2
(2 + +2)2
Note that 0rm(7, 11 = 1)/Orz 2 0 when 6, > 0 and p > -3. We will use this fact later. Since we have Vrm(ū, n = 1), we will define and compute (z),
i dz
2 (1 PP)
(p 1)0 -2 2
[6;); =
rm = 1)
rm ( 1)
+
Or
(2+ 87)
(2 + 8
16

---

Published in Transactions on Machine Learning Research (12/2023)
We would like to note that if p 1 then L1(51) for all 81 0. Now will show that if p then I1 (61) > 1 for some 81. Since all the terms of 8 (7, 17 = 1) are positive, can remove the absolute value
8zg
from this term. Now let 81 = (2c/ (v 1))²/P for E (0), which when p > is posilive, then we have
(6,) = 2+ + (1 - (2c) 1))
+ (p - 1)(2c/(-1))--
(2 + 2c/lp 1))²
(2 + 1))2
(p 1)(2c/ (p- -p-1) -2
(p - 1)(2c/(p (122/(p-111 -2
4c -4 (2 + 2c/(p 1))²
+
+ 1.
(2 + 20/ (p- 1))²
+ 2c/(p/(p 1))2
Now for c C>> we have that
4e +1>1
I I1())
(2 + 2c/(p 1))2
where the first inequality holds by the definitions of L1 and L1 (ō1) and the second strict inequality holds since 4c 4 0 when c > Putting this all together, we have shown that if p > 1 for the gap emphasis function g(v) oP and n then there exists a vector I IR² such that if 61 = 22 x1 (2c/(v - 1))/(1)))// for (1,, then L, 1. This means that if put more emphasis the gaps, resmax is longer non-expansion.
This proof can be generalized for arbitrary d by letting j = arg max, H; d and considering the special case when 51 = 62 =.. = 5 s (ed/(p- - 1/, Thus the current gap emphasis function for resmax is tight in the sense that adding more emphasis to the gaps would mean resmax is no longer a non-expansion.
B.2 Is the Escort Transform a Non- Von-Expansion?
In this section, we will show that a version the escort transform does not have the non-expansion property. The escort transform (Mei al., 2020) was introduced for policy gradient methods (with action preferences), but can naturally be defined for action-values
11(s,a)PP
and p 2 Ta(a)s) = E,la(s,b)'' for all (s,a) ES
Now we will analyze the following escort operator ct(g(s, (.p) = E. Te(d'(s)((s, a'). Now let H E I22 be two dimensional vector that lies in the first Cartesian quadrant, meaning the vector contains only non-negative elements. Then the escort operator becomes
P+1 + 2
et(ī,p) =
x + x2
Note the absolute values are dropped because Z is assumed to be in the first Cartesian quadrant. Now taking dcrivatives with respect to both Ii and x2 we get
of (of
+ (p + 1) pm-tl)
ct(E,p)=
(of + x2
and
pt = J2 (1z(0) + x2)) par parf(1 I2))
Or
(x" + x²)
Now we want compute the Lipschitz constant with respect to the max norm of ct(E,)), L, & Day et(z,p)). Let 7 = [1, 0] and 1, we have
a ct
cF
p)
1-11 11-1 + (1 ()) (1)())+))) 10- 00-1 -00)
- > 1.
L
(1 + 01)²
0
Thus we can conclude that the escort operator does not have the nlon-expansion property since we have an instance when I 1.
17

---

Published in Transactions on Machine Learning Research (12/2023)
C Experimental Configuration
C.1
Computational Infrastructure
We ran our experiments on a compute cluster. Each job used a single CPU core, except for Atari experiments that used GPUs. The compute cluster allocated CPUs based on availability. The possible options were 2.1Ghz Intel CPUs with model numbers E3-2683 V4 Broadwell, E7-4809 V4 Broadwell, or Platinum 8160F Skylake, as well 2.4Ghz Intel Platinum 8260 Cascade Lake. For GPU experiments, we used V100 Volta GPU. We also requested 400MB for the labular selting. In the case of deep RL, requested 16GB of memory for Atari environments. Different algorithms and exploration heuristics within one environment used the same configurations of resources.
C.2 Logging Procedure
To save returns and steps per episode, average returns the number of steps per episode for all of the episodes that have been finished in a specific number of steps that we call log-interval. To elaborate, returns and the number of steps per episode for all the episodes that are finished in the log interval will be accumulated and averaged. We only store this final averaged value. For instance, if we set the total number of steps to 100, 000, and define a log-interval of 1, 000, then 100 values will be stored. This way of storing the results of our experiments can save us both memory and space. At the same time, the stored results are proportional the performance of each of the employed algorithms. We use log-interval of 000 for all our tabular experiments and log-interval of 100, 000 for all the Atari experiments.
C.3 Hyperparameters
In this section, we present the hyperparameters that are used in our experiments and the reason for selecting them. One of the hyperparameters needed to set fairly across different soft-greedy operators were their respective exploration parameters, such as n and To do this, swept over large set of hyperparameters for each soft-greedy operator to make sure that the exploration parameter with the near-best performance resides in this set.
the deep RL setting, we use the DQN algorithm. We chose a fixed set of parameters that work well across three benchmark environments. These parameters are presented Appendix Table We swept over
three different step sizes across all our experiments: 0.0005, 0.0001, 0.00005. Our experiments with these step sizes show that step size of 0.0001 works best across all large-scale atari environments. We present the results in this paper based on these step sizes.
Parameter Name
Fixed Value
Optimizer
Adam
81
0.9
82
0.999
e for Adam
10-5
Batch size
64
Buffer size
100, 000
Number of training steps per ileration
Target network update frequency
1, 000
Number of steps before learning starts
50, 000
7
0.99
Table 1:
The fixed parameters used to run DQN experiments.
For implementing the neural networks we used the PyTorch framework. We used a convolutional neural network as the function approximation, with three convolutional layers that are followed by two fully connected layers. ReLU is used as the activation function for these networks. Convolutional layers have 32, 64, and 64 filters; kernel size of 4, and 3, and stride of 4, 2, and 1, respectively. The first. fully connected includes
18

---

Published in Transactions on Machine Learning Research (12/2023)
512 neurons, and the second one outputs the action values. We use uniform Xavier initialization to initialize the weights of the worw
19


Reviews:
[{"review_id": "IxaFinrxyo", "reviewer": "Reviewer_G7Vg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The idea of alternative operator to Softmax, Mellowmax and greedy operator is interesting, since it can potentially point to promising new directions to learn Q-functions while ensuring better exploration. This paper should be of interest to general RL audience, including theorist and practitioners.\n\nHowever, overall I feel the results presented in the paper are a bit underwhelming. The theoretical result is essentially the non-expansion property, while advantages on better exploration are more or less based on empirical arguments. The empirical gains are most presented on tabular domains, whereas the improvements on deep RL domains are not very statistically significant. This makes it difficult to consider the Resmax operator as a well-balanced alternative to other existing operators.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes Resmax, a control operator as an alternative to Softmax, Mellowmax and classic greedy operator. Resmax averages over Q-functions using a policy that is defined using the action gaps between actions. The paper shows that Resmax is a non-expansion, and entails better exploration property compared to Mellowmax. The paper demonstrates some empirical gains on tabular hard exploration problems and improvements in some deep RL settings.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I have some questions that I hope the authors can respond to.\n\n=== **Eqn 4** ===\n\nIs there a typo in Eqn 4? The Mellowmax operator should have $x_i$ in the exponents, while the current form in the paper only has $x_i$ inside the log.\n\n=== **Non-expansion vs. contraction of the operator** ===\n\nI think a major theoretical bottleneck here is that Mellowmax operator is contractive, regardless of the temperature parameter $w$, while Greedy operator as a limiting case is always contractive too. This implies that when iterating both operators on a set of Q-functions, we expect convergence. This might be a major motivation to replace Softmax by Mellowmax, since the former does not guarantee convergence as shown by Asadi et al.\n\nSince Resmax is only a non-expansion, there is no formal guarantee that it converges during iterations. Will we be able to identify cases where it does not converge? Like the two fixed point of Softmax case identified in Asadi. I think this would be a valuable finding in general. Meanwhile, since Resmax is not proved strictly contractive or convergent, we may not be able to expect an improvement over alternative operators in terms of value based learning.\n\n=== **Other properties of the operators** ===\n\nI feel there can be many more theoretical properties of the operator worth discussing. What's the fixed point? Is the fixed point unique? At fixed point do we recover optimal policy or entropy regularized policy? These properties will distinguish Resmax from Softmax and greedy even in case the Resmax operator is proved convergent, and will be useful to show.\n\n=== **Empirical performance Fig 2 & 4** ===\n\nMinor: x-axis in Fig 2, do we mean all three parameters for different algorithms take the same value on the axis? Why not use the same temperature parameter to represent them all, as they represent similar quantities.\n\nIn Fig 4 on the right, why do we not have a comparison of the state visitation between Resmax and Mellowmax, are they very similar?\n\nFig 5, it'd be nice to have log plots for the right three algorithms, it is hard to tell the difference from the current plot.\n\n=== **Deep RL** ===\n\nFig 9: it looks like Resmax does not achieve significantly better performance compared to Softmax nor Mellowmax? In certain cases, eps-greedy performs quite well (and in the reviewer's experience, eps-greedy is a very strong baseline in Atari domains). The testing environments are breakout and asterix, which may not be the best testing benchmark for hard explorations. There are a number of exploration oriented games in the Atari suite, which eps-greedy struggles to solve, why not benchmark on those envs since we are comparing the exploration ability?\n\nFig 10: do we plot the final performance of these algorithms on a suite of games? It is a bit hard to tell what conclusion to make of the plot, since it seems that Resmax is just as sensitive to other algorithms wrt the temperature parameter, it is hard to argue that Resmax offers empirical gains in terms of final performance nor robustness.\n\n=== **Why Resmax** ===\n\nA central question is: why would one tend to use Resmax compared to alternative operators? Resmax is not provably convergent compared to Mellowmax. And although by design Resmax enjoys certain intuitive empirical properties such as no-early-commitment compared to Softmax, there should be a spectrum of operators that enjoy similar properties. Then, why the specific form of Resmax operator? Empirically, it is also hard to see the performance gains achieved by Resmax compared to others, over standard benchmark tasks. I think the paper can be further improved by strengthening both theoretical and empirical results.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ab82YGRbJ7", "reviewer": "Reviewer_YcUX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The paper is clearly written and structured. The presentation is straightforward. \n\n- The proposed operator is original and its benefits are clearly articulated.\n\n- The theoretical analysis supports the claimed properties of Resmax.\n\n- The empirical evaluation is thorough.\n\nWeaknesses\n\n- The main weakness is that for the large-scale problems tested by the authors, there doesn’t seem to be any real performance benefit in using Resmax. I don’t see this method being widely adopted in practice without a convincing demonstration of its empirical effectiveness.  I would be most curious about Resmax’s interaction with state-of-the-art exploration methods that can solve challenging tasks like Montezuma’s Revenge (and the other exploration domains presented). I would also be curious to see performance on deep RL tasks beyond Atari.\n\n- There is not much discussion which fosters understanding of Resmax's performance on the deep RL tasks.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce Resmax, a soft-greedy operator for value-based RL approaches that combines the advantages of the $\\epsilon$-greedy (full coverage of action space) and softmax (sampling prioritizes promising actions) operators while avoiding their downsides (value agnosticism and over-concentration, respectively). It shares these properties with the Mellowmax operator while retaining the computational efficiency of standard operators. Unlike the softmax operator, it is a non-expansion. Experiments are carried out comparing the performance of these operators in both the tabular and function approximation settings. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I would appreciate  experimental results and/or discussion regarding the combination of Resmax with strong exploration methods. Are such approaches complementary? \n\n- Any deep RL results beyond Atari would also be appreciated, though not necessary. \n\n- I would also like to see more discussion regarding why Resmax does not provide real gains on deep RL tasks---is there some property of a task/agent that would make Resmax more useful? \n", "broader_impact_concerns": "I do not have any broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ek0d44dzw2", "reviewer": "Reviewer_Lagu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Overall I like this paper a lot, the idea proposed by the authors is extremely straightforward but yet effective.\n- The paper is very clearly written and nicely structured, the authors begin by enumerating the limitations of the most commonly used soft-greedy operators (mainly $\\epsilon$-greedy and softmax, and to a lesser extent mellowmax) and very clearly shows how ResMax addresses each one of these limitations.\n- Theoretical results in the paper are sound, the proofs are quite easy to read.\n- On the theoretical results, I was also quite surprised that the general for of ResMax is not a non-expansion, do you have any intuition why this may be the case? Specifically why would placing more emphasize on the action gaps no longer make it a non-expansion?\n- What I particularly like about this paper is how the authors use very simple and clear toy examples to demonstrate the properties of ResMax and they were very effective in making the results of the paper intuitive. I think this is a practice that should be promoted more within the RL community.\n- One question I have for the authors is that it seems ResMax is limited to the setting of discrete action spaces, I was wondering if it is possible to extend the use of the operator to continuous action spaces. Since one advantage of a softmax is that it can be applied to both discrete and continuous action spaces", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors proposed a new soft-greedy operator for exploration in RL called Resmax, this operator uses the action-value gap while also avoids the overemphasize issue stemming from the exponential softmax operator. The authors furthered showed that their operator is a non-expansion. A series of toy examples are given to illustrate the intuition behind the new operator, the authors also demonstrated the effectiveness of their approach in DRL settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "One change I would recommend is for the authors to bring forward the definition of a non-expansion to the background section of the paper since it is a fairly important property of the ResMax operator and in the current paper does not actually come up until nearly towards the end.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new soft-greedy operator, called resmax, in which the probability of taking an action is (inversely) proportional to the gap between their (estimated) value and the one of the (estimated) optimal action. Resmax operator tries to combine the benefits of previous soft-greedy operators avoiding their main flaws, i.e., random dithering for $\epsilon$-greedy, over-concentration for softmax, and high computational cost for mellowmax. The resulting operator is simple, intuitive, and easy to plug in the existing value-based algorithms, while bringing good theoretical properties.

The reviewers found the resmax operator to be interesting and well-presented, providing a mostly positive evaluation of the paper. On the other hand, all the reviewers ultimately agreed that the experimental evaluation, especially in deep RL domains, does not fully support the empirical superiority of resmax against previous operators, which would make it hard to convince practitioner to employ resmax in place of more established operators, potentially limiting the impact of the work. I think this is a valid concern. However, as one reviewer rightly noted, the latter is "largely orthogonal to the acceptance criteria for TMLR".

Hence, I am providing an Accept recommendation while encouraging the authors to implement a few minor changes to the manuscript, which are listed below.

- Change the last sentence in the Introduction to avoid overstating the empirical superiority of resmax;
- Fix equation 4;
- Highlight why non-expansion of the operator is enough to ensure convergence of value-based methods;
- Explicitly comment on why mellowmax curve is not in Figure 4;
- Incorporate the new version of Figure 5 with log scale;
- Please use \citep when authors names are not part of the sentence;
- Background, second paragraph: to learns -> to learn;
- In the plots, consider using line patterns to ease the visual inspection of the curves and reporting bootstrapped confidence intervals instead of standard errors;
- Figure 5, 8, 9 can be reduced in size without hurting legibility.
Decision: Accept (poster)