DELAYED GEOMETRIC DISCOUNTS: AN ALTERNATIVE CRITERION FOR REINFORCEMENT LEARNING
Year: 2021

Abstract:
The endeavor of artificial intelligence (AI) is to design autonomous agents capable of achieving complex tasks. Namely, reinforcement learning (RL) proposes a theoretical background to learn optimal behaviors. In practice, RL algorithms rely on geometric discounts to evaluate this optimality. Unfortunately, this does not cover decision processes where future returns are not exponentially less valuable. Depending on the problem, this limitation induces sample-inefficiency (as feed-backs are exponentially decayed) and requires additional curricula/exploration mechanisms (to deal with sparse, deceptive or adversarial rewards). In this paper, we tackle these issues by generalizing the discounted problem formulation with a family of delayed objective functions. We investigate the underlying RL problem to derive: 1) the optimal stationary solution and 2) an approximation of the optimal non-stationary control. The devised algorithms solved hard exploration problems on tabular environment and improved sample-efficiency on classic simulated robotics benchmarks.

heading: 1 INTRODUCTION
text: In the infinite horizon setting, and without further assumptions on the underlying Markov Decision Process (MDP), available RL algorithms learn optimal policies only in the sense of the discounted cumulative rewards (Puterman, 2014). While the geometric discounting is well suited to model a termination probability or an exponentially decaying interest in the future, it is not flexible enough to model alternative weighting of the returns. Consider for example settings where the agent is willing to sacrifice short term rewards in favor of the long term outcome. Clearly, for such situations, a discounted optimality criterion is limited and does not describe the actual objective function.
This is particularly true in hard exploration tasks, where rewards can be sparse, deceptive or adversarial. In such scenarios, performing random exploration can rarely lead to successful states and thus rarely obtain meaningful feedback. Geometric discounts tend to fail in such scenarios. Consider for example the U-maze environment in Figure 1a, where the reinforcement signal provides a high reward (+1) when reaching the green dot in the bottom arm, a deceptive reward (+0.9) when reaching the blue dot in the upper arm, and a negative reward (−1) for crossing the red corridor. If the agent is only interested in the long term returns, then the optimal control should always lead to the green dot. However, depending on the initial state, optimal policies in the sense of the discounted RL problem are likely to prefer the deceptive reward due to the exponentially decaying interest in the future (Figure 1b).
Naturally, higher discount factors are associated with optimal policies that also optimize the average returns (Blackwell, 1962), which can solve in principle the described hard exploration problem. However, in practice, such discount values can be arbitrarily close to 1 which entails severe computational instabilities. In addition, and particularly in continuous settings or when tasks span over extremely long episodes, discount-based RL approaches are sample-inefficient and are slow at propagating interesting feed-backs to early states.
In this paper, we generalize the geometric discount to derive a variety of alternative time weighting distributions and we investigate the underlying implications of solving the associated RL problem both theoretically and practically. Our contributions are twofold. First, we introduce a novel family that generalizes the geometrically discounted criteria, which we call the delayed discounted criteria. Second, we derive tractable solutions for optimal control for both stationary and non-stationary
policies using these novel criteria. Finally, we evaluate our methods on both hard exploration tabular environments and continuous long-episodic robotics tasks where we show that:
1. Our agents can solve the hard exploration problem in a proof of concept setup. 2. Our methods improve sample-efficiency on continuous robotics tasks compared to Soft-
Actor-Critic.
Figure 1c showcases how non-geometrically discounted criteria impacts the profile of optimal value function in the U-maze example.

heading: 2 REINFORCEMENT LEARNING WITH NON GEOMETRIC DISCOUNT
text: Consider an infinite horizon Markov Decision Process (MDP) M = {S,A,P, r, γ, p0}, where S and A are either finite or compact subsets of respectively Rd and Rd′ , (d, d′) ∈ N2, P : S × A → ∆(S) is a state transition kernel1, r : S × A → R is a continuous reward function, p0 ∈ ∆(S) an initial state distribution and γ ∈ (0, 1) is the discount factor. A policy π is a mapping indicating, at each time step t ∈ N, the action at to be chosen at the current state st. The goal of geometrically discounted reinforcement learning algorithms is to optimize the discounted returns:
L(π, r) := Eπ,p0 [ ∞∑ t=0 γtrt ] ; rt := r(st, at)
where Eπ,p0 denotes the expectation over trajectories generated in M using the policy π and initialised according to p0.
In this section, we present our methods. First, Section 2.1 introduces our delayed discounted family of optimality criteria. Then, Section 2.2 investigates the optimization of the linear combination of this new family in the context of stationary policies. Finally, Section 2.3 generalizes the optimal control to non-stationary policies.

heading: 2.1 BEYOND THE GEOMETRIC DISCOUNT: A DELAYED DISCOUNTED CRITERION
text: We propose to investigate a particular parametric family of optimality criteria that is defined by a sequence of discount factors. For a given delay parameter D ∈ N, we define the discount factors γd ∈ (0, 1) for any integer d ∈ [0, D] and we consider the following loss function:
LD(π, r) := Eπ,p0 [ ∞∑ t=0 ΦD(t)rt ] where ΦD(t) := ∑ {ad∈N}Dd=0
such that ∑ d ad=t
N∏ d=0 γadd (1)
This class of optimality criteria can be seen as a generalization of the classical geometric discount. In fact, we highlight that for D = 0, Φ0(t) = γt0 which implies that L0(π, r) = L(π, r) for any policy π and any reward function r. In Figure 2, we report the normalized distribution of the weights ΦD(t) (i.e. y(t) = ΦD(t)∑
i ΦD(i) ) over time as we vary the parameter D ∈ {0, . . . , 9}. Notice how the mode of
the probability distribution is shifted towards the right as we increased the delay, thus putting more weights on future time steps.
1∆(S) denotes the set of probability measures over S
Intuitively, the proposed criterion (LD) describes the goal of an agent that discards short term gains in favor of long term discounted returns. In the following, we consider yet a more diverse problem formulation by using a linear combination of the delayed losses. Let Lη be the following objective function defined as:
Lη(π, r) := Eπ,p0 [∑
t
η(t)rt
] such that η(t) = D∑ d=0 wdΦd(t) (2)
where the depth D ∈ N and the coefficients wd ∈ R for any d ∈ [0, D] are known. In general, the optimal control in the sense of Lη is not stationary. However, learning solutions that admit compact representations is crucial for obvious computational reasons. In the following we propose to learn either the optimal stationary solution, or to approximate the optimal control with a non-stationary policy over a finite horizon followed by a stationary one.

heading: 2.2 OPTIMAL STATIONARY POLICIES
text: In this section, we propose to learn stationary solution using a policy-iteration like algorithmic scheme. As in the classical setting, the goal is to learn a control π∗ that maximizes the state-action value function Qπη :
Qπη (s, a) = D∑ d=0 wdQ π d (s, a) where Q π d (s, a) := Eπ [ ∞∑ t=0 Φd(t)rt|s0, a0 = s, a ]
(3)
This is done by iteratively evaluatingQπη and then updating the policy to maximize the learned value function. Due to the linearity illustrated in Equation 3, the policy evaluation step is reduced to the estimation of Qπd . In geometrically discounted setting, the value function is the fixed point of the Bellman optimality operator. Luckily, this property is also valid for the quantities Qπd :
Proposition 1 For any discount parameters (γd)Dd=0, the value functions QπD is the unique fixed point of the following γD-contraction:
[TDπ (q)](s, a) = Es′∼P(s,a) a′∼π(s′)
[ r(s, a) + D−1∑ d=0 γdQ π d (s ′, a′) ] ︸ ︷︷ ︸
:= rπD(s, a)
+γDEs′∼P(s,a) a′∼π(s′)
[ q(s′, a′) ] . (4)
The valueQπD of a policy π with respect to the delayed criterionLD is the state-action value function of the same policy using an augmented policy dependent reward rπD w.r.t. the γD-discounted returns.
Intuitively, the instantaneous worth of an action (rπD) is the sum of the environments’ myopic returns (r(s, a)) and the long term evaluations (with lower delay parameters (Qπd )d<D).
This has the beneficial side-effect of enhancing sample efficiency as it helps the agent to rapidly back-propagate long-term feed-backs to early states. This reduces the time needed to distinguish good from bad behaviors, particularly in continuous settings where function approximation are typically used to learn policies. This is discussed in details in Section 4.2.
Similarly to standard value based RL algorithms, given a data set of trajectories D = {s, a, s′}, the value function Qπd can be approximated with parametric approximator Qθd by optimising J Q d (θ):
JQD (θ) = E s,a,s′∼D a′∼πφ(s′)
[1 2 ( Qθ − (r(s, a) + D∑ d=0 γdQθ̄d(s ′, a′))2 )] (5)
As for the policy update step, inspired from the Soft-Actor-Critic (SAC) algorithm, we propose to optimize an entropy regularized soft Q-value using the following loss where α is a parameter:
Jπη (φ) = −Es∼D,a∼πφ [ D∑ d=0 wdQθ̄d(s, a)− α log(πφ(a|s)) ]
(6)
We use Equations 5 and 6 to construct Algorithm 1, a generalization of the SAC algorithm that approximates optimal stationary policies in the sense of Lη . In practice, this can be further improved using the double Q-network trick and the automatic tuning of the regularization parameter α. This is discussed in Appendix A.1. Unfortunately, unlike the geometrically discounted setting, the policy improvement theorem is no longer guaranteed in the sense of Lη . This means that depending on the initialization parameters, Algorithm 1 can either converge to the optimal stationary control or get stuck in a loop of sub-optimal policies. This is discussed in detail in Section 4.1.
Algorithm 1 Generalized Soft Actor Critic
1: Input: initial parameters (θd)Dd=0, φ, learning rates (λd)Dd=0, λπ , pollyak parameter τ 2: initialise target network θ̄d ← θd and initialise replay buffer D ← ∅ 3: for each iteration do 4: for each environment step do 5: at ∼ πφ(st), st+1 ∼ P(st, at), D ← D ∪ {(st, at, st+1, ct)} 6: for d ∈ [0, D] do 7: for each Qd gradient step do 8: update parameter θd ← θd − λd∇̂θdJ Q d (θd) and update target θ̄d ← τ θ̄d + (1− τ)θd
9: for each policy update do 10: update policy φ← φ− λπ∇̂φJπη (φ) 11: Return: πφ, (Qθ̄d) D d=0

heading: 2.3 APPROXIMATE OPTIMAL CONTROL
text: In the general case, the optimal control is not necessarily stationary. Consider the problem of learning the optimal action profile a∗0:∞ which yields the following optimal value function:
V ∗η (s) := max a0:∞ Ea0:∞ [ ∞∑ t=0 η(t)rt|s0 = s ] , (7)
where Ea0:∞ is the expectation over trajectories generated using the designated action profile. In this section we propose to solve this problem by approximating the value function from Equation 7. This is achieved by applying the Bellman optimality principle in order to decompose V ∗η into a finite horizon control problem (optimising a0:H with H ∈ N) and the optimal value function V ∗fH+1(η)
where f is an operator transforming the weighting distribution as follows:[ f(η) ] (t) := D∑ d=0 ( γd D∑ j=d wj ) Φd(t) = 〈Γ ·w,Φ(t)〉
where Γ :=  γ0 γ0 γ0 ... γ0 0 γ1 γ1 ... γ1 0 0 γ2 ... γ2 ... . . . ...
0 0 0 ... γD  ; w :=  w0 w1 w2 ... wD  ; Φ(t) :=  Φ0(t) Φ1(t) Φ2(t) ... ΦD(t)  For the sake of simplicity, let 〈1, η〉 := ∑D d=0 wd and [ fn(η) ] (t) := 〈Γn ·w,Φ(t)〉 for any n, t ∈ N.
Proposition 2 For any state s0 ∈ S, the following identity holds:
V ∗η (s0) = max a0:H
{ Ea0:H [ H∑ t=0 〈1, f t(η)〉rt ] + Ea0,a1,..,aH [ V ∗fH+1(η)(sH+1) ]} (8)
As a consequence, the optimal policy in the sense of Lη is to execute the minimising arguments a∗0:H of Equation 8 and the execute the optimal policy in the sense LfH+1(η). Unfortunately, solving LfH+1(η) is not easier than the original problem. However, under mild assumptions, for H large enough, this criterion can be approximated with a simpler one. In order to derive this approximation, recall that the power iteration algorithm described by the recurrence vk+1 = Γ·vk‖Γ·vk‖ converges to the unit eigenvector corresponding to the largest eigenvalue of the matrix Γ whenever that it is diagonalizable. In particular, if 0 < γD < ... < γ0 < 1 and v0 = w, then Γ is diagonalizable, γ0 is it’s largest eigenvalue and the following holds:
lim n→∞ vn+1 = lim n→∞ Γn ·w∏ k≤n ‖Γ · vk‖ = [1i=0]i∈[0,D] and lim n→∞ V ∗fn(η)(s)∏ k≤n ‖Γ · vk‖ = V ∗(s) (9)
Under these premises, the right hand term of the minimisation problem in Equation 8 can be approximated with the optimal value function in the sense of the classical RL criterion L (which optimal policy can be computed using any standard reinforcement learning algorithm in the literature).
Formally, we propose to approximate Equation 8 with a proxy optimal value function Ṽ ∗η,H(s0):
Ṽ ∗η,H(s0) = max a0:H
{ Ea0:H [ H∑ t=0 〈1, f t(η)〉rt ] + ( ∏ k≤H ‖Γ · vk‖ ) Ea0,a1,..,aH [ V ∗(sH+1) ]} (10)
A direct consequence of Equation 9 is that limH→∞ Ṽ ∗η,H(s) = V ∗ η (s) for any state s ∈ S. In addition, for a given horizonH , the optimal decisions in the sense of the proxy problem formulation are to execute for the first H steps the minimising arguments a∗0:H of Equation 10 (they can be computed using dynamic programming) and then execute the optimal policy in the sense of the γ0-discounted RL (which can be computed using any standard RL algorithm).
Algorithm 2 H-close optimal control 1: Compute π∗, V ∗ ← POLICY ITERATION and initialize v0 ← w 2: for t ∈ [1, H] do 3: Compute vt ← Γ · vt−1 4: Initialise VH+1(s)← V ∗(s) · ∏ k≤H ‖Γ · vk‖
5: for t ∈ [H, 0] do 6: Solve πt(s)← arg maxa ‖vt‖ · c(s, a) + Es′∼P(s,a) [ Vt+1(s ′) ]
7: Compute Vt(s)← ‖vt‖ · c(s, πt(s)) + Es′∼P(s,πt(s)) [ Vt+1(s ′) ] 8: Return: (πt)t∈[0,H], π∗

heading: 3 RELATED WORK
text: Bridging the gap between discounted and average rewards criteria. It is well known that defining optimality with respect to the cumulative discounted reward criterion induces a built-in bias against policies with longer mixing times. In fact, due to the exponential decay of future returns, the contribution of the behaviour from the T th observation up to infinity is scaled down by a factor of the order of γT . In the literature, the standard approach to avoid this downfall is to define optimality with respect to the average reward criterion L̄ defined as :
L̄(π, c) := lim T→∞
1 T Eπ,p0 [ T∑ t=0 rt ] This setting as well as dynamic programming algorithms for finding the optimal average return policies have been long studied in the literature (Howard, 1960; Veinott, 1966; Blackwell, 1962; Puterman, 2014). Several value based approaches (Schwartz, 1993; Abounadi et al., 2001; Wei et al., 2020) as well as policy based ones (Kakade, 2001; Baxter & Bartlett, 2001) have been investigated to solve this problem. These approaches are limited in the sense that they require particular MDP structures to enjoy theoretical guarantees.
Another line of research, is based on the existence of a critical discount factor γcrit < 1 such that for any discount γ ∈ (γcrit, 1) the optimal policy in the sense of the γ-discounted criterion also optimises the average returns Blackwell (1962). Unfortunately, this critical value can be arbitrarily close to 1 which induces computational instabilities in practice. For this reason, previous works attempted to mitigate this issue by increasing the discount factor during training (Prokhorov & Wunsch, 1997), learning higher-discount solution via learning a sequence of lower-discount value functions (Romoff et al., 2019) or tweaking the reinforcement signal to equivalently learn the optimal policy using lower discounts (Tessler & Mannor, 2020).
Exploration Strategies. Another line of research attempted to tackle the hard exploration problem by further driving the agents exploration towards interesting states. Inspired by intrinsic motivation in psychology (Oudeyer & Kaplan, 2008), some approaches train policies with rewards composed of extrinsic and intrinsic terms. Namely, count-based exploration methods keep track of the agents’ past experience and aim at guiding them towards rarely visited states rather than common ones (Bellemare et al., 2016; Colas et al., 2019). Alternatively, prediction-based exploration define the intrinsic rewards with respect to the agents’ familiarity with their environments by estimating a dynamics prediction model (Stadie et al., 2015; Pathak et al., 2019). Other approaches maintain a memory of interesting states (Ecoffet et al., 2019; 2021), trajectories (Guo et al., 2019) or goals (Guo & Brunskill, 2019). Ecoffet et al. (2019; 2021) first return to interesting states using either a deterministic simulator or a goal-conditioned policy and start exploration from there. (Guo et al., 2019) train a trajectory-based policy to rather prefer trajectories that end with rare states. Guo & Brunskill (2019) revisit goals that have higher uncertainty. Finally, based on the options framework (Sutton et al., 1999), options-based exploration aims at learning policies with termination conditions—or macro-actions. This allows the introduction of abstract actions, hence driving the agents’ exploration towards behaviours of interest (Gregor et al., 2016; Achiam et al., 2017).

heading: 4 EXPERIMENTS
text: In this section, we evaluate the performance of the proposed algorithm. Our goal is to answer the following questions:
• How does the different parameters impact the ability of the proposed algorithms to solve hard exploration problems?
• How does the proposed algorithm impact the performance in classical continuous control problems?

heading: 4.1 HARD EXPLORATION PROBLEMS
text: In this section we investigate our ability to approximate the solution of the proposed family of RL problems (w.r.t. the optimality criterion LD) in discrete hard exploration maze environments. We
fix the discount factor values to γi = 0.99− i1000 throughout the experiments. This guarantees that Γ is diagonalizable and that its largest eigenvalue is γ0 = 0.99. We evaluate the performance of both Algorithms 1 and 2 as we vary the depth parameter and as we increase the non stationarity horizon.
In Figure 3 we reported the shapes of the mazes as well as the used reinforcement signal in each of them. We used a sparse signal where the green dots represent the best achievable reward, the blue dots are associated with a deceptive reward and the red dots are associated with a penalty. This setting is akin to hard exploration problems as the agent might learn sub-optimal behaviour because of the deceiving signal or because of the penalty.
Stationary solutions: We start by evaluating the performances of the learned policies using Generalised Soft Actor Critic (GSAC) (Algorithm 1). As discussed earlier, unlike the geometrically discounted setting, the policy update is not guaranteed to improve the performances. For this reason, depending on the initialisation, the algorithm can either converge to the optimal stationary policy, or get stuck in a sequence of sub-optimal policies.
In Figure 4, we reported the performances of GSAC as we varied the depth hyper-parameter D using two initialisation protocol. The blue curves are associated with a randomly selected initial parameters while the red curves are associated with experiment were the policy is initialised with the solution of the geometrically discounted problem. The solid lines correspond to the average reward while the dashed lines correspond to the Φd weighted loss (as computed in Ld). In all experiments, the expectations were averaged across 25 runs of the algorithm using trajectories of length 4000 initialised in all possible states (uniform p0) . A common observation is that for a depth D higher than 6 7 the algorithm was unstable and we couldn’t learn a good stationary policy in a reliable way. However, the learned stationary policies with even a relatively shallow depth parameter yielded reliable policies that not only maximise the ΦD weighted rewards but also improved the average returns. Notice how the baseline (D = 0, i.e. the geometrically discounted case) always under-performs when compared to the learned policies for a depth parameter around 3.
We also observe that the algorithm is sensitive to the used initialisation. Using the optimal policy in the sens of the geometrically discounted objective (red curves) helped stabilise the learning procedure in most cases: this is particularly true in the random maze environment where a random initialisation of the policy yielded bad performances even with a low depth parameter. This heurestic is not guaranteed to produce better performances in all cases, notice how for D = 7 in the T-maze, a random initialisation outperformed this heuristic consistently.
Optimal Control Approximation: In this section we investigate the performances of the policies learned using Algorithm 2 as we vary the non stationarity horizon H for three depth parameters D ∈ {5, 10, 15} (respectively the red, blue and green curves). As in the last experiment, we reported the expected returns as the average rewards using the continuous lines and as the ΦD weighted rewards using dashed lines. As a baseline, we can observe in each figure the performances of the optimal policy in the sense of the geometrically discounted criterion when the non-stationarity
horizonH = 0. We also reported the performances of the best stationary policy learned using GSAC for a depth parameter D = 5.
The first notable observation is that increasing the hyper-parameter H does indeed improve performances. In addition, we notice that the maximum possible improvement is reached using a finite horizon H (i.e. the maximising argument of both Ṽη,H and V ∗η are the same). Intuitively, the H non stationary steps enable the agent to get to an intermediate state from which the optimal policy in the sense of the discounted RL formulation can lead to the state with the highest rewards. This explains the effectiveness of current hard exploration RL algorithmsEcoffet et al. (2019); Eysenbach et al. (2019): by learning a policy that reaches interesting intermediate state, these methods are implicitly learning an approximate solution of Ṽη,H .
In the case of D = 5, the obtained performances using Algorithm 2 converged to the performances of the optimal stationary policy obtained using GSAC in both the random and T-maze. On the other hand, unlike GSAC, the H-close algorithm is not sensitive to initialisation: in fact notice how even for arbitrarily high depth parameter (D = 15) the learned policy ends up saturating the average and the ΦD weighted rewards. More interestingly, increasing the depth can be beneficial as we observe empirically that for higher depth parameters, the non-stationarity horizon required to achieve the best possible performances decreases.

heading: 4.2 SIMULATED CONTINUOUS CONTROL BENCHMARKS
text: In this section, we propose to evaluate our methods on several continuous robotics domains. To this end, we consider 5 different environments within the MUJOCO physics engine (Todorov et al., 2012): Walker2d, Hopper, HalfCheetah, Ant and Humanoid. As results in the tabular settings showed that there exists a threshold beyond which increasing the value of the delay D hurts the performance, we fix D = 1, where only two discount factors γ0 and γ1 are used. We compare our proposed methods to the classic SAC algorithm to investigate the implications of using both critics Qθ0 and Qθ1 .
In Figure 6, we report the average rewards obtained by the different agents over time. Note that in all the environments, the GSAC agents are faster in collecting positive rewards than the SAC
agents. The main reason behind this is that the critic tends to discern good from bad actions in the GSAC agents faster than the SAC agents. In fact, on the one hand, the SAC agents spend more time uncertain about the quality of their actions for a given state, and thus need more time and more experience to make the estimations of their critic more accurate and reflecting the true long-term value of an action. On the other hand, in the GSAC agents, Qθ1 takes into consideration the estimate of the actions by Qθ0 , and thus there is less uncertainty about their quality.

heading: 5 CONCLUSION
text: Designing human-like autonomous agents requires (among other things) the ability to discard short term returns in favor of long term outcomes. Unfortunately, existing formulations of the reinforcement learning problem stand on the premise of either discounted or average returns: both providing a monotonic weighting of the rewards over time.
In this work, we propose a family of delayed discounted objective functions that captures a wide range of non-monotonic time-preference models. We analyse the new formulation to construct 1) the Bellman optimality criterion of stationary solution and 2) a feasible iterative scheme to approximate the optimal control. The derived algorithms successfully solved tabular hard exploration problems and out-performed the sample efficiency of SAC in various continuous control problems; thus closing the gap between what is conceivable and what is numerically feasible.
BROADER IMPACT STATEMENT
Reinforcement learning provides a framework to solve complex tasks and learn sophisticated behaviors in simulated environments. However, its incapacity to deal with very sparse, deceptive and adversarial rewards, as well as its sample-inefficiency, prevent it from being widely applied in real-world scenarios. We believe investigating the classic optimality criteria is crucial for the deployment of RL. By introducing a novel family of optimality criteria that goes beyond exponentially discounted returns, we believe that we take a step towards more applicable RL. In that spirit, we also commit ourselves to releasing our code soon in order to allow the wider community to extend our work in the future.

Review summaries:
The paper proposes an optimality criterion for RL that is generalizes the standard exponential discount commonly used in a large volume of RL research. In particular, the paper proposes substituting the standard discount term $\gamma^t$ by a generalized discount term $\Phi_D(t)$ built from a set of discounts $\gamma_0,\ldots,\gamma_D$. The discount terms $\Phi_D$ can be defined recursively as   - $\Phi_0(t)=\gamma_0^t$ (corresponding to the standard exponential discounting); - $\Phi_D(t)=\sum_{k=0}^t\gamma_D^k\Phi_{D-1}(t-k)$  The consideration of non-uniform discounting implies that the resulting optimal policies may, in general, be non-stationary. Nevertheless, the paper introduces an equivalent to the $Q$-functions in the standard RL formulation and shows that these functions can be computed using standard dynamic programming. Letting  $Q_d^\pi(s,a)=E\left[\sum_{t=0}^\infty\Phi_d(t)r_t\mid s_0=s,a_0=a\right]$,  it follows that $Q_0^\pi$ corresponds to the standard $Q^\pi$ with discount factor $\gamma_0$, and each $Q_d^\pi$ can be computed from $Q_{d-1}^\pi$ using a dynamic programming operator $T_\pi^d$ that has $Q_d^\pi$ as its unique fixed point. These functions can also be computed using existing RL algorithms as part of a generalized policy iteration cycle that---upon convergence---may yield an "optimal stationary policy" (convergence is not guaranteed, though).  The paper also proposes an approximate policy that breaks down the value function for the problem into two terms---one that accounts for the rewards in the first $H$ steps and the other for the remaining steps. Under mild conditions on the discounts $\gamma_0,\ldots,\gamma_D$, the latter term can be approximated by the standard optimal value function (with standard exponential discounting) for large $H$, allowing for the computation of an approximate (non-stationary) policy.

**Summary.** This paper proposes a generalization to the typical geometric discounting scheme in reinforcement learning. In particular, a new _delayed_ discount criterion is proposed (Equation 1) that captures traditional geometric discounting when $D=0$, but also allows for other kinds of temporal preferences. Figure 2 provides a clean visual illustrating the different kinds of discounting afforded for different settings of $D$. Following this, several natural questions emerge: What happens to the $\gamma$-contraction in repeated application of the Bellman Operator? Proposition 1 addresses this question by stating that a $\gamma_D$-contraction is still guaranteed. How might typical reinforcement learning algorithms handle this new discounting scheme? The paper addresses this question by introducing two new algorithms. The first (Algorithm 1) is a generalization of Soft Actor Critic, while the second (Algorithm 2) is a policy-based algorithm that approximates the finite horizon problem. The first set of experiments focus on three maze problems. Estimates of the expected returns of learned stationary policies are reported for Algorithm 1 under four conditions as the depth $D$ is changed from 0 up to 8. The main finding is that as $D$, returns do tend to increase up to a certain point, at which point it drops (around when $D = 6$ or $7$). Further experiments inspect the performance of Algorithm 2 in a similar way (Figure 5), and contrast the performance of Algorithm 1 with typical SAC on continuous control games like Ant-v2. On the chosen four games (Figure 6), the average reward of GSAC improves over SAC slightly, though the variance is arguably slightly higher.