Title: A State Representation for Diminishing Rewards
Venue: NeurIPS 2023
Accepted: yes
Keywords: reinforcement learning

successor features

successor representation

neuroscience
Abstract: A common setting in multitask reinforcement learning (RL) demands that an agent rapidly adapt to various stationary reward functions randomly sampled from a fixed distribution. In such situations, the successor representation (SR) is a popular framework which supports rapid policy evaluation by decoupling a policy's expected discounted, cumulative state occupancies from a specific reward function. However, in the natural world, sequential tasks are rarely independent, and instead reflect shifting priorities based on the availability and subjective perception of rewarding stimuli. Reflecting this disjunction, in this paper we study the phenomenon of diminishing marginal utility and introduce a novel state representation, the $\lambda$ representation ($\lambda$R) which, surprisingly, is required for policy evaluation in this setting and which generalizes the SR as well as several other state representations from the literature. We establish the $\lambda$R's formal properties and examine its normative advantages in the context of machine learning, as well as its usefulness for studying natural behaviors, particularly foraging.

A State Representation for Diminishing Rewards

Abstract

A common setting in multitask reinforcement learning (RL) demands that an agent rapidly adapt to various stationary reward functions randomly sampled from a fixed distribution. In such situations, the successor representation (SR) is a popular framework which supports rapid policy evaluation by decoupling a policy's expected discounted, cumulative state occupancies from a specific reward function. However, in the natural world, sequential tasks are rarely independent, and instead reflect shifting priorities based on the availability and subjective perception of rewarding stimuli. Reflecting this disjunction, in this paper we study the phenomenon of diminishing marginal utility and introduce a novel state representation, the λ representation ( λ R) which, surprisingly, is required for policy evaluation in this setting and which generalizes the SR as well as several other state representations from the literature. We establish the λ R's formal properties and examine its normative advantages in the context of machine learning, as well as its usefulness for studying natural behaviors, particularly foraging.

1 Introduction

The second ice cream cone rarely tastes as good as the first, and once all the most accessible brambles have been picked, the same investment of effort yields less fruit. In everyday life, the availability and our enjoyment of stimuli is sensitive to our past interactions with them. Thus, to evaluate different courses of action and act accordingly, we might expect our brains to form representations sensitive to the non-stationarity of rewards. Evidence in fields from behavioral economics [1, 2] to neuroscience [3] supports this hypothesis. Surprisingly, however, most of reinforcement learning (RL) takes place under the assumptions of the Markov Decision Process [MDP; 4], where rewards and optimal decision-making remain stationary.

In this paper, we seek to bridge this gap by studying the phenomenon of diminishing marginal utility [5] in the context of RL. Diminishing marginal utility (DMU) is the subjective phenomenon by which repeated exposure to a rewarding stimulus reduces the perceived utility one experiences. While DMU is thought to have its roots in the maintenance of homeostatic equilibrium (too much ice cream can result in a stomach ache), it also manifests itself in domains in which the collected rewards are abstract, such as economics ($10 vs. $0 is perceived as a bigger difference in value than $1,010 vs. $1,000), where it is closely related to risk aversion [6, 7]. While DMU is well-studied in other fields, relatively few RL studies have explored diminishing reward functions [8, 9], and, to our

knowledge, none contain a formal analysis of DMU within RL. Here, we seek to characterize both its importance and the challenge it poses for current RL approaches (Section 3).

Surprisingly, we find that evaluating policies under diminishing rewards requires agents to learn a novel state representation which we term the λ representation ( λ R , Section 4). The λ R generalizes several state representations from the RL literature: the successor representation [SR; 10], the first-occupancy representation [FR; 11], and the forward-backward representation [FBR; 12], adapting them for non-stationary environments. Interestingly, despite the non-stationarity of the underlying reward functions, we show that the λ R still admits a Bellman recursion, allowing for efficient computation via dynamic programming (or approximate dynamic programming) and prove its convergence. We demonstrate the scalability of the λ R to large and continuous state spaces (Section 4.1), show that it supports policy evaluation, improvement, and composition (Section 5), and show that the behavior it induces is consistent with optimal foraging theory (Section 6).

2 Preliminaries

Standard RL In standard RL, the goal of the agent is to act within its environment so as to maximize its discounted cumulative reward for some task T [13]. Typically, T is modeled as a discounted MDP, T = ( S , A , p, r, γ, µ 0 ) , where S is the state space, A is the set of available actions, p : S × A ↦→ P ( S ) is the transition kernel (where P ( S ) is the space of probability distributions over S ), r : S ↦→ R is the reward function, γ ∈ [0 , 1) is a discount factor, and µ 0 ∈ P ( S ) is the distribution over initial states. Note that the reward function is also frequently defined over stateaction pairs ( s, a ) or triples ( s, a, s ′ ) , but to simplify notation we mostly consider state-based rewards. All of the following analysis, unless otherwise noted, can easily be extended to the ( s, a ) - or ( s, a, s ′ ) -dependent cases. The goal of the agent is to maximize its expected return , or discounted cumulative reward ∑ t γ t r ( s t ) . The role of the discount factor is twofold: from a theoretical standpoint, it ensures that this sum is finite for bounded rewards, and it induces myopia in the agent's behavior. To simplify notation, we will frequently write r ( s t ) ≜ r t and r ∈ R |S| as the vector of rewards for each state. The agent acts according to a stationary policy π : S ↦→ P ( A ) . For finite MDPs, we can describe the expected transition probabilities under π using a |S| × |S| matrix P π such that P π s,s ′ = p π ( s ′ | s ) ≜ ∑ a p ( s ′ | s, a ) π ( a | s ) . Given π and a reward function r , the expected return associated with taking action a in state s is

The Q π r are called the state-action values or simply the Q -values of π . The expectation E π [ · ] is taken with respect to the randomness of both the policy and the transition dynamics. For simplicity of notation, from here onwards we will write expectations of the form E π [ ·| s t = s, a t = a ] as E π [ ·| s t , a t ] . The recursive form in Eq. (2.1) is called the Bellman equation , and it makes the process of estimating Q π r -termed policy evaluation -tractable via dynamic programming. In particular, successive applications of the Bellman operator T π Q ≜ r + γP π Q are guaranteed to converge to the true value function Q π for any initial real-valued |S|×|A| matrix Q . Once a policy has been evaluated, policy improvement identifies a new policy π ′ such that Q π r ( s, a ) ≥ Q π ′ r ( s, a ) , ∀ ( s, a ) ∈ Q π r ( s, a ) . Helpfully, such a policy can be defined as π ′ ( s ) ∈ argmax a Q π r ( s, a ) .

Value Decomposition Often, it can be useful for an agent to evaluate the policies it's learned on new reward functions. In order to do so efficiently, it can make use of the successor representation [SR; 10], which decomposes the value function as follows:

where 1 ( s t ) is a one-hot representation of state s t and M π , the SR, is an |S| × |S| matrix such that

Because the SR satisfies a Bellman recursion, it can be learned in a similar manner to the value function, with the added benefit that if the transition kernel p is fixed, it can be used to instantly evaluate a policy on a task determined by any reward vector r . The SR can also be conditioned on actions, M π ( s, a, s ′ ) , in which case multiplication by r produces the Q -values of π . The SR was originally motivated by the idea that a representation of state in the brain should be dependent on the similarity of future paths through the environment, and there is evidence that SR-like representations are present in the hippocampus [14]. A representation closely related to the SR is the first-occupancy representation [FR; 11], which modifies the SR by only counting the first visit to each state:

The FR can be used in the same way as the SR, with the difference that the values it computes are predicated on ephemeral rewards-those that are consumed or vanish after the first visit to a state.

Policy Composition If the agent has access to a set of policies Π = { π } and their associated SRs (or FRs) and is faced by a new task determined by reward vector r , it can instantly evaluate these policies by simply multiplying: { Q π ( s, a ) = r T M π ( s, a, · ) }. This process is termed generalized policy evaluation [GPE; 15]. Similarly, generalized policy improvement (GPI) is defined as the identification of a new policy π ′ such that Q π ′ ( s, a ) ≥ sup π ∈ Π Q π ( s, a ) ∀ s, a ∈ S × A . Combining both provides a way for an agent to efficiently compose its policies Π -that is, to combine them to produce a new policy without additional learning. This process, which we refer to as GPE+GPI, produces the following policy, which is guaranteed to perform at least as well as any policy in Π [16]:

3 Diminishing Marginal Utility

Problem Statement Motivated by DMU's importance in decision-making, our goal is to understand RL in the context of the following class of non-Markov reward functions:

where n ( s, t ) ∈ N is the agent's visit count at s up to time t and ¯ r ( s ) describes the reward at the first visit to s . λ ( s ) therefore encodes the extent to which reward diminishes after each visit to s . Note that for λ ( s ) = λ = 1 we recover the usual stationary reward given by ¯ r , and so this family of rewards strictly generalizes the stationary Markovian rewards typically used in RL.

DMUis Challenging An immediate question when considering reward functions of this form is whether or not we can still define a Bellman equation over the resulting value function. If this is the case, standard RL approaches still apply. However, the following result shows otherwise.

Lemma 3.1 (Impossibility; Informal) . Given a reward function of the form Eq. (3.1) , it is impossible to define a Bellman equation solely using the resulting value function and immediate reward.

We provide a more precise statement, along with proofs for all theoretical results, in Appendix B. This result means that we can't write the value function corresponding to rewards of the form Eq. (3.1) recursively only in terms of rewards and value in an analogous manner to Eq. (2.1). Nonetheless, we found that it is in fact possible to derive a recursive relationship in this setting, but only by positing a novel state representation that generalizes the SR and the FR, which we term the λ representation ( λ R ). In the following sections, we define the λ R , establish its formal properties, and demonstrate its necessity for RL problems with diminishing rewards.

4 The λ Representation

A Representation for DMU We now the derive the λ R by decomposing the value function for rewards of the form Eq. (3.1) and show that it admits a Bellman recursion. To simplify notation, we

use a single λ for all states, but the results below readily apply to non-uniform λ . We have

where we call Φ π λ ( s ) the λ representation ( λ R ), and n t ( s, k ) ≜ ∑ k -1 j =0 ✶ ( s t + j = s ) , is the number of times state s is visited from time t up to-but not including-time t + k . Formally:

Definition 4.1 ( λ R ) . For an MDP with finite S and λ ∈ [0 , 1] |S| , the λ representation is given by Φ π λ such that

We can immediately see that for λ = 0 , the λ R recovers the FR (we take 0 0 = 1 ), and for λ = 1 , it recovers the SR. For λ ∈ (0 , 1) , the λ R interpolates between the two, with higher values of λ reflecting greater persistence of reward in a given state or state-action pair and lower values of λ reflecting more ephemeral rewards.

where n t ( s, k ) ≜ ∑ k -1 j =0 ✶ ( s t + j = s ) is the number of times state s is visited from time t until time t + k -1 .

The λ R admits a recursive relationship:

where ( i ) follows from n t ( s ′ , k ) = n t ( s ′ , 1)+ n t +1 ( s ′ , k -1) . Amore detailed derivation is provided in Appendix A. Thus, we can define a tractable Bellman operator:

Definition 4.2 ( λ R Operator) . Let Φ ∈ R |S|×|S| be an arbitrary real-valued matrix, and let G π λ denote the λ R Bellman operator for π , such that

where ⊙ denotes elementwise multiplication and I is the |S| × |S| identity matrix. In particular, for a stationary policy π , G π λ Φ π λ = Φ π λ .

The following result establishes that successive applications of G π λ converge to the λ R .

Proposition 4.1 (Convergence) . Under the conditions assumed above, set Φ (0) = (1 -λ ) I . For k = 1 , 2 , . . . , suppose that Φ ( k +1) = G π λ Φ ( k ) . Then

While such analysis is fairly standard in MDP theory, it is noteworthy that the analysis extends to this case despite Lemma 3.1. That is, for a ethologically relevant class of non-Markovian reward functions [17], it is possible to define a Markovian Bellman operator and prove that repeated applications of it converge to the desired representation. Furthermore, unlike in the stationary reward case, where decomposing the value function in terms of the reward and the SR is 'optional' to perform prediction or control, in this setting the structure of the problem requires that this representation be learned. To get an intuition for the λ R consider the simple gridworld presented in Fig. 4.1, where we visualize the λ R for varying values of λ . For λ = 0 , the representation recovers the FR, encoding the expected discount at first occupancy of each state, while as λ increases, effective occupancy is accumulated accordingly at states which are revisited. We offer a more detailed visualization in Fig. F.2.

=

0.0 (FR)

=

0.5

=

1.0 (SR)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

π

4.1 Continuous State Spaces

When the state space is large or continuous, it becomes impossible to use a tabular representation, and we must turn to function approximation. There are several ways to approach this, each with their own advantages and drawbacks.

Feature-Based Representations For the SR and the FR, compatibility with function approximation is most commonly achieved by simply replacing the indicator functions in their definitions with a base feature function ϕ : S ↦→ R D to create successor features [SFs; 15, 16] and first-occupancy features [FFs; 11], respectively. The intuition in this case is that ϕ for the SR and the FR is just a one-hot encoding of the state (or state-action pair), and so for cases when |S| is too large, we can replace it with a compressed representation. That is, we have the following definition

One key fact to note, however, is that due to this compression, all notion of state 'occupancy' is lost and these representations instead measure feature accumulation. For any feasible λ then, it is most natural to define these representations using their recursive forms:

Definition 4.4 ( λ F) . For λ ∈ [0 , 1] and bounded base features ϕ : S ↦→ [0 , 1] D , the λ -feature ( λ F) representation of state s is given by φ π λ such that

In order to maintain their usefulness for policy evaluation, the main requirement of the base features is that the reward should lie in their span. That is, a given feature function ϕ is most useful for an associated set of reward functions R , given by

However, Barreto et al. [18] demonstrate that good performance can still be achieved for an arbitrary reward function as long as it's sufficiently close to some r ∈ R .

Set-Theoretic Formulations As noted above, computing expectations of accumulated abstract features is unsatisfying because it requires that we lose the occupancy-based interpretation of these representations. It also restricts the agent to reward functions which lie within R . An alternative approach to extending the SR to continuous MDPs that avoids this issue is the successor measure [SM; 19], which treats the distribution of future states as a measure over S :

which can be expressed in the discrete case as M π = I + γP π M π . In the continuous case, matrices are replaced by their corresponding measures. Note that SFs can be recovered by integrating: φ π 1 ( s, a ) = ∫ s ′ M π ( s, a, ds ′ ) ϕ ( s ′ ) . We can define an analogous object for the λ R as follows

g

true

= 0.5

0

2

4

6

8

Iterations

0

1

2

3

4

5

Q

-Error

Dynamic Programming

0.5

1.0

0

500

1000

1500

Episodes

0.04

0.06

0.08

0.10

0.12

Tabular TD Learning

0

500

1000

1500

2000

Episodes

0.0

0.5

1.0

1.5

2.0

2.5

F TD Learning

where n t ( X,k ) ≜ ∑ k -1 j =0 δ s t + j ( X ) . However, this is not a measure because it fails to satisfy additivity for λ < 1 , i.e., for measurable disjoint sets A,B ⊆ S , Φ π λ ( s, A ∪ B ) < Φ π λ ( s, A ) + Φ π λ ( s, B ) (Lemma B.4). For this reason, we call Eq. (4.8) the λ operator ( λ O). We then minimize the following squared Bellman error loss for Φ π λ (dropping sub/superscripts for concision):

where ρ and µ are densities over S and Φ π λ ( s ) ≜ φ π λ ( s )diag( µ ) in the discrete case, with φ π λ parametrized by neural network. ¯ · indicates a stop-gradient operation, i.e., a target network. A detailed derivation and discussion are given in Appendix H. While ρ can be any training distribution of transitions we can sample from, we require an analytic expression for µ . Eq. (4.9) recovers the SM loss of Touati and Ollivier [12] when λ = 1 .

5 Policy Evaluation, Learning, and Composition under DMU

In the following sections, we experimentally validate the formal properties of the λ R and explore its usefulness for solving RL problems with DMU. The majority of our experiments center on navigation tasks, as we believe this is the most natural setting for studying behavior under diminishing rewards. However, in Appendix I we also explore potential for the λ R 's use other areas, such as continuous control, even when rewards do not diminish. There is also the inherent question of whether the agent has access to λ . In a naturalistic context, λ can be seen as an internal variable that the agent likely knows, especially if the agent has experienced the related stimulus before. Therefore, in subsequent experiments, treating λ as a "given" can be taken to imply the agent has prior experience with the relevant stimulus. Further details for all experiments can be found in Appendix E.

5.1 Policy Evaluation

In Section 4, we showed that in order to perform policy evaluation under DMU, an agent is required to learn the λ R . In our first experimental setting, we verify this analysis empirically for the policy depicted in Fig. 4.1 with a rewarded location in the state indicated by a g in Fig. 5.1 with λ true = 0 . 5 . We then compare the performance for agents using different values of λ across dynamic programming (DP), tabular TD learning, and λ F TD learning with Laplacian features. For the latter two, we use a linear function approximator with a one-hot encoding of the state as the base feature function. We then compute the Q -values using the λ R with λ ∈ { 0 . 5 , 1 . 0 } (with λ = 1 corresponding to the SR) and compare the resulting value estimates to the true Q -values. Consistent with our theoretical analysis, Fig. 5.1 shows that the λ R with λ = λ true is required to produce accurate value estimates.

5.2 Policy Learning

To demonstrate that λ R is useful in supporting policy improvement under diminishing rewards, we implemented modified forms of Q -learning [20] (which we term Q λ -learning) and advantage

Then,

Note that for λ = 1 , we recover the original GPI bound due to Barreto et al. [18] with an additional term quantifying error accrued if incorrectly assuming λ < 1 .

Tabular Navigation We can see this result reflected empirically in Fig. 5.3, where we consider the following experimental set-up in the classic FourRooms domain [24]. The agent is assumed to be given or have previously acquired four policies { π 0 , π 1 , π 2 , π 3 } individually optimized to reach rewards located in each of the four rooms of the environment. There are three reward locations { g 0 , g 1 , g 2 } scattered across the rooms, each with its own initial reward and all with λ = 0 . 5 . At the

true

=0.5

0

100

200

300

400

500

Episodes

0

5

10

15

20

25

30

Return

Q

-Learning

0.0

0.5

1.0

0

2000

4000

6000

Episodes

0

5

10

15

20

25

RNN A2C-

actor-critic [A2C; 21] and applied them to the TwoRooms domain from the NeuroNav benchmark task set [22] (see Fig. 5.2). For a transition ( s t , a t , s t +1 ) , we define the following operator:

where a t +1 = argmax a Q λ ( s t , a ) = argmax a r T Φ( s t , a ) . This is an improvement operator for Q λ . The results in Fig. 5.2 show that Q λ -learning outperforms standard Q -learning ( λ = 1 ) for diminishing rewards, and that the 'correct' λ produces the best performance. To implement A2C with a λ R critic, we modified the standard TD target in a similar manner as follows:

where ϕ were one-hot state encodings, and the policy, value function, and λ F were output heads of a shared LSTM network [23]. Note this target is equivalent to Definition 4.4 multiplied by the reward (derivation in Appendix E). Fig. 5.2 shows again that correct value targets lead to improved performance. Videos of agent behavior can be found at lambdarepresentation.github.io .

5.3 Policy Composition

As we've seen, DMU problems of this form have an interesting property wherein solving one task requires the computation of a representation which on its own is task agnostic. In the same way that the SR and FR facilitate generalization across reward functions, the λ R facilitates generalization across reward functions with different ¯ r s.The following result shows that there is a benefit to having the 'correct' λ for a given resource.

Theorem 5.1 (GPI) . Let { M j } n j =1 ⊆ M and M ∈ M be a set of tasks in an environment M and let Q π ∗ j denote the action-value function of an optimal policy of M j when executed in M . Assume that the agent uses diminishing rate ˆ λ that may differ from the true environment diminishing rate λ . Given estimates ˜ Q π j such that ∥ Q π ∗ j -˜ Q π j ∥ ∞ ≤ ϵ for all j , define

0.0

0.5

1.0

15.0

17.5

20.0

22.5

25.0

Return

=0.5

s

0

g

0

g

1

g

2

=0.0

s

0

g

0

g

1

g

2

=1.0

s

0

g

0

g

1

g

2

0

5

10

15

20

25

30

35

40

t

Tabular GPI

Example Trajectories

0.0

0.5

1.0

30

32

34

36

38

40

Return

Values

=

0.0

Values

=

0.5

Values

=

1.0

Pixel-based GPI

Feature Analysis

Example Observation

beginning of each episode, an initial state s 0 is sampled uniformly from the set of available states. An episode terminates either when the maximum reward remaining in any of the goal states is less than 0 . 1 or when the maximum number of steps H = 40 is reached (when λ = 1 , the latter is the only applicable condition). For each of the four policies, we learn λ R s with λ ∈ { 0 . 0 , 0 . 5 , 1 . 0 } using dynamic programming and record the returns obtained while performing GPE+GPI with each of these representations over 50 episodes. Bellman error curves for the λ R s are shown in Fig. F.3, and demonstrate that convergence is faster for lower λ . In the left panel of Fig. 5.3, we can indeed see that using the correct λ (0.5) nets the highest returns. Example trajectories for each agent λ are shown in the remaining panels.

Pixel-Based Navigation We verified that the previous result is reflected in larger scales by repeating the experiment in a partially-observed version of FourRooms in which the agent receives 128 × 128 RGB egocentric observations of the environment (Fig. 5.4, left) with H = 50 . In this case, the agent learns λ Fs for each policy for λ ∈ { 0 . 0 , 0 . 5 , 1 . 0 } , where each λ F is parameterized by a feedforward convolutional network with the last seven previous frames stacked to account for the partial observability. The base features were Laplacian eigenfunctions normalized to [0 , 1] , which which were shown by Touati et al. [25] to perform the best of all base features for SFs across a range of environments including navigation.

6 Understanding Natural Behavior

Naturalistic environments often exhibit diminishing reward and give insight into animal behavior. The problem of foraging in an environment with multiple diminishing food patches (i.e., reward states) is of interest in behavioral science [26-28]. The cornerstone of foraging theory is the marginal value theorem [MVT; 28, 29], which states that the optimal time to leave a patch is when the patch's reward rate matches the average reward rate of the environment. However, the MVT does not describe which patch to move to once an agent leaves its current patch. We show that the λ O recovers MVT-like behavior in discrete environments and improves upon the MVT by not only predicting when agents should leave rewarding patches, but also where they should go. Moreover, we provide a scheme for learning λ alongside the λ Ousing feedback from the environment.

a

b

d

1.0

：Per-Episode Reward

InitialReward

入 =1.0

入 =0.1

6°0

2.5

50

50

0.8

2.0

4

40

40

Learned 入

0.7

Fixed 入

1.5

3

30

30

0.6

True 入

1.0

S°0

AverageR

2

20

20

0.4

0.5

0.3

0.0

1

10

10

0

20

40

Epoch

60

80

100

0

20

40

Epoch

60

80

100

1

0

To learn the λ O, we take inspiration from the FBR [12] and use the following parametrization: Φ π z λ ( s, a, s ′ ) = F ( s, a, z ) ⊤ B ( s ′ ) µ ( s ′ ) and π z ( s ) = argmax a F ( s, a, z ) ⊤ z , where µ is a density with full support on S , e.g., uniform. We then optimize using the the loss in Eq. (4.9) under this parameterization (details in Appendix E). Given a reward function r : S → R at evaluation time, the agent acts according to argmax a F ( s, a, z R ) ⊤ z R , where z R = E s ∼ µ [ r ( s ) B ( s )] . Because the environment is non-stationary, z R has to be re-computed at every time step. To emulate a more realistic foraging task, the agent learns λ by minimizing the loss in Eq. (H.1) in parallel with the loss

✶ which provably recovers the correct value of λ provided that ρ is sufficiently exploratory. In Appendix H.1 we provide experiments showing that using an incorrect value of λ leads to poor performance on tabular tasks. In Fig. 6.1 we show that the agent learns the correct value of λ , increasing its performance. We illustrate the behavior of the λ O in an asymmetric environment that has one large reward state on one side and many small reward states (with higher total reward) on the other. Different values of λ lead to very different optimal foraging strategies, which the λ O recovers and exhibits MVT-like behavior (see Appendix H.2 for a more detailed analysis). Our hope is that the λ R may provide a framework for new theoretical studies of foraging behavior and possibly mechanisms for posing new hypotheses. For example, an overly large λ may lead to overstaying in depleted patches, a frequently observed phenomenon [30].

7 Conclusion

Limitations Despite its advantages, there are several drawbacks to the representation which are a direct result of the challenge of the DMU setting. First, the λ R is only useful for transfer across diminishing reward functions when the value of λ at each state is consistent across tasks. In natural settings, this is fairly reasonable, as λ can be thought of as encoding the type of the resource available at each state (i.e., each resource has its own associated decay rate). Second, as noted in Section 4.1, the λ R does not admit a measure-theoretic formulation, which makes it challenging to define a principled, occupancy-based version compatible with continuous state spaces. Third, the λ R is a prospective representation, and so while it is used to correctly evaluate a policy's future return under DMU, it is not inherently memory-based and so performs this evaluation as if the agent hasn't visited locations with diminishing reward before. Additional mechanisms (i.e., recurrence or frame-stacking) are necessary to account for previous visits. Finally, the λ R is dependent on an episodic task setting for rewards to reset, as otherwise the agent would eventually consume all reward in the environment. An even more natural reward structure would include a mechanism for reward replenishment in addition to depletion. We describe several such candidates in Appendix J, but leave a more thorough investigation of this issue to future work.

In this work, we aimed to lay the groundwork for understanding policy evaluation, learning, and composition under diminishing rewards. To solve such problems, we introduced-and showed the necessity of-the λ representation , which generalizes the SR and FR. We demonstrated its usefulness for rapid policy evaluation and by extension, composition, as well as control. We believe the λ R represents a useful step in the development of state representations for naturalistic environments.

References

Daniel Kahneman and Amos Tversky. Prospect theory: An analysis of decision under risk. Econometrica, 47(2):263-292, 1979.

Matthew Rabin. Risk aversion and expected-utility theory: a calibration theorem. Econometrica, 68(5):1281-1292, 2000. doi: 10.2307/2999450.

Alex Pine, Ben Seymour, Jonathan P Roiser, Peter Bossaerts, Karl J Friston, H Valerie Curran, and Raymond J Dolan. Encoding of marginal utility across time in the human brain. Journal of Neuroscience, 29(30):9575-9581, 2009.

Martin L. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley and Sons, 2010.

Hermann Heinrich Gossen and Rudolph C Blitz. The laws of human relations and the rules of human action derived therefrom. Mit Press Cambridge, MA, 1983.

Kenneth J Arrow. The theory of risk-bearing: small and great risks. Journal of risk and uncertainty, 12:103-111, 1996.

John W Pratt. Risk aversion in the small and in the large. In Uncertainty in economics, pages 59-79. Elsevier, 1978.

Nathan Wispinski, Andrew Butcher, Kory Wallace Mathewson, Craig S Chapman, Matthew Botvinick, and Patrick M. Pilarski. Adaptive patch foraging in deep reinforcement learning agents. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https: //openreview.net/forum?id=a0T3nOP9sB .

Sergey Shuvaev, Sarah Starosta, Duda Kvitsiani, Adam Kepecs, and Alexei Koulakov. Rlearning in actor-critic model offers a biologically relevant mechanism for sequential decisionmaking. Advances in neural information processing systems, 33:18872-18882, 2020.

Peter Dayan. Improving generalization for temporal difference learning: The successor representation. Neural Computation, 5(4):613-624, 1993. doi: 10.1162/neco.1993.5.4.613.

Ted Moskovitz, Spencer R Wilson, and Maneesh Sahani. A first-occupancy representation for reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=JBAZe2yN6Ub .

Ahmed Touati and Yann Ollivier. Learning one representation to optimize all rewards. Advances in Neural Information Processing Systems, 34:13-23, 2021.

Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018. URL http://incompleteideas.net/book/the-book-2nd. html .

Kimberly L Stachenfeld, Matthew M Botvinick, and Samuel J Gershman. The hippocampus as a predictive map. Nature neuroscience, 20(11):1643-1653, 2017.

Andre Barreto, Shaobo Hou, Diana Borsa, David Silver, and Doina Precup. Fast reinforcement learning with generalized policy updates. Proceedings of the National Academy of Sciences, 117(48):30079-30087, 2020. ISSN 0027-8424. doi: 10.1073/pnas.1907370117. URL https: //www.pnas.org/content/117/48/30079 .

André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. Advances in neural information processing systems, 30, 2017.

Maor Gaon and Ronen Brafman. Reinforcement learning with non-markovian rewards. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 3980-3987, 2020.

André Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Žídek, and Rémi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement, 2019.

Léonard Blier and Yann Ollivier. The description length of deep learning models. Advances in Neural Information Processing Systems, 31, 2018.

Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8:279-292, 1992.

Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928-1937. PMLR, 2016.

Arthur Juliani, Samuel Barnett, Brandon Davis, Margaret Sereno, and Ida Momennejad. Neuronav: a library for neurally-plausible reinforcement learning. arXiv preprint arXiv:2206.03312, 2022.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9 (8):1735-1780, 1997.

Richard S. Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112 (1):181-211, 1999. URL https://www.sciencedirect.com/science/article/pii/ S0004370299000521 .

Ahmed Touati, Jérémy Rapin, and Yann Ollivier. Does zero-shot reinforcement learning exist? In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=MYEap_OcQI .

Benjamin Y Hayden, John M Pearson, and Michael L Platt. Neuronal basis of sequential foraging decisions in a patchy environment. Nature neuroscience, 14(7):933-939, 2011.

Tehrim Yoon, Robert B Geary, Alaa A Ahmed, and Reza Shadmehr. Control of movement vigor and decision making during foraging. Proceedings of the National Academy of Sciences, 115 (44):E10476-E10485, 2018.

Anthony S Gabay and Matthew AJ Apps. Foraging optimally in social neuroscience: computations and methodological considerations. Social Cognitive and Affective Neuroscience, 16(8): 782-794, 2021.

Eric L. Charnov. Optimal foraging, the marginal value theorem. Theoretical Population Biology, 9(2):129-136, 1976. ISSN 0040-5809.

Peter Nonacs. State dependent behavior and the marginal value theorem. Behavioral Ecology, 12(1):71-83, 2001.

Ruosong Wang, Hanrui Zhang, Devendra Singh Chaplot, Denis Garagi´ c, and Ruslan Salakhutdinov. Planning with submodular objective functions. arXiv preprint arXiv:2010.11863, 2020.

Manish Prajapat, Mojmír Mutn` y, Melanie N Zeilinger, and Andreas Krause. Submodular reinforcement learning. arXiv preprint arXiv:2307.13372, 2023.

Tom Zahavy, Brendan O'Donoghue, Guillaume Desjardins, and Satinder Singh. Reward is enough for convex mdps, 2021.

Eitan Altman. Constrained Markov decision processes. Routledge, 2021.

Ted Moskovitz, Brendan O'Donoghue, Vivek Veeriah, Sebastian Flennerhag, Satinder Singh, and Tom Zahavy. Reload: Reinforcement learning with optimistic ascent-descent for last-iterate convergence in constrained mdps. In International Conference on Machine Learning, pages 25303-25336. PMLR, 2023.

Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvari, Satinder Singh, et al. Behaviour suite for reinforcement learning. arXiv preprint arXiv:1908.03568, 2019.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.

Martin Riedmiller. Neural fitted q iteration-first experiences with a data efficient neural reinforcement learning method. In Machine Learning: ECML 2005: 16th European Conference on Machine Learning, Porto, Portugal, October 3-7, 2005. Proceedings 16, pages 317-328. Springer, 2005.

Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. Advances in neural information processing systems, 5, 1992.

Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, 2018.

Ted Moskovitz, Michael Arbel, Ferenc Huszar, and Arthur Gretton. Efficient wasserstein natural gradients for reinforcement learning, 2020.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, 2018.

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.

Ted Moskovitz, Jack Parker-Holder, Aldo Pacchiano, Michael Arbel, and Michael Jordan. Tactical optimism and pessimism for deep reinforcement learning. Advances in Neural Information Processing Systems, 34:12849-12863, 2021.

Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cambridge university press, 2006.

Appendix: A State Representation for Diminishing Rewards

GIFs of navigation agents can be found at lambdarepresentation.github.io and in the supplementary material.

A Derivation of λ R Recursion

We provide a step-by-step derivation of the λ R recursion in Eq. (4.3):

where ( i ) is because n t ( s ′ , k ) = n t ( s ′ , 1) + n t +1 ( s ′ , k ) and ( ii ) is because

B Theoretical Analysis

Here, we provide proofs for the theoretical results in the main text.

Lemma 3.1 (Impossibility; Informal) . Given a reward function of the form Eq. (3.1) , it is impossible to define a Bellman equation solely using the resulting value function and immediate reward.

Before providing the formal statement and proof for Lemma 3.1, we introduce a definition for a Bellman operator.

Definition B.1 (Bellman Operator) . A Bellman operator is a contractive operator R |S| → R |S| that depends solely on ¯ r , one-step expectations under p π , and learning hyperparameters (in our case γ and λ ).

We can now give a formal statement of Lemma 3.1:

Lemma B.1 (Impossibility; Formal) . Assume that |S| > 1 . Then, there does not exist a Bellman operator T with fixed point V π .

Proof. Assume for a contradiction that T is a Bellman operator. By the Banach fixed-point theorem, V π must be the unique fixed point of T . Hence, TV π must take on the following form (see the proof of Lemma 3.1 in Appendix B): for s ∈ S ,

For the assumption to hold, there must exist a function f such that, for any s ∈ S ,

Now, by definition,

̸

¯ r is a vector in R |S| , so as long as S > 1 , ¯ r ⊥ is non-trivial. Fix any ¯ r , V π , s . Pick a vector w ∈ ¯ r ⊥ \ { 0 } and define

for any s, s ′ ∈ S . Note that

as w ⊥ ¯ r . However,

̸

The final term evaluates to w ( s ) . Because w = 0 , there must exist some s such that w ( s ) = 0 . For this s , we have a single input ( ¯ r , V π , γ, λ, s ) to f that corresponds to two distinct outputs: E s ′ ∼ p π ( ·| s ) Φ π λ ( s ′ , s ) and E s ′ ∼ p π ( ·| s ) ˜ Φ π λ ( s ′ , s ) .

Hence, f is a one-to-many mapping: for fixed input, there is more than one output. Therefore, f is not a function, yielding a contradiction.

The following establishes G π λ as a contraction.

Lemma B.2 (Contraction) . Let G π λ be the operator as defined in Definition 4.2 for some stationary policy π . Then for any two matrices Φ , Φ ′ ∈ R |S|×|S| ,

Proof. We have

where ( i ) comes from using λ ≤ 1 and simplifying.

Note that we can actually get a tighter contraction factor of λγ for s = s ′ . Given this contractive property, we can prove its convergence with the use of the following lemma.

Lemma B.3 (Max λ R ) . The maximum possible value of Φ π λ ( s, s ′ ) is

This is just the standard SR recursion with discount factor λγ , so the maximum is

̸

For s = s ′ , ✶ ( s t = s ′ ) = 0 , so

Observe that Φ π λ ( s, s ) ≥ Φ π λ ( s, s ′ ) for s ′ = s , so the maximum is attained for s t +1 = s ′ . We can then use the result for s = s ′ to get

Combining Eq. (B.1) and Eq. (B.2) yields the desired result.

Proposition 4.1 (Convergence) . Under the conditions assumed above, set Φ (0) = (1 -λ ) I . For k = 1 , 2 , . . . , suppose that Φ ( k +1) = G π λ Φ ( k ) . Then

Proof. Using the notation X s,s ′ = X ( s, s ′ ) for a matrix X :

̸

where ( i ) is due to Lemma B.2, ( ii ) is because Φ (0) ( s, s ′ ) = 0 for s = s ′ , and ( iii ) is due to Lemma B.3.

Lemma B.4 (Subadditivity) . For any s ∈ S , policy π , λ ∈ [0 , 1) , and disjoint measurable sets A, B ⊆ S ,

Proof. For s = s ′ ,

̸

Proof. Note that for disjoint sets A, B , we have n t ( A ∪ B,k ) = n t ( A, k ) + n t ( B,k ) . Hence, conditioned on some policy π and s t = s ,

where the first line follows from P ( s t + k ∈ A ∪ B ) = P ( s t + k ∈ A ) + P ( s t + k ∈ B ) . Equality holds over all A, B, t, k if and only if λ = 1 . Summing over k yields the result.

B.1 Proof of Theorem 5.1

We first prove two results, which rely throughout on the fact that Φ λ ( s, a, s ′ ) ≤ 1 1 -λγ for all s, a, s ′ , which follows from Lemma B.3. For simplicity, we also assume throughout that all rewards are non-negative, but this assumption can easily be dropped by taking absolute values of rewards. The proofs presented here borrow ideas from those of [16].

LemmaB.5. Let { M j } n j =1 ⊆ M and M ∈ M be a set of tasks in an environment M with diminishing rate λ and let Q π ∗ j denote the action-value function of an optimal policy of M j when executed in M . Given estimates ˜ Q π j such that ∥ Q π ∗ j -˜ Q π j ∥ ∞ ≤ ϵ for all j , define

Then,

where r denotes the reward function of M .

Proof. Define ˜ Q max ( s, a ) := max j ˜ Q π j ( s, a ) and Q max ( s, a ) := max j Q π ∗ j ( s, a ) . Let T ν denote the Bellman operator of a policy ν in task M . For all ( s, a ) ∈ S × A and

all j ,

This holds for any j , so

Next, note that for any c ∈ R ,

Putting everything together, and using the fact that T ν is monotonic and contractive,

This holds for every j , hence the result.

Lemma B.6. Let ν be any policy, λ, ˆ λ ∈ [0 , 1] , and Q λ denote a value function with respecting to diminishing rate λ . Then,

Theorem 5.1 (GPI) . Let { M j } n j =1 ⊆ M and M ∈ M be a set of tasks in an environment M and let Q π ∗ j denote the action-value function of an optimal policy of M j when executed in M . Assume that the agent uses diminishing rate ˆ λ that may differ from the true environment diminishing rate λ . Given estimates ˜ Q π j such that ∥ Q π ∗ j -˜ Q π j ∥ ∞ ≤ ϵ for all j , define

Then,

Proof. Let Q λ denote a value function with respect to diminishing constant λ . We wish to bound

i.e., the value of the GPI policy with respect to the true λ compared to the maximum value of the constituent policies π ∗ j used for GPI, which were used assuming ˆ λ . By the triangle inequality,

We bound (1) by Lemma B.5 and (2) by Lemma B.6 to get the result.

B.2 An Extension of Theorem 5.1

Inspired by [18], we prove an extension of Theorem 5.1:

Theorem B.1. Let M ∈ M be a task in an environment M with true diminishing constant λ . Suppose we perform GPI assuming a diminishing constant ˆ λ :

Let { M j } n j =1 and M i be tasks in M and let Q π ∗ j i denote the actionvalue function of an optimal policy of M j when executed in M i . Given estimates ˜ Q π j i such that ∥ Q π ∗ j i -˜ Q π j i ∥ ∞ ≤ ϵ for all j , define π ( s ) ∈ argmax a max j ˜ Q π j i ( s, a ) .

Let Q π ˆ λ and Q π ∗ λ denote the action-value functions of π and the M -optimal policy π ∗ when executed in M , respectively. Then,

where C is a positive constant not depending on λ :

̸

Note that when λ = 1 , we recover Proposition 1 of [18] with an additional term quantifying error incurred by ˆ λ = λ . The proof relies on two other technical lemmas, presented below.

Lemma B.7.

Proof. Define ∆ i := ∥ Q π ∗ -Q π ∗ i i ∥ ∞ . For any ( s, a ) ∈ S × A ,

The third term decomposes as

We could equivalently use the following decomposition:

and so

The inequalities above hold for all s, a and so

Hence the result.

Proof. Write ∆ i := ∥ Q π i -Q π ∥ ∞ . Proceeding as in the previous lemma, for all ( s, a ) ∈ S × A , we have

Finally, we prove Theorem B.1:

Proof of Theorem B.1. By the triangle inequality,

By Lemma B.6, the second term is bounded above by

The first term decomposes as follows (dropping the λ subscript on all action-value functions for clarity):

Applying Lemma B.5 to (2) (but with respect to M i rather than M ), we have that for any j ,

We bound (2.1) using Lemma B.7 and (2.2) using Lemma B.8 (but with respect to M j rather than M ):

We then apply Lemma B.7 to (1) and Lemma B.8 to (3) to get the result.

C An n th Occupancy Representation

To generalize the first occupancy representation to account for reward functions of this type, it's natural to consider an N th occupancy representation -that is, one which accumulates value only for the first N occupancies of one state s ′ starting from another state s :

Definition C.1 (NR) . For an MDP with finite S , the N th-occupancy representation (NR) for a policy π is given by F π ∈ [0 , N ] |S|×|S| such that

Intuitively, such a representation sums the first N (discounted) occupancies of s ′ from time t to t + k starting from s t = s . We can also note that Φ π (1) is simply the FR and Φ (0) ( s, s ′ ) = 0 ∀ s, s ′ . As with the FR and the SR, we can derive a recursive relationship for the NR:

where the expectation is wrt p π ( s t +1 | s t ) . Once again, we can confirm that this is consistent with the FR by noting that for N = 1 , the NR recursion recovers the FR recursion. Crucially, we also recover the SR recursion in the limit as N →∞ :

✶ This is consistent with the intuition that the SR accumulates every (discounted) state occupancy in a potentially infinite time horizon of experience. While Definition C.1 admits a recursive form which is consistent with our intuition, Eq. (C.2) reveals an inconvenient intractability: the Bellman target for Φ π ( N ) requires the availability of Φ π ( N -1) . This is a challenge, because it means that if we'd like to learn any NR for finite N > 1 , the agent also must learn and store Φ π (1) , . . . Φ π ( N -1) . Given these challenges, the question of how to learn a tractable general occupancy representation remains. From a neuroscientific perspective, a fixed depletion amount is also inconsistent with both behavioral observations and neural imaging [3], which indicate instead that utility disappears at a fixed rate in proportion to the current remaining utility , rather than in proportion to the original utility . We address these theoretical and practical issues in the next section.

D Additional Related Work

Another important and relevant sub-field of reinforcement learning is work which studies non-stationary rewards. Perhaps most relevant, the setting of DMU can be seen as a special case of submodular planning and reward structures [31, 32]. [31] focus specifically on planning and not the form of diminishment we study, while [32] is a concurrent work which focuses on the general class of submodular

reward problems and introduces a policy-based, REINFORCE-like method which is necessarily on-policy. In contrast, we focus on a particular sub-class of problems especially relevant to natural behavior and introduce a family of approaches which exploit this reward structure, are value-based (and which can be used to modify the critic in policy-based methods), and are compatible with off-policy learning. Other important areas include convex [33] and constrained [34, 35] MDPs. In these cases, non-stationarity is introduced by way of a primal-dual formulation of distinct problem classes into min-max games.

E Further Experimental Details

E.1 Policy Evaluation

We perform policy evaluation for the policy shown in Fig. 4.1 on the 6 × 6 gridworld shown. The discount factor γ was set to 0.9 for all experiments, which were run for H = 10 steps per episode. The error metric was the mean squared error:

where Q π is the ground truth Q -values and ˆ Q is the estimate. Transitions are deterministic. For the dynamic programming result, we learned the λ R using Eq. (4.3) for λ ∈ { 0 . 5 , 1 . 0 } and then measured the resulting values by multiplying the resulting λ R by the associated reward vector r ∈ {-1 , 0 , 1 } 36 , which was -1 in all wall states and +1 at the reward state g . We compared the results to the ground truth values. Dynamic programming was run until the maximum Bellman error across state-action pairs reduced below 5e-2. For the tabular TD learning result, we ran the policy for three episodes starting from every available (non-wall) state in the environment, and learned the λ R for λ ∈ { 0 . 5 , 1 . 0 } as above, but using the online TD update:

where a t +1 ∼ π ( · | s t +1 ) . The learned Q -values were then computed in the same way as the dynamic programming case and compared to the ground truth. For the λ F result, we first learned Laplacian eigenfunction base features as described in [25] from a uniform exploration policy and normalized them to the range [0 , 1] . We parameterized the base feature network as a 2-layer MLP with ReLU activations and 16 units in the hidden layer. We then used the base features to learn the λ Fs as in the tabular case, but with the λ F network parameterized as a three-layer MLP with 16 units in each of the hidden layers and ReLU activations. All networks were optimized using Adam with a learning rate of 3e-4. The tabular and neural network experiments were repeated for three random seeds, the former was run for 1,500 episodes and the latter for 2,000.

E.2 Policy Learning

We ran the experiments for Fig. 5.2 in a version of the TwoRooms environment from the NeuroNav benchmark [22] with reward modified to decay with a specified

1: Require: Current λ R -values Φ ( t ) λ ∈ R |S|×|A|×|S| , current reward vector r ( t ) , observed ( s t , a t , s t +1 ) tuple

2: Compute Q λ -values: Q ( t ) λ ← (Φ ( t ) λ ) T r ( t )

3: Select greedy action: a t +1 ← argmax a ∈A Q ( t ) λ ( s t +1 , a )

4: Update Φ λ :

5: Return updated Φ ( t +1) λ

λ true = 0 . 5 and discount factor γ = 0 . 95 . The initial rewards in the top right goal and the lower room goal locations were 5 and the top left goal had initial reward 10 . The observations in the neural network experiment were one-hot state indicators. The tabular Q λ experiments run the algorithm in Algorithm 1 for 500 episodes for λ ∈ { 0 . 0 , 0 . 5 , 1 . 0 } , with λ true set to 0.5, repeated for three random seeds. Experiments used a constant step size α = 0 . 1 . There were five possible actions: up, right, down, left, and stay. The recurrent A2C agents were based on the implementation from the BSuite library [36] and were run for 7,500 episodes of maximum length H = 100 with γ = 0 . 99 using the Adam optimizer with learning rate 3e-4. The experiment was repeated for three random seeds. The RNN was an LSTM with 128 hidden units and three output heads: one for the policy, one for the value function, and one for the λ F. The base features were one-hot representations of the current state, 121-dimensional in this case.

E.3 Tabular GPI

The agent is assumed to be given or have previously acquired four policies { π 0 , π 1 , π 2 , π 3 } individually optimized to reach rewards located in each of the four rooms of the environment. There are three reward locations { g 0 , g 1 , g 2 } scattered across the rooms, each with its own initial reward ¯ r = [5 , 10 , 5] and all with λ = 0 . 5 . At the beginning of each episode, an initial state s 0 is sampled uniformly from the set of available states. An episode terminates either when the maximum reward remaining in any of the goal states is less than 0 . 1 or when the maximum number of steps H = 40 is reached. Empty states carry a reward of 0 , encountering a wall gives a reward of -1 , and the discount factor is set to γ = 0 . 97 .

For each of the four policies, we learn λ R s with λ equal to 0 , 0 . 5 , and 1 . 0 using standard dynamic programming (Bellman error curves plotted in Fig. F.3), and record the returns obtained while performing GPE+GPI with each of these representations over the course of 50 episodes. Bellman error curves for the λ R s are In the left panel of Fig. 5.3, we can indeed see that using the correct λ (0.5) nets the highest returns. Example trajectories for each of λ R are shown in the remaining panels.

E.4 Pixel-Based GPI

In this case, the base policies Π were identical to those used in the tabular GPI experiments. First, we collected a dataset consisting of 340 observation trajectories ( o 0 , o 1 , . . . , o H -1 ) ∈ O H with H = 19 from each policy, totalling 6 , 460 observations. Raw observations were 128 × 128 × 3 and were converted to grayscale. The previous seven observations were stacked and used to train a Laplacian eigenfunction base feature network in the same way as [25]. For observations less than seven steps from the start of an episode, the remaining frames were filled in as all black observations (i.e., zeros). The network consisted of four convolutional layers with 32 3 × 3 filters with strides (2 , 2 , 2 , 1) , each followed by a ReLU nonlinearity. This was then flattened and passed through a Layer Norm layer [37] and a tanh nonlinearity before three fully fully connected layers, the first two with 64 units each and ReLU nonlinearities and the final, output layer with 50 units. The output was L 2 -normalized as in [25]. This network ϕ : O 7 ↦→ R D (with D = 50 ) was trained on the stacked observations for 10 epochs using the Adam optimizer and learning rate 1e-4 with batch size B = 64 . To perform policy evaluation, the resulting features, evaluated on the dataset of stacked observations were collected into their own dataset of ( s t , a t +1 , s t +1 , a t +1 ) tuples, where s t ≜ o t -6: t . The 'states' were normalized to be between 0 and 1, and a vector w was fit to the actual associated rewards via linear regression on the complete dataset. The λ F network was then trained using a form of neural fitted Q-iteration [FQI; 38] modified for policy evaluation with λ Fs (Algorithm 2). The architecture for the λ F network was identical to the base feature network, with the exception that the hidden size of the fully connected layers was 128 and the output dimension was D |A| = 250 . FQI was run for K = 20 outer loop iterations, with each inner loop supervised learning setting run for L = 100 epochs on the current dataset. Supervised learning was done using Adam with learning rate 3e-4 and batch size B = 64 . Given the trained networks, GPI proceeded as in the tabular case, i.e.,

50 episodes were run from random starting locations for H = 50 steps and the returns measured. Learning curves for the base features and for λ F fitting are shown in Fig. E.1. The λ F curve measures the mean squared error as in Eq. (E.1).

The feature visualizations were created by performing PCA to reduce the average λ F representations for observations at each state in the environment to 2D. Each point in the scatter plot represents the reduced representation on the xy plane, and is colored according to the λ -conditioned value of the underlying state.

E.5 Continuous Control

λ -SAC See Appendix I for details.

E.6 Learning the λ Owith FB

Training the λ O with the FB parameterization proceeds in much the same way as in [12], but adjusted for a different norm and non-Markovian environment. We

0

2

4

6

8

Epochs

0

500

1000

1500

2000

2500

3000

3500

Loss

Base Feature Learning

0

5

10

15

FQI Iterations

0.0

0.2

0.4

0.6

0.8

Q

MSE

F Learning

summarize the learning procedure in Algorithm 3. The loss function L is derived in Appendix H, with the addition of the following regularizer:

This regularizer encourages B to be approximately orthonormal, which promotes identifiability of F θ and B ω [12].

F Additional Results

See surrounding sections.

Algorithm 3: λ O FB Learning

s0

sg

1

2

3

0

20

40

60

Iteration

0.0

0.5

1.0

1.5

2.0

Bellman Error

Noise = 0.0

0.0

0.25

0.5

0.75

1.0

0

20

40

60

Iteration

Noise = 0.2

0.0

0.5

1.0

10.0

12.5

15.0

17.5

20.0

22.5

Return

=0.5

s

0

g

0

g

1

g

2

=0.0

s

0

g

0

g

1

g

2

=1.0

s

0

g

0

g

1

g

2

0

5

10

15

20

25

30

35

40

t

G Advantage of the Correct λ

Importantly, for GPE using the λ R to work in this setting, the agent must either learn or be provided with the updated reward vector r λ after each step/encounter with a rewarded state. This is because the λ R is forward-looking in that it measures the (diminished) expected occupancies of states in the future without an explicit mechanism for remembering previous visits. For simplicity in this case, we provide this vector to the agent at each step-though if we view such a multitask agent as simply as a module carrying out the directives of a higher-level module or policy within a hierarchical framework as in, e.g., Feudal RL [39], the explicit provision of reward information is not unrealistic. Regardless, a natural question in this case is whether there is actually any value in using the λ R with the corret value of λ in this

s0

sg

M

1

s0

sg

M

2

s0

sg

M

3

s0

sg

1

1.0

s0

sg

2

1.0

s0

sg

3

1.0

s0

sg

1

0.8

s0

sg

2

0.8

s0

sg

3

0.8

s0

sg

1

0.4

s0

sg

2

0.4

s0

sg

3

0.4

s0

sg

1

0.0

s0

sg

2

0.0

s0

sg

3

0.0

s0

sg

F

1

s0

sg

F

2

s0

sg

F

3

0.0

0.2

0.4

0.6

0.8

1.0

setting: If the agent is provided with the correct reward vector, then wouldn't policy evaluation work with any λ R ?

To see that this is not the case, consider the three-state toy MDP shown in Figure Fig. G.1, where ¯ r ( s 1 ) = 10 , ¯ r ( s 2 ) = 6 , ¯ r ( s 0 ) = 0 , λ ( s 1 ) = 0 , λ ( s 2 ) = 1 . 0 , and γ = 0 . 99 . At time t = 0 , the agent starts in s 0 . Performing policy evaluation with λ ( s 1 ) = λ ( s 2 ) = 1 (i.e., with the SR) would lead the agent to go left to s 1 . However,

a=1

a=1

a=0

S0

S1

S2

a=1

a=0

a=0

the reward would then disappear, and policy evaluation on the second step would lead it to then move right to s 0 and then s 2 , where it would stay for the remainder of the episode. In contrast, performing PI with the correct values of λ would lead the agent to go right to s 2 and stay there. In the first two timesteps, the first policy nets a total reward of 10 + 0 = 10 , while the second policy nets 6 + 5 . 94 = 11 . 94 . (The remaining decisions are identical between the two policies.) This is a clear example of the benefit of having the correct λ , as incorrect value estimation leads to suboptimal decisions even when the correct reward vector/function is provided at each step.

H The λ Operator

To learn the λ O, we would like to define Φ π λ ( s t , ds ′ ) ≜ φ π λ ( s t , s ′ ) µ ( ds ′ ) for some base policy µ . However, this would lead to a contradiction:

for all disjoint A, B , contradicting Lemma B.4.

For now, we describe how to learn the λ O for discrete S , in which case we have Φ π λ ( s, s ′ ) = φ π λ ( s, s ′ ) µ ( s ′ ) , i.e., by learning φ we learn a weighted version of Φ . We define the following norm, inspired by Touati et al. [25]:

where µ is any density on S . In the case of finite S , we let µ be the uniform density. We then minimize the Bellman error for Φ π λ with respect to ∥ · ∥ 2 ρ,µ (dropping the sub/superscripts on Φ and φ for clarity):

Note that we recover the SM loss when λ = 1 . Also, an interesting interpretation is that when the agent can never return to its previous state (i.e., φ ( s t +1 , s t ) = 0 ), then we also recover the SM loss, regardless of λ . In this way, the above loss appears to 'correct' for repeated state visits so that the measure only reflects the first visit.

Even though the λ O is not a measure, we can use the above loss to the continuous case, pretending as though we could take the Radon-Nikodym derivative Φ( s,ds ′ ) µ ( ds ′ ) .

H.1 Experimental Results with the FB Parameterization

To show that knowing the correct value of λ leads to improved performance, we trained λ O with the FB parameterization on the FourRooms task of Fig. 5.3, but with each episode initialized at a random start state and with two random goal states. Average per-epoch reward is shown in Fig. H.2. We tested performance with λ true , λ agent ∈ { 0 . 5 , 1 . 0 } , where λ true denotes the true environment diminishing rate and λ agent denotes the diminishing rate that the agent uses. For the purpose of illustration, we do not allow the agent to learn λ . We see in Fig. H.2 that using the correct λ leads to significantly increased performance. In particular, the left plot shows that assuming λ = 1 , i.e., using the SR, in a diminishing environment can lead to highly suboptimal performance.

Hyperparameters used are given in Table 1 (notation as in Algorithm 3).

0.5

1.0

agent

0

1

2

3

Average Expert Per-Epoch Reward

true =0.5

0.5

1.0

agent

0.0

12.5

25.0

37.5

Average Expert Per-Epoch Reward

true =1.0

H.2 λ Oand the Marginal Value Theorem

To study whether the agent's behavior is similar to behavior predicted by the MVT, we use a very simple task with constant starting state and vary the distance between rewards (see Fig. H.1(a)). When an agent is in a reward state, we define an MVT-optimal leaving time as follows (similar to that of [8] but accounting for the non-stationarity of the reward).

Let R denote the average per-episode reward received by a trained agent, r ( s t ) denote the reward received at time t in a given episode, R t = ∑ t u =0 r ( s u ) denote the total reward received until time t in the episode, and let T be episode length. Then, on average, the agent should leave its current reward state at time t if the next reward that it would receive by staying in s t , i.e., λr ( s t ) , is less than

In other words, the agent should leave a reward state when its incoming reward falls below the diminished average per-step reward of the environment. We compute R by averaging reward received by a trained agent over many episodes.

Previous studies have trained agents that assume stationary reward to perform foraging tasks, even though the reward in these tasks is non-stationary. These agents can still achieve good performance and MVT-like behavior [8]. However, because

a

RewardForDistance=1

RewardForDistance=3RewardForDistance=5

1.0

: in Patch Relative to Discounted I

LAW

10

y=0.99

C

Time in Patch Relative to MVT

10

y =1.0

0.8

0.6

S0

0.4

0.2

0.0

-10

Distance between reward states

3

-10

1

Distance between reward states

3

5

they target the standard RL objective

which requires γ < 1 for convergence, optimal behavior is recovered only with respect to the discounted MVT , in which R (and in our case, R t ) weights rewards by powers of γ [8].

In Fig. H.1(b-c) we perform a similar analysis to that of [8] and show that, on average over multiple distances between rewards, λ O-FB performs similarly to the discounted MVT for γ = 0 . 99 and the standard MVT for γ = 1 . 0 . An advantage of the λ O is that it is finite for γ = 1 . 0 provided that λ < 1 . Hence, as opposed to previous work, we can recover the standard MVT without the need to adjust for discounting.

Hyperparameters used are given in Table 1 (notation as in Algorithm 3).

I SAC

Mitigating Value Overestimation One well-known challenge in deep RL is that the use of function approximation to compute values is prone to overestimation. Standard approaches to mitigate this issue typically do so by using two value functions and either taking the minimum min i ∈{ 1 , 2 } Q π i ( s, a ) to form the Bellman target for a given ( s, a ) pair [40] or combining them in other ways [41]. However, creating multiple networks is expensive in both computation and memory. Instead, we hypothesized that it might be possible to address this issue by using λ -based values. To test this idea, we modified the Soft Actor-Critic [SAC; 42] algorithm to compute λ Fs-based values by augmenting the soft value target T soft Q = r t + γ E V soft ( s t +1 ) , where V soft ( s t +1 ) is given by the expression

Aderivation as well as pseudocode for the modified loss is provided in Appendix E.5. Observe that for λ = 1 , we recover the standard SAC value target, corresponding to an assumed stationary reward. We apply this modified SAC algorithm, which we term λ -SAC to feature-based Mujoco continuous control tasks within OpenAI Gym [43]. We found that concatenating the raw state and action observations ˜ ϕ t = [ s t , a t ] and normalizing them to [0 , 1] make effective regressors to the reward. That is, we compute base features as

where b indexes ( s t , a t ) within a batch. Let X ∈ [0 , 1] B × D be the concatenated matrix of features for a batch, where D = dim( S ) + dim( A ) . Then,

where here r denotes the vector of rewards from the batch. In addition to using a fixed λ value, ideally we'd like an agent to adaptively update λ to achieve the best balance of optimism and pessimism in its value estimates. Following [44], we frame this decision as a multi-armed bandit problem, discretizing λ into three possible values { 0 , 0 . 5 , 1 . 0 } representing the arms of the bandit. At the start of each episode, a random value of λ is sampled from these arms and used in the value function update. The probability of each arm is updated using the Exponentially Weighted Average Forecasting algorithm [45], which modulates the probabilities in proportion to a feedback score. As in [44], we use the difference in cumulative (undiscounted) reward between the current episode ℓ and the previous one ℓ -1 as this feedback signal: R ℓ -R ℓ -1 . That is, the probability of selecting a given value of λ increases if performance is improving and decreases if it's decreasing. We use identical settings for the bandit algorithm as in [44]. We call this variant λ -SAC.

Weplot the results for SAC with two critics (as is standard), SAC with one critic, SAC with a single critic trained with λ F-based values (' x -SAC' denotes SAC trained with a fixed λ = x ), and λ -SAC trained on the HalfCheetah-v2 and Hopper-v2 Mujoco environments. All experiments were repeated over eight random seeds. HalfCheetah-v2 was found by [44] to support 'optimistic' value estimates in that even without pessimism to reduce overestimation it was possible to perform well. Consistent with this, we found that single-critic SAC matched the performance of standard SAC, as did 1-SAC (which amounts to training a standard value function with the auxiliary task of SF prediction). Fixing lower values of λ performed poorly, indicating that over-pessimism is harmful in this environment. However, λ -SAC eventually manages to learn to set λ = 1 and matches the final performance of the best fixed algorithms. Similarly, in [44] it was observed that strong performance in Hopper-v2 was associated with pessimistic value estimates. Consistent with this, λ -SAC learns to select lower values of λ , again matching the performance of SAC while only requiring one critic and significantly reducing the required FLOPS Fig. I.2. We consider these results to be very preliminary, and hope to perform more experiments on other environments. We also believe λ -SAC could be improved by using the difference between the current episode's total reward and the average of the total rewards from previous episodes R ℓ -( ℓ -1) -1 ∑ ℓ -1 i =1 R i as a more

stable feedback signal for the bandit. There is also non-stationarity in the base features due to the per-batch normalization, which could also likely be improved. Hyperparameters are described in Table 2.

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

1e6

-2500

0

2500

5000

7500

10000

12500

Test Reward

HalfCheetah-v2

SAC (2 critics)

SAC

1

-SAC

0.5

-SAC

0

-SAC

-SAC

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

1e6

0

500

1000

1500

2000

2500

Hopper-v2

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

1e6

0.0

0.2

0.4

0.6

0.8

1.0

Choice

HalfCheetah-v2

Hopper-v2

300000

350000

400000

FLOPS

13000

13200

13400

13600

13800

Final  T est Reward

HalfCheetah-v2

SAC (2 critics)

-SAC

300000

350000

400000

FLOPS

1600

1800

2000

2200

2400

2600

Hopper-v2

J Replenishing Rewards

We list below a few candidate reward replenishment schemes, which are visualized in Fig. J.1.

Time elapsed rewards

where m ( s, t ) is the time elapsed since the last visit to state s :

Due to the max term in m ( s, t ) , the corresponding representation

does not admit a closed-form recursion. However, we empirically tested a version of this type of reward with Q λ -learning in the TwoRooms environment, modified so that the exponent on λ is n ( s, t ) / (0 . 1 m ( s, t )) . This was done so that reward replenishes at a slow rate, reducing the deviation from the standard diminishing setting. Episode length was capped at H = 100 . All other settings are identical to the Q λ experiment described in Appendix E. Results are presented in Fig. J.2 and a GIF is included in the supplementary material.

Eligibility trace rewards

where λ d , λ r ∈ [0 , 1] are diminishment and replenishment constants, respectively. Denoting the corresponding representation by Ω π , i.e.,

we obtain the following recursion:

where M π γλ r denotes the successor representation of π with discount factor γλ r . This representation could be learned by alternating TD learning between Ω π and M π γλ r . We leave this to future work.

Total time rewards

where λ d , λ r ∈ [0 , 1] are diminishment and replenishment constants, respectively. The corresponding representation is

0

2

4

6

8

10

12

14

Time

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Reward

Time Elapsed

0

2

4

6

8

10

12

14

Time

0.75

0.80

0.85

0.90

0.95

1.00

Reward

Eligibility T race

0

2

4

6

8

10

12

14

Time

0.4

0.6

0.8

1.0

1.2

Reward

Total Time

0

100

200

300

400

500

Episodes

0

5

10

15

20

25

30

Return

Q

Time Elapsed Replenishing Rewards

=

0.5

=

1.0

which satisfies the following recursion:

✶ ✶ ✶ While neither the reward nor the representation are guaranteed to be finite, P π could be learned via TD updates capped at a suitable upper bound.

We now briefly discuss the interaction between the temporal discount factor γ commonly used in RL and the diminishing utility rate λ . The key distinction between the two is that all rewards decay in value every time step with respect to γ , regardless of whether a state is visited or not. With λ , however, decay is specific to each state (or ( s, a ) pair) and only occurs when the agent visits that state. Thus, γ decays reward in a global manner which is independent of the agent's behavior, and λ decays reward in a local manner which dependent on the agent's behavior. In combination, they have the beneficial effect of accelerating convergence in dynamic programming (Fig. F.3). This indicates the potential for the use of higher discount factors in practice, as paired with a decay factor λ , similar (or faster) convergence rates could be observed even as agents are able to act with a longer effective temporal horizon.

L Compute Resources

The λ F-based experiments shown were run on a single NVIDIA GeForce GTX 1080 GPU. The recurrent A2C experiments took roughly 30 minutes, base feature learning for policy composition took approximately 45 minutes, λ F learning for policy composition took approximately 10 hours, and the SAC experiments took approximately 8 hours per run. The λ F training required roughly 30GB of memory due to the size of the dataset. All experiments in Section 6 and Appendix H were run on a single RTX5000 GPU and each training and evaluation run took about 30 minutes. All other experiments were run on a 2020 MacBook Air laptop 1.1 GHz Quad-Core Intel Core i5 CPU and took less than one hour to train.

M Broader Impact Statement

We consider this work to be primarily of a theoretical nature pertaining to sequential decision-making primarily in the context of natural intelligence. While it may have applications for more efficient training of artificial RL agents, it is hard to predict long-term societal impacts.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study a problem of DMU in RL, e.g. if visiting the same state may result in a smaller reward. They introduce a \\lambda R representation that considers a particular form of diminishing rewards and provides convergence results. They extend it to continuous domains and perform preliminary experimental studies.\n\nSTRENGTHS:\n- Typically, such problems having non-markovian rewards are hard, but by having assumptions of \\lamda rewards representation, the authors are able to recover some near-optimality convergence results.\n\n- Extension to the continuous domain\n\nWEAKNESSES:\nI think the assumption of decay with \\lambda is quite strong (very restrictive), and it's hard to imagine practical problems satisfying this. E.g. Authors have motivated the problem with examples of ice cream, finance and neuroscience, but it is hard(impossible) that a decay of visiting again is exactly a factor of 0<\\lambda<1. Arguably a more interesting setting to consider would be:\n- \\lambda R for is a lower or upper bound on the true rewards that one may earn after visiting the states multiple times, i.e., the reward will decay at least by that amount or at max by a certain amount if visited again. \n- I think another interesting setting will be if \\lambda changes with state visitation frequency and thus needs to be estimated.\n\nBoth these settings will consider quite a large part of DMU-type problems.\n\nLemma 3.1 is perhaps not the right way to impose Impossibility/hardness results, and the proof needs to be rigorous. I think it can be interpreted in multiple ways, and also, the proof considers a specific format in which the reward is written and a specific definition of the value function. For the lemma statement to be true, the proof shall eradicate all the ways in which value function and rewards can be defined. For e.g. consider a reward function $r(s) = F(\\tau \\cup s) - F(\\tau)$, where $F(\\tau)$ is a general set function, $\\tau$ is the agent's trajectory. $F(\\tau)$ denotes the cumulative value earned for a trajectory $\\tau$ as per the reward function (eqn 3.1). This can be written as Bellman recursion with $r(s_i) = F(s_{0:i}) - F(s_{0:i-1})$ (Imagine $\\tau = \\{s_0, s_1, s_2 ..... \\}$). I would suggest not having it as a lemma but just in the text mentioning it is difficult to solve it with the Bellman equation directly.\n\nExperiments: I think authors should consider more diverse reward functions, e.g., coverage functions, experiment design, and informative path planning objectives. For e.g., consider a camera-type measurement model, and if we observe the same region again, I will not get much information (there can be some noise model to establish \\lamda type of decay). Similarly, in experiment design/ informative planning objectives: Information gained on the nearby states will decrease if I gather/collect measurements at my current location.\n\nQUESTIONS:\nCan you mention the assumptions behind Proposition 4.1? It is counterintuitive for me. $\\lambda=0$ (FR) should be a much harder problem as compared to $\\lamda=1$ (SR). However, error bound is better for $\\lambda=0$?\n\nI would suggest authors add an algorithm block for a high-level understanding of the algorithm.\n\nI did not fully understand the experiments.\n- Are your environments deterministic?\n- In fig 5.2 b), why does \\lambda =1 perform well? I would expect it to stay at one of the blue cells forever since it assumes that it gets the same reward always, and on the plot, it shall have an exponential decay.\n- Is there an assumption of finite state-action spaces, or can you extend it to continuous dynamics as well?\n\nTypos: line 268 the the, 156 when when,\n\nLIMITATIONS:\nno societal negative impact", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

comments: SUMMARY:
This paper studies the phenomenon of diminishing marginal utility in RL, where the a state-based reward r(s) decays as the agent visits it more often. To solve this problem setting, this work introduced a novel state representation, named the λ representation (λR). The author showed that we fail to define a Bellman equation in the λR setting and introduced a recursive relationship.

STRENGTHS:
- The probelm setting of DMU in RL is interesting.
- The writing is clear and easy to follow.
- The paper provides theoretical analysis of the λ representations.

WEAKNESSES:
- Typos and grammar error: line 156 (when when ...), line 156 (Instead of keeping ...), line 177 (The mechanism)
- Missing baselines: the current experiments do not compare to other SOTA exploration baselines with intrinsic rewards. So it's hard to evaluate the effectiveness of the proposed methods.
- Experiments for continuous control is only conducted on Halfcheetah, a toy example that most RL baselines can easily solve.  Experiments on more complex tasks, i.e., humanoid-run, hopper-hop, acrobot-swingup, fish-swim, from dm_control would be more persuasive.

QUESTIONS:
- Since exploration is the natural setting for  λR, why there is no comparisons to different exploration baselines, i.e., RND, state entropy, pseudo-count ...?
- I am curious of how will the λR perform in more complex pixel-based tasks.

LIMITATIONS:
- The application of the proposed method seems to be narrow, and there is a lack of comparisons to SOTA exploration baselines.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the case of non-Markov rewards, where rewards at states decay over time (motivated by the diminishing marginal utility phenomenon). While successor representations (SR) have been used in standard MDPs, the decaying of rewards means that they cannot be used here, and so the paper develops a \"successor-like\" representation that can handle this case, which generalises SR and similar other representations. Theoretical results ensure that these representations can be learned using a Bellman backup (as in the case of SR), which can be used to evaluate and improve policies. Further extensions to the function approximation case (where features are required) are also demonstrated, and bounds related to composition (similar to the successor features bound) are presented. Results in tabular and pixel-based domains showcase how the representation can be used with model-free RL algorithms to solve this class of tasks, where taking into account the decaying rewards (instead of assuming rewards are stationary) is shown to be beneficial.\n\nSTRENGTHS:\nThe paper considers an interesting extension to the typical MDPs considered in reinforcement learning. The motivation regarding natural behaviours is an interesting one, but I think on a more fundamental level, any approach that attacks decision problems in RL that aren't Markovian in some way is of interest to the field.\n\nThis paper packs a lot into it - it develops many different ideas, ranging from the $\\lambda$ representation in the tabular case to the function approximation \"feature\" case. It also shows how these can be used to evaluate and learn policies, compose existing value functions, and be applied in a deep RL context. \n\nAside from the extensive theory presented, the paper also does a good job of relating the approach to existing work in successor representations and demonstrating under what conditions that approach here simplifies to existing work. This helps to unify these otherwise disparate methods.\n\nWEAKNESSES:\nAs presented, the experiments demonstrate that the method can be applied in both tabular and function approximation settings, that using the right value for $\\lambda$ is necessary to achieve the best performance, and that composition can be achieved just as in the SR/SF case. While these experiments confirm the utility of the representation and go hand in hand with the theory, it would have been nice to see the performance of the approach compared to different baselines. This could answer questions like how it compares to SR/FR/SM under a variety of conditions: when $\\lambda < 1$ or when $\\lambda =1$ or when states are only ever visited once (because it's continuous/very high dimensional).\n\nThe pixel domain was helpful to showcase that the approach can be applied in high-dimensional domains like this, but it would have been nice to see a more \"compositional\" domain as has been demonstrated in previous work to investigate composition (e.g. the domains where the agent collects objects of different colours/shapes with different priorities) [1,2,3]. \n\nI realise space is an issue, but it would have been helpful to see a very brief discussion on related work that focuses on non-Markov rewards (outside of the context of successor representations). e.g. [4]\n\nMinor:\n\nSome of the drawbacks of composition with SF are also inherited by this approach. For instance, the bound in Theorem 5.1 is pretty loose (see [2]), but I understand the point is to show it's the same bound with an extra term that is positive only when the $\\lambda$ estimates are wrong, so it's not very important\n\nWhile I found the motivation for considering these forms of non-Markovian rewards interesting, it may be the case that many RL practitioners who are not interested in modelling animal/real-world behaviour will not adopt this since for many practical applications (or popular benchmarks like Atari or MuJoCo), such a formulation is not natural. \n\nSome of the contractions used made it hard to parse sentences (e.g. L74). I'd recommend replacing these with the full words.\n\n[1] Barreto, André, et al. \"Fast reinforcement learning with generalised policy updates.\" Proceedings of the National Academy of Sciences 117.48 (2020): 30079-30087.\n\n[2] Nangue Tasse, Geraud, et al. \"Generalisation in lifelong reinforcement learning through logical composition.\" International Conference on Learning Representations. 2022.\n\n[3] Alver, Safa, and Doina Precup. \"Constructing a Good Behavior Basis for Transfer using Generalised Policy Updates.\" International Conference on Learning Representations. 2022.\n\n[4] Gaon, Maor, and Ronen Brafman. \"Reinforcement learning with non-markovian rewards.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 04. 2020.\n\nQUESTIONS:\nOn line 291, it states that there is not yet a measure-theoretic version of the representation. Having said that, while there may not be strict theoretical justification, would there be any issue in practice with applying this approach to continuous state/action domains?\n\nI'm slightly confused about the composition in the Four Rooms domain. There are four base policies but three goals. Is each policy to reach the centre of each room (and in three of the rooms, a goal exists)?\n\nLIMITATIONS:\nThe limitations are well described in Section 7, including many that I would not have even considered.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors explore diminishing marginal utility in the context of reinforcement learning. Specifically, they study reinforcement learning when reward obtained at a state diminishes with each visit to that state (following a particular mathematical form). The authors show that, under such diminishing rewards, agent policies can be evaluated by using a novel state representation introduced in the paper. This representation generalises earlier state representations in the literature, such as the successor representation.\n\nSTRENGTHS:\nOriginally: The concept of diminishing rewards is well known and explored in other fields. The paper presents new results within the context of reinforcement learning. \n\nQuality: The authors provide a thoughtful and useful analysis under the assumptions they make. This includes the discussion of how the proposed representation relates to existing state representations in RL. The recursion they have identified is interesting and potentially useful.\n\nThe paper is well structured. The writing is clear.\n\nWEAKNESSES:\nThe problem is not strongly motivated: why is studying diminishing marginal utility within the context of RL, with the specific assumptions made in this paper, important? Reasoning that goes beyond \"people have diminishing marginal utility\" would be useful. Why and how is this concept, in the form studied in the paper, useful for reinforcement learning agents? \n\nThe writing should clearly distinguish between two cases: (1) an objective quantity in the environment (e.g., available food) gets smaller, (2) the subjective value attached by the agent to an outside quantity gets smaller (e.g., the second ice cream cone does not taste as good). \n\nIn Section 6, the authors write that naturalistic environments often exhibit diminishing reward and give the example of foraging with diminishing food patches. I see a discrepancy between this example and the problem addressed in the rest of the paper. In the foraging example, the available food decreases because other agents consume it, and the amount of reduction should generally be a function of the time that passed between visits to the site. In contrast, the core assumption in the paper is that the reduction in reward in a given state is a function of how many times the agent itself visited that particular state, regardless of how much time passed since its last visit to that state. \n\nAdditional comments: Forward-backward representation should be explained in Section 2 (Preliminaries).\n\nQUESTIONS:\nI do not have any particular questions but, as I noted above, I do not find the paper particularly well-motivated in its present form. Why and how is diminishing marginal utility, in the form studied in the paper, useful for reinforcement learning agents? \n\nI have read the rebuttal and thank the authors for their additional thoughts on the subject.\n\nLIMITATIONS:\nYes, the limitations have been discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}