Title: State Sequences Prediction via Fourier Transform for Representation Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement learning

Representation learning

State sequences prediction

Fourier transform
Abstract: While deep reinforcement learning (RL) has been demonstrated effective in solving complex control tasks, sample efficiency remains a key challenge due to the large amounts of data required for remarkable performance. Existing research explores the application of representation learning for data-efficient RL, e.g., learning predictive representations by predicting long-term future states. However, many existing methods do not fully exploit the structural information inherent in sequential state signals, which can potentially improve the quality of long-term decision-making but is difficult to discern in the time domain. To tackle this problem, we propose State Sequences Prediction via Fourier Transform (SPF), a novel method that exploits the frequency domain of state sequences to extract the underlying patterns in time series data for learning expressive representations efficiently. Specifically, we theoretically analyze the existence of structural information in state sequences, which is closely related to policy performance and signal regularity, and then propose to predict the Fourier transform of infinite-step future state sequences to extract such information. One of the appealing features of SPF is that it is simple to implement while not requiring storage of infinite-step future states as prediction targets. Experiments demonstrate that the proposed method outperforms several state-of-the-art algorithms in terms of both sample efficiency and performance.

State Sequences Prediction via Fourier Transform for Representation Learning

Abstract

While deep reinforcement learning (RL) has been demonstrated effective in solving complex control tasks, sample efficiency remains a key challenge due to the large amounts of data required for remarkable performance. Existing research explores the application of representation learning for data-efficient RL, e.g., learning predictive representations by predicting long-term future states. However, many existing methods do not fully exploit the structural information inherent in sequential state signals, which can potentially improve the quality of long-term decision-making but is difficult to discern in the time domain. To tackle this problem, we propose S tate Sequences P rediction via F ourier Transform (SPF), a novel method that exploits the frequency domain of state sequences to extract the underlying patterns in time series data for learning expressive representations efficiently. Specifically, we theoretically analyze the existence of structural information in state sequences, which is closely related to policy performance and signal regularity, and then propose to predict the Fourier transform of infinite-step future state sequences to extract such information. One of the appealing features of SPF is that it is simple to implement while not requiring storage of infinite-step future states as prediction targets. Experiments demonstrate that the proposed method outperforms several state-of-the-art algorithms in terms of both sample efficiency and performance. 2

1 Introduction

Deep reinforcement learning (RL) has achieved remarkable success in complex sequential decisionmaking tasks, such as computer games [1], robotic control [2], and combinatorial optimization [3]. However, these methods typically require large amounts of training data to learn good control, which limits the applicability of RL algorithms to real-world problems. The crucial challenge is to improve the sample efficiency of RL methods. To address this challenge, previous research has focused on representation learning to extract adequate and valuable information from raw sensory data and train RL agents in the learned representation space, which has been shown to be significantly more data-efficient [4-8]. Many of these algorithms rely on auxiliary self-supervision tasks, such as predicting future reward signals [4] and reconstructing future observations [5], to incorporate prior knowledge about the environment into the representations.

Due to the sequential nature of RL tasks, multi-step future signals inherently contain more features that are valuable for long-term decision-making than immediate future signals. Recent work has demonstrated that leveraging future reward sequences as supervisory signals is effective in improving the generalization performance of visual RL algorithms [9, 10]. However, we argue that the state sequence provides a more informative supervisory signal compared to the sparse reward signal. As shown in the top portion of Figure 1, the sequence of future states essentially determines future actions and further influences the sequence of future rewards. Therefore, state sequences maximally preserve the influence of the transition intrinsic to the environment and the effect of actions generated from the current policy.

Despite these benefits of state sequences, it is challenging to extract features from these sequential data through long-term prediction tasks. A substantial obstacle is the difficulty of learning accurate long-term

A state sequence under

prediction models for feature extraction. Previous methods propose making multi-step predictions using a one-step dynamic model by repeatedly feeding the prediction back into the learned model [1113]. However, these approaches require a high degree of accuracy in the one-step model to avoid accumulating errors in multi-step predictions [13]. Another obstacle is the storage of multi-step prediction targets. For instance, some method learns a specific dynamics model to directly predict multi-step future states [14], which requires significant additional memory to store multi-step future states as prediction targets.

To tackle these problems, we propose utilizing the structural information inherent in the sequential state signals to extract useful features, thus circumventing the difficulty of learning an accurate prediction model. In Section 4, we theoretically demonstrate two types of sequential dependency structures present in state sequences. The first structure involves the dependency between reward sequences and state sequences, where the state sequences implicitly reflect the performance of the current policy and exhibit significant differences under good and bad policies. The second structure pertains to the temporal dependencies among the state signals, namely the regularity patterns exhibited by the state sequences. By exploiting the structural information, representations can focus on the underlying critical features of long-term signals, thereby reducing the need for high prediction accuracy and improving training stability [13].

Building upon our theoretical analyses, we propose S tate Sequences P rediction via F ourier Transform (SPF), a novel method that exploits the frequency domain of state sequences to efficiently extract the underlying structural information of long-term signals. Utilizing the frequency domain offers several advantages. Firstly, it is widely accepted that the frequency domain shows the regularity properties of the time-series data [15-17]. Secondly, we demonstrate in Section 5.1 that the Fourier transform of state sequences retains the ability to indicate policy performance under certain assumptions. Moreover, Figure 1 provides an intuitive understanding that the frequency domain enables more effective discrimination of two similar temporal signals that are difficult to differentiate in the time domain, thereby improving the efficiency of policy performance distinction and policy learning.

Specifically, our method performs an auxiliary self-supervision task that predicts the Fourier transform (FT) of infinite-step state sequences to improve the efficiency of representation learning. To facilitate the practical implementation of our method, we reformulate the Fourier transform of state sequences as a recursive form, allowing the auxiliary loss to take the form of a TD error [18], which depends only on the single-step future state. Therefore, SPF is simple to implement and eliminates the requirements for storing infinite-step state sequences when computing the labels of FT. Experiments demonstrate that our method outperforms several state-of-the-art algorithms in terms of both sample efficiency and performance on six MuJoCo tasks. Additionally, we visualize the fine distinctions between the multi-step future states recovered from our predicted FT and the true states, which indicates that our representation effectively captures the inherent structures of future state sequences.

2 Related Work

Learning Predictive Representations in RL. Many existing methods leverage auxiliary tasks that predict single-step future reward or state signals to improve the efficiency of representation learning [2, 5, 19]. However, multi-step future signals inherently contain more valuable features for long-term decision-making than single-step signals. Recent work has demonstrated the effectiveness of using future reward sequences as supervisory signals to improve the generalization performance of visual RL algorithms [9, 10]. Several studies propose making multi-step predictions of state sequences using a one-step dynamic model by repeatedly feeding the prediction back into the learned model, which is applicable to both model-free [11] and model-based [12] RL. However, these approaches require a high degree of accuracy in the one-step model to prevent accumulating errors [13]. To tackle this problem, the existing method learned a prediction model to directly predict multi-step future states [14], which results in significant additional storage for the prediction labels. In our work, we propose to predict the FT of state sequences, which reduces the demand for high prediction accuracy and eliminates the need to store multi-step future states as prediction targets.

Incorporating the Fourier Features. There existing many traditional RL methods to express the Fourier features. Early works have investigated representations based on a fixed basis such as Fourier basis to decompose the function into a sum of simpler periodic functions [20, 18]. Another research explored enriching the representational capacity using random Fourier features of the observations. Moreover, in the field of self-supervised pre-training, neuro2vec [16] conducts representation learning by predicting the Fourier transform of the masked part of the input signal, which is similar to our work but needs to store the entire signal as the label.

3 Preliminaries

3.1 MDPNotation

In RL tasks, the interaction between the environment and the agent is modeled as a Markov decision process (MDP). We consider the standard MDP framework [21], in which the environment is given by the tuple M := ⟨S , A , R, P, µ, γ ⟩ , where S is the set of states, A is the set of actions, R : S × A × S → [ -R max , R max ] is the reward function, P : S × A × S → [0 , 1] is the transition probability function, µ : S - → [0 , 1] is the initial state distribution, and γ ∈ [0 , 1) is the discount factor. A policy π defines a probability distribution over actions conditioned on the state, i.e. π ( a | s ) . The environment starts at an initial state s 0 ∼ µ . At time t ≥ 0 , the agent follows a policy π and selects an action a t ∼ π ( ·| s t ) . The environment then stochastically transitions to a state s t +1 ∼ P ( ·| s t , a t ) and produces a reward r t = R ( s t , a t , s t +1 ) . The goal of RL is to select an optimal policy π ∗ that maximizes the cumulative sum of future rewards. Following previous work [22, 23], we define the performance of a policy π as its expected sum of future discounted rewards:

where τ := ( s 0 , a 0 , s 1 , a 1 , · · · ) denotes a trajectory generated from the interaction process and τ ∼ ( π, M ) indicates that the distribution of τ depends on π and the environment model M . For simplicity, we write J ( π ) and τ ∼ π as shorthand since our environment is stationary. We also interest about the discounted future state distribution d π , which is defined by d π ( s ) = (1 -γ ) ∑ ∞ t =0 γ t P ( s t = s | π, M ) . It allows us to express the expected discounted total reward compactly as

The proof of (2) can be found in [24] or Section A.1 in the appendix.

Given that we aim to train an encoder that effectively captures the useful aspects of the environment, we also consider a latent MDP M = 〈 S , A , R, P , µ, γ, 〉 , where S ⊂ R D for finite D and the action space A is shared by M and M . We aim to learn an embedding function ϕ : S → S , which connects the state spaces of these two MDPs. We similarly denote π ∗ as the optimal policy in M . For ease of notation, we use π ( ·| s ) := π ( ·| ϕ ( s )) to represent first using ϕ to map s ∈ S to the latent state space S and subsequently using π to generate the probability distribution over actions.

3.2 Discrete-Time Fourier Transform

The discrete-time Fourier transform (DTFT) is a powerful way to decompose a time-domain signal into different frequency components. It converts a real or complex sequence { x n } + ∞ n = -∞ into a complex-valued function F ( ω ) = ∑ ∞ n = -∞ x n e -jωn , where ω is a frequency variable. Due to the discrete-time nature of the original signal, the DTFT is 2 π -periodic with respect to its frequency variable, i.e., F ( ω +2 π ) = F ( ω ) . Therefore, all of our interest lies in the range ω ∈ [0 , 2 π ] that contains all the necessary information of the infinite-horizon time series.

It is important to note that the signals considered in this paper, namely state sequences, are real-valued, which ensures the conjugate symmetry property of DTFT, i.e., F (2 π -ω ) = F ∗ ( ω ) . Therefore, in practice, it suffices to predict the DTFT only on the range of [0 , π ] , which can reduce the number of parameters and further save storage space.

4 Structural Information in State Sequences

In this section, we theoretically demonstrate the existence of the structural information inherently in the state sequences. We argue that there are two types of sequential dependency structures present in state sequences, which are useful for indicating policy performance and capturing regularity features of the states, respectively.

4.1 Policy Performance Distinction via State Sequences

In the RL setting, it is widely recognized that pursuing the highest reward at each time step in a greedy manner does not guarantee the maximum long-term benefit. For that reason, RL algorithms optimize the objective of cumulative reward over an episode, rather than the immediate reward, to encourage the agent to make farsighted decisions. This is why previous work has leveraged information about future reward sequences to capture long-term features for stronger representation learning [9].

Compared to the sparse reward signals, we claim that sequential state signals contain richer information. In MDP, the stochasticity of a trajectory derives from random actions selected by the agent and the environment's subsequent transitions to the next state and reward. These two sources of stochasticity are modeled as the policy π ( a | s ) and the transition p ( s ′ , r | s, a ) , respectively. Both of them are conditioned on the current state. Over long interaction periods, the dependencies of action and reward sequences on state sequences become more evident. That is, the sequence of future states largely determines the sequence of actions that the agent selects and further determines the corresponding sequence of rewards, which implies the trend and performance of the current policy, respectively. Thus, state sequences not only explicitly contain information about the environment's dynamics model, but also implicitly reveal information about policy performance.

We provide further theoretical justification for the above statement, which shows that the distribution distance between two state sequences obtained from different policies provides an upper bound on the performance difference between those policies, under certain assumptions about the reward function.

Theorem 1. Suppose that the reward function R ( s, a, s ′ ) = R ( s ) is related to the state s , then the performance difference between two arbitrary policies π 1 and π 2 is bounded by the L1 norm of the difference between their state sequence distributions:

where P ( s 0 , s 1 , s 2 , . . . | π, M ) means the joint distribution of the infinite-horizon state sequence S = { s 0 , s 1 , s 2 , . . . } conditioned on the policy π and the environment model M .

The proof of this theorem is provided in Appendix A.1. The theorem demonstrates that the greater the difference in policy performance, the greater the difference in their corresponding state sequence distributions. When we adjust the ratio R max 1 -γ to take a relatively small value by scaling the reward, the theorem indicates that good and bad policies generate significantly different state sequence distributions. Furthermore, it confirms that learning via state sequences can significantly influence the search for policies with good performance.

4.2 Asymptotic Periodicity of States in MDP

Many tasks in real scenarios exhibit periodic behavior as the underlying dynamics of the environment are inherently periodic, such as industrial robots, car driving in specific scenarios, and area sweeping tasks. Take the assembly robot as an example, the robot is trained to assemble parts together to create a final product. When the robot reaches a stable policy, it executes a periodic sequence of movements that allow it to efficiently assemble the parts together. In the case of MuJoCo tasks, the agent also exhibits periodic locomotion when reaching a stable policy. We provide a corresponding video in the supplementary material to show the periodic locomotion of several MuJoCo tasks.

Inspired by these cases, we provide some theoretical analysis to demonstrate that, under some assumptions about the transition probability matrices, the state sequences in finite state space may exhibit asymptotically periodic behaviors when the agent reaches a stable policy.

Theorem 2. Suppose that the state space S is finite with a transition probability matrix P ∈ R |S|×|S| and S has α recurrent classes. Let R 1 , R 2 , . . . , R α be the probability submatrices corresponding to the recurrent classes and let d 1 , d 2 , . . . , d α be the number of the eigenvalues of modulus 1 that the submatrices R 1 , R 2 , . . . , R α has. Then for any initial distribution µ 0 , P n µ 0 is asymptotically periodic with period d = lcm( d 1 , d 2 , . . . , d α ) .

The proof of the theorem is provided in Appendix A.2. The above theorem demonstrates that there exist regular and highly-structured features in the state sequences, which can be used to learn an expressive representation. Note that in an infinite state space, if the Markov chain contains a recurrent class, then after a sufficient number of steps, the state will inevitably enter one of the recurrent classes. In this scenario, the asymptotic periodicity of the state sequences can be also analyzed using the aforementioned theorem. Furthermore, even if the state sequences do not exhibit a strictly periodic pattern, regularities still exist within the sequential data that can be extracted as representations to facilitate policy learning.

5 Method

In the previous section, we demonstrate that state sequences contain rich structural information which implicitly indicates the policy performance and regular behavior of states. However, such information is not explicitly shown in state sequences in the time domain. In this section, we describe how to effectively leverage the inherent structural information in time-series data.

5.1 Learning via Frequency Domain of State Sequences

In this part, we will discuss the advantages of leveraging the frequency pattern of state sequences for capturing the inherent structural information above explicitly and efficiently.

Based on the following theorem, we find that the FT of the state sequences preserves the property in the time domain that the distribution difference between state sequences controls the performance difference between the corresponding two policies, but is subject to some stronger assumptions.

Theorem 3. Suppose that S ⊂ R D the reward function R ( s, a, s ′ ) = R ( s ) is an nth-degree polynomial function with respect to s ∈ S , then for any two policies π 1 and π 2 , their performance difference can be bounded as follows:

where F ( k ) π ( ω ) denotes the DTFT of the time series S ( k ) = { s 0 k , s 1 k , s 2 k , . . . } for any integer k ∈ [1 , n ] and S ( k ) means the kth power of the state sequence produced by the policy π . The dimensionality of ω is the same as s .

We provide the proof in Appendix A.1. Similar to the analysis of Theorem 1, the above theorem shows that state sequences in the frequency domain can indicate the policy performance and can be leveraged to enhance the search for optimal policies. Furthermore, the Fourier transform can decompose the state sequence signal into multiple physically meaningful components. This operator enables the analysis of time-domain signals in a higher dimensional space, making it easier to

distinguish between two segments of signals that appear similar in the time domain. In addition, periodic signals have distinctive characteristics in their Fourier transforms due to their discrete spectra. We provide a visualization of the DTFT of the state sequences in Appendix E, which reveals that the DTFT of the periodic state sequence is approximately discrete. This observation suggests that the periodic information of the signal can be explicitly extracted in the frequency domain, particularly for the periodic cases provided by Theorem 2. For non-periodic sequences, some regularity information can still be obtained by the frequency range that the signals carry.

In addition to those advantages, the operation of the Fourier transforms also yields a concise auxiliary objective similar to the TD-error loss [18], which we will discuss in detail in the following section.

5.2 Learning Objective of SPF

In this part, we propose our method, S tate Sequences P rediction via F ourier Transform (SPF), and describe how to utilize the frequency pattern of state sequences to learn an expressive representation. Specifically, our method performs an auxiliary self-supervision task by predicting the discrete-time Fourier transform (DTFT) of infinite-step state sequences to capture the structural information in the state sequences for representation learning, hence improving upon the sample efficiency of learning.

Now we model the auxiliary self-supervision task. Given the current observation s t and the current action a t , we define the expectation of future state sequence ˜ s t over infinite horizon as

Then the discrete-time Fourier transform of ˜ s t is F ˜ s t ( ω ) = ∑ + ∞ n =0 [ ˜ s t ] n e -jωn , where ω represents the frequency variable. The discount factor γ in (5) is used to ensure the convergence of the Fourier transform and also serves as the contraction factor in the following Theorem 4. Since the state sequences are discrete-time signals, the corresponding DTFT is 2 π -periodic with respect to ω . Based on this property, a common practice for operational feasibility is to compute a discrete approximation of the DTFT over one period, by sampling the DTFT at discrete points over [0 , 2 π ] . In practice, we take L equally-spaced samples of the DTFT. Then the prediction target is a matrix with size L ∗ D , where D is the dimension of the state space. We can derive that the DTFT functions at successive time steps are related to each other in a recursive form:

The detailed derivation and the specific form of ˜ S t and Γ is provided in Appendix A.3.

Based on the recursive formula (6), we can obtain the prediction loss by computing the difference between the estimated Fourier value F π,p ( s t , a t ) and the better estimate ˜ S t +Γ E π,p [ F ( s t +1 , a t +1 )] , just like the TD error. Similar to the TD-learning of value functions, the recursive relationship can be reformulated as contraction mapping T , as shown in the following theorem (see proof in Appendix A.3). Due to the properties of contraction mappings, we can iteratively apply the operator T to compute the target DTFT function of long-term state sequences until convergence in tabular settings. When calculating the prediction loss, we only need to utilize the current state s t , the current action a t , and the next state s t +1 . Therefore, one notable advantage of SPF is that there is no need to store multi-step future states as labels for predicting future state sequences.

Theorem 4. Let F denote the set of all functions F : S × A → C L ∗ D and define the norm on F as

where [ F ( s, a ) ] k represents the kth row vector of F ( s, a ) . We show that the mapping T : F → F defined as

is a contraction mapping, where ˜ S t and Γ are defined in Appendix A.3.

As the Fourier transform of the real state signals has the property of conjugate symmetry, we only need to predict the DTFT on a half-period interval [0 , π ] . Therefore, we reduce the row size of the

Online Encoder

Actor

Critic

Critic Loss

Predictor

Projection

Target Encoder

Target

Predictor

Target

Projection

target

target

Target

Target

Target

Target

Prediction

Loss

Actor Loss

prediction target to half for reducing redundant information and saving storage space. In practice, we train a parameterized prediction model F to predict the DTFT of state sequences. Note that the value of the prediction target is on the complex plane, so the prediction network employs two separate output modules F Re and F Im as real and imaginary parts respectively. Then we define the auxiliary prediction loss function as:

where s t = ϕ ( s t ) means the representation of the state, Γ Re and Γ Im denote the real and imaginary parts of the constant Γ , and d denotes an arbitrary similarity measure. We choose d as cosine similarity in practice. The algorithm pseudo-code is shown in Appendix B.

5.3 Network Architecture of SPF

Here we provide more details about the network architecture of our method. In addition to the encoder and the predictor, we use projection heads ψ [25] that project the predicted values onto a low-dimensional space to prevent overfitting when computing the prediction loss directly from the high-dimensional predicted values. In practice, we use a loss function called freqloss , which preserves the low and high-frequency components of the predicted DTFT without the dimensionality reduction process (See Appendix C for more details). Furthermore, when computing the target predicted value, we follow prior work [26, 11] to use the target encoder, predictor, and projection for more stable performance. We periodically overwrite the target network parameters with an exponential moving average of the online network parameters.

In the training process, we train the encoder ϕ , the predictor F , and the projection ψ to minimize the auxiliary prediction loss L pred ( ϕ, F , ψ ) , and alternately update the actor-critic models of RL tasks using the trained encoder ϕ . We illustrate the overall architecture of SPF and the gradient flows during training in Figure 2.

6 Experiments

We quantitatively evaluate our method on a standard continuous control benchmark-the set of MuJoCo [27] environments implemented in OpenAI Gym.

6.1 Comparative Evaluation on MuJoCo

To evaluate the effect of learned representations, we measure the performance of two traditional RL methods SAC [28] and PPO [29] with raw states, OFENet representations [19], and SPF represen-

Environment

SAC-SPF

SAC-OFE

SAC-raw

PPO-SPF

PPO-OFE

PPO-raw

500K step

HalfCheetah

12739

±

139

(+7%)

11932

±

75

9004

±

150

2448

±

357

(+1%)

2419

±

296

1475

±

238

Hopper

3443

±

24

(+15%)

2983

±

123

2343

±

183

2923

±

98

(+36%)

1529

±

157

2156

±

245

Walker2d

4868

±

53

(+29%)

3762

±

162

2023

±

269

1203

±

211

(+45%)

339

±

32

827

±

156

Ant

6531

±

175

(+17%)

5587

±

253

2691

±

76

867

±

34

696

±

47

929

±

15

Swimmer

45

±

0

45

±

0

42

±

0

90

±

11

(+15%)

78

±

12

66

±

6

Humanoid

4613

±

118

(+13%)

4071

±

172

2953

±

136

395

±

10

405

±

18

383

±

9

1M step

HalfCheetah

15822

±

109

(+10%) 14425

±

112

10745

±

159

3154

±

1358

(+3%) 3066

±

326

2259

±

344

Hopper

3517

±

20

(+10%)

3197

±

147

3056

±

91

3152

±

137

(+16%) 2370

±

258

2721

±

222

Walker2d

5158

±

69

(+7%)

4833

±

59

3367

±

110

2229

±

299

1080

±

236

2302

±

158

Ant

7241

±

85

(+7%)

6738

±

150

4220

±

164

1053

±

79

(+15%)

913

±

18

867

±

17

Swimmer

45

±

0

45

±

0

43

±

0

99

±

13

(+19%)

73

±

10

83

±

9

Humanoid

5633

±

112

6241

±

98

4618

±

118

441

±

17

448

±

17

439

±

18

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Steps

1e6

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

Evaluation returns

1e4

HalfCheetah

0.0

0.2

0.4

0.6

0.8

1.0

Steps

1e6

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Evaluation returns

1e3

Hopper

0

1

2

3

4

5

Steps

1e6

0

1

2

3

4

5

6

Evaluation returns

1e3

Walker2d

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Steps

1e6

0.0

0.2

0.4

0.6

0.8

1.0

Evaluation returns

1e4

Ant

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Steps

1e6

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Evaluation returns

1e2

Swimmer

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Steps

1e6

0

2

4

6

8

Evaluation returns

1e3

Humanoid

SAC-SPF

SAC-OFE

SAC-raw

PPO-SPF

PPO-OFE

PPO-raw

tations. We train our representations via the Fourier transform of infinite-horizon state sequences, which means the representations learn to capture the information of infinite-step future states. Hence, we compare our representation with OFENet, which uses the auxiliary task of predicting singlestep future states for representation learning. For a fair comparison, we use an encoder with the same network structure as that of OFENet. See Appendix C and D for more details about network architectures and hyperparameters setting.

As described above, our comparable methods include: 1) SAC-SPF : SAC with SPF representations; 2) SAC-OFE : SAC with OFENet representations; 3) SAC-raw : SAC with raw states; 4) PPO-SPF : PPO with SPF representations; 5) PPO-OFE : PPO with OFENet representations; 6) PPO-raw : PPO with raw states. Figure 3 shows the learning curves of the above methods. As our results suggest, SPF shows superior performance compared to the original algorithms, SAC-raw and PPO-raw, across all six MuJoCo tasks and also outperforms OFENet in terms of both sample efficiency and asymptotic performance on five out of six MuJoCo tasks. According to the results in Table 1, SPF achieves an average gain of +19.5% over other methods at 500 K step and 1 Mstep. These results indicate that our approach learns more quickly than other methods when the number of interactions is limited. One possible explanation for the comparatively weaker performance of our method on the Humanoid-v2

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Steps

1e6

0

2500

5000

7500

10000

12500

15000

17500

Evaluation returns

HalfCheetah

SPF

notarg

nofreqloss

noproj

mlp

mlp_cat

(a)

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Steps

1e6

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Evaluation returns

1e4

HalfCheetah

SPF

Sta1

Sta2

Sta3

Sta5

Rew1

RewFT

(b)

0

50

100

150

200

Horizon

Future states

Walker2d's state 13

0

50

100

150

200

Horizon

Future states

Walker2d's state 14

0

50

100

150

200

Horizon

Future states

Walker2d's state 15

0

50

100

150

200

Horizon

Future states

Walker2d's state 16

0

50

100

150

200

Horizon

Future states

Walker2d's state 14

true

1-step pred

2-step pred

3-step pred

4-step pred

5-step pred

(c)

task is that its state contains the external forces to bodies, which show limited regularities due to their discontinuous and sparse.

6.2 Ablation Study

In this part, we will verify that just predicting the FT of state sequences may fall short of the expected performance and that using SPF is necessary to get better performance. To this end, we conducted an ablation study to identify the specific components that contribute to the performance improvements achieved by SPF. Figure 4(a) shows the ablation study over SAC with HalfCheetah-v2 environment.

notarg removes all target networks of the encoder, predictor, and projection layer from SPF. Based on the results, the variant of SPF exhibits significantly reduced performance when target estimations in the auxiliary loss are generated by the online encoder without a stopgradient. Therefore, using a separate target encoder is vital, which can significantly improve the stability and convergence properties of our algorithm.

nofreqloss computes the cosine similarity of the projection layer's outputs directly without any special treatment to the low-frequency and high-frequency components of our predicted DTFT. The reduced convergence rate of nofreqloss suggests that preserving the complete information of low and high-frequency components can encourage the representations to capture more structural information in the frequency domain.

noproj removes the projection layer from SPF and computes the cosine similarity of the predicted values as the objective. The performance did not significantly deteriorate after removing the projection layer, which indicates that the prediction accuracy of state sequences in the frequency domain may not have a strong impact on the quality of representation learning. Therefore, it can be inferred that SPF places a greater emphasis on capturing the underlying structural information of the state sequences, and is capable of reconstructing the state sequences with a low risk of overfitting.

mlp changes the lay block of the encoder from MLP-DenseNet to MLP. The much lower scores of mlp indicate that both the raw state and the output of hidden layers contain important information that contributes to the quality of the learned representations. This result underscores the importance of leveraging sufficient information for representation learning.

mlp-cat uses a modified block of MLP as the layer of the encoder, which concat the output of MLP with the raw state. The performance of mlp-cat does increase compared to mlp , but is still not as good as SPF in terms of both sample efficiency and performance.

6.3 Comparison of Different Prediction Targets

This section aims to test the effect of our prediction target-infinite-step state sequences in the frequency domain-on the efficiency of representation learning. We test five types of prediction

targets: 1) Sta1 : single-step future state; 2) StaN : N-step state sequences, where we choose N = 2 , 3 , 5 ; 3) SPF : infinite-step state sequences in frequency domain; 4) Rew1 : single-step future reward; 5) RewFT : infinite-step reward sequences in frequency domain.

As shown in Figure 4(b), SPF outperforms all other competitors in terms of sample efficiency, which indicates that infinite-step state sequences in the frequency domain contain more underlying valuable information that can facilitate efficient representation learning. Since Sta1 and SPF outperform Rew1 and RewFT respectively, it can be referred that learning via states is more effective for representation learning than learning via rewards. Notably, the lower performance of StaN compared to Sta1 could be attributed to the model's tendency to prioritize prediction accuracy over capturing the underlying structured information in the sequential data, which may impede its overall learning efficiency.

6.4 Visualization of Recovered State Sequences

This section aims to demonstrate that the representations learned by SPF effectively capture the structural information contained in infinite-step state sequences. To this end, we compare the true state sequences with the states recovered from the predicted DTFT via the inverse DTFT (See Appendix E for more implementation details). Figure 4(c) shows that the learned representations can recover the true state sequences even using the historical states that are far from the current time step. In Appendix E, we also provide a visualization of the predicted DTFT, which is less accurate than the results in Figure 4(c). Those results highlight the ability of SPF to effectively extract the underlying structural information in infinite-step state sequences without relying on high prediction accuracy.

We further provide a comparison table that measures the distance between the real DTFT and the predicted DTFT using cosine similarity. The results provided in Appendix F indicate that the prediction module F exhibits moderate predictive accuracy in approximating the real Fourier transform, with an average cosine similarity value of -0 . 6 .

7 Conclusion

In this paper, we theoretically analyzed the existence of structural information in state sequences, which is closely related to policy performance and signal regularity, and then introduced State Sequences Prediction via Fourier Transform (SPF), a representation learning method that predicts the FT of state sequences to extract the underlying structural information in state sequences for learning expressive representations efficiently. SPF outperforms several state-of-the-art algorithms in terms of both sample efficiency and performance. Our additional experiments and visualization show that SPF encourages representations to place a greater emphasis on capturing the underlying pattern of time-series data, rather than pursuing high accuracy of prediction tasks.

Limitations One of the main limitations of our paper is that we have only evaluated our method on tasks where the state sequences exhibit strong asymptotic periodicity. Considering that the Fourier transform converts non-periodic signals into a continuous frequency domain, it is more difficult to extract meaningful frequency features of non-periodic signals compared to periodic signals that have a discrete frequency domain. Moreover, the frequency features extracted by our approach inherently depend on the policy and task, which limits their reusability across multiple tasks. Further research is needed to analyze the applicability of our approach to non-periodic tasks and the potential for generalization across multiple tasks.

Acknowledgments

The authors would like to thank all the anonymous reviewers for their insightful comments. This work was supported by National Key R&D Program of China under contract 2022ZD0119801, National Nature Science Foundations of China grants U19B2026, U19B2044, 61836011, 62021001, and 61836006.

References

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540):529-533, 2015.

Tim de Bruin, Jens Kober, Karl Tuyls, and Robert Babuska. Integrating state representation learning into deep reinforcement learning. IEEE Robotics Autom. Lett. , 3(2):1394-1401, 2018.

Thomas D. Barrett, William R. Clements, Jakob N. Foerster, and A. I. Lvovsky. Exploratory combinatorial optimization with reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 3243-3250, 2020.

Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In 5th International Conference on Learning Representations (ICLR) , 2017.

Jelle Munk, Jens Kober, and Robert Babuska. Learning state representation for deep actor-critic control. In 55th IEEE Conference on Decision and Control (CDC) , pages 4667-4673, 2016.

Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. In 9th International Conference on Learning Representations (ICLR) , 2021.

Zhihai Wang, Jie Wang, Qi Zhou, Bin Li, and Houqiang Li. Sample-efficient reinforcement learning via conservative model-based actor-critic. In Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI) , pages 8612-8620, 2022.

Zhihai Wang, Taoxing Pan, Qi Zhou, and Jie Wang. Efficient exploration in resource-restricted reinforcement learning. In Brian Williams, Yiling Chen, and Jennifer Neville, editors, Thirty-Seventh AAAI Conference on Artificial Intelligence (AAAI) , pages 10279-10287, 2023.

Rui Yang, Jie Wang, Zijie Geng, Mingxuan Ye, Shuiwang Ji, Bin Li, and Feng Wu. Learning taskrelevant representations for generalization via characteristic functions of reward sequence distributions. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 2242-2252, 2022.

Jie Wang, Rui Yang, Zijie Geng, Zhihao Shi, Mingxuan Ye, Qi Zhou, Shuiwang Ji, Bin Li, Yongdong Zhang, and Feng Wu. Generalization in visual reinforcement learning with the reward sequence distribution. 2023. doi: 10.48550/arXiv.2302.09601.

Max Schwarzer, Ankesh Anand, Rishab Goel, R. Devon Hjelm, Aaron C. Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. In 9th International Conference on Learning Representations, ICLR , 2021.

Danijar Hafner, Timothy P. Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In Proceedings of the 36th International Conference on Machine Learning(ICML) , volume 97, pages 2555-2565, 2019.

Zhaohan Daniel Guo, Bernardo Ávila Pires, Bilal Piot, Jean-Bastien Grill, Florent Altché, Rémi Munos, and Mohammad Gheshlaghi Azar. Bootstrap latent-predictive representations for multitask reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning(ICML) , volume 119, pages 3875-3886, 2020.

Nikhil Mishra, Pieter Abbeel, and Igor Mordatch. Prediction and control with temporal segment models. In Proceedings of the 34th International Conference on Machine Learning(ICML) , volume 70, pages 2459-2468, 2017.

Sungwook Moon and Ahmed Helmy. Understanding periodicity and regularity of nodal encounters in mobile networks: A spectral analysis. In Proceedings of the Global Communications Conference. GLOBECOM , pages 1-5. IEEE, 2010.

Di Wu, Siyuan Li, Jie Yang, and Mohamad Sawan. neuro2vec: Masked fourier spectrum prediction for neurophysiological representation learning. arXiv preprint arXiv:2204.12440 , 2022.

Anthony Tompkins and Fabio Ramos. Fourier feature approximations for periodic kernels in time-series modelling. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence , pages 41554162. AAAI Press, 2018.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Kei Ota, Tomoaki Oiki, Devesh K. Jha, Toshisada Mariyama, and Daniel Nikovski. Can increasing input dimensionality improve deep reinforcement learning? In Proceedings of the 37th International Conference on Machine Learning, ICML , volume 119, pages 7424-7433, 2020.

Sridhar Mahadevan. Proto-value functions: Developmental reinforcement learning. In Proceedings of the 22nd international conference on Machine learning , pages 553-560, 2005.

Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming . Wiley Series in Probability and Statistics. Wiley, 1994.

Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proceedings of the 34th International Conference on Machine Learning , volume 70, pages 22-31, 2017.

Ofir Nachum and Mengjiao Yang. Provable representation learning for imitation with contrastive fourier features. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021 , pages 30100-30112, 2021.

Sham M. Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Machine Learning, Proceedings of the Nineteenth International Conference , pages 267-274, 2002.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, ICML , volume 119, pages 1597-1607, 2020.

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent - A new approach to selfsupervised learning. volume 33, pages 21271-21284, 2020.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In International Conference on Intelligent Robots and Systems, IROS , pages 5026-5033. IEEE, 2012.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, ICML , volume 80, pages 1856-1865, 2018.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR , 2017.

Geoffrey Grimmett and David Stirzaker. Probability and random processes . Oxford university press, 2020.

Eugene Seneta. Non-negative matrices and Markov chains . Springer Science & Business Media, 2006.

Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning, ICML , volume 80 of Proceedings of Machine Learning Research , pages 1582-1591, 2018.

A Proof

A.1 Proof of Performance Difference Distinction via State Sequences

Following the previous work [22], our analysis will make use of the discounted future state distribution, d π , which is defined as

It allows us to express the expected discounted total reward compactly as

where we define R π ( s ) := E a ∼ π,s ′ ∼ P [ R ( s, a, s ′ )] . It should be clear from a ∼ π ( ·| s ) and s ′ ∼ P ( ·| s, a ) that a and s ′ depend on s . Thus, the reward function R π is only related to s when the policy π is fixed.

Firstly, we prove that the distance between two state sequence distributions obtained from two distinct policies serves as an upper bound on the performance difference between those policies, provided that certain assumptions regarding the reward function hold.

Theorem 1. Suppose that the reward function R ( s, a, s ′ ) = R ( s ) is related to the state s , then the performance difference between two arbitrary policies π 1 and π 2 is bounded by the L1 norm of the difference between their state sequence distributions:

where P ( s 0 , s 1 , s 2 , . . . | π, M ) means the joint distribution of the infinite-horizon state sequence S = { s 0 , s 1 , s 2 , . . . } conditioned on the policy π and the environment model M .

Proof. According to the equation (1), the difference in performance between two policies π 1 , π 2 can be bounded as follows.

Let T →∞ , then we obtain the bound proposed by (3).

We are further interested in bounding the performance difference between two policies by their state sequences in the frequency domain. Benefiting from the properties of the discrete-time Fourier transform (DTFT), we can constrain the performance difference using the Fourier transform over the interval [0 , 2 π ] , instead of using the distribution functions of the state sequences in unbounded space.

Theorem 2. Suppose that S ⊂ R D the reward function R ( s, a, s ′ ) = R ( s ) is an nth-degree polynomial function with respect to s ∈ S , then for any two policies π 1 and π 2 , their performance difference can be bounded as follows:

where F ( k ) π ( ω ) denotes the DTFT of the time series S ( k ) = { s 0 k , s 1 k , s 2 k , . . . } for any integer k ∈ [1 , n ] and S ( k ) means the kth power of the state sequence produced by the policy π . The dimensionality of ω is the same as s .

Proof. For sake of simplicity, we define p t ( s | π i ) = P ( s t = s | π i , M ) for i = 1 , 2 . We denote ε t as

Based on the Taylor series expansion, we can rewrite the reward function as R ( s ) = ∑ n k =0 R ( k ) (0) T k ! s k , then for any integer k ∈ [1 , n ] , we have

Since the inverse DTFT of F ( k ) π ( ω ) is the original time series S ( k ) , we have

Then we have

Substituting (8) into (6), then we obtain

For the sake of DTFT, the upper bound of ϵ t is independent of t , then we could derive the performance difference bound as follows.

and so we immediately achieve the desired bound in (4).

A.2 Proof of the Asymptotic Periodicity of States in MDP

This section focuses on analyzing the asymptotic behavior of the state sequences generated from an MDP. We begin by discussing the limiting process of MDP with a finite state space S . Let P be the transition probability matrix and let µ i be the probability distribution of the states at time t i . Then we have µ i +1 = Pµ i for any i ≥ 0 . If the sequence { µ i } ∞ i =0 splits into d subsequences with d cyclic limits { µ r ∞ } d -1 r =0 that follow the cycle:

then we say that the states of the MDP exhibit asymptotic periodicity . Such cyclic asymptotic behavior implies that the limiting distribution of the states eventually repeats in a specific period after a certain number of steps.

We begin by providing some essential definitions in the field of stochastic processes [30], which will be utilized in the following proof. Let P be a transition probability matrix corresponding to n states ( n ≥ 1 ). Two states i and j are said to intercommunicate if there exist paths from i to j as well as from j to i . The matrix P is called irreducible if any two states intercommunicate. A set of states is called irreducible if any two states in the set intercommunicate. Moreover, a state i is called recurrent if the probability of eventual return to i , having started from i , is 1 . If this probability is strictly less than 1 , the state i is called transient .

Note that if the whole state space S is irreducible, then its transition matrix P is also irreducible. The following lemma demonstrates that if the state space is irreducible, then its asymptotical periodicity is determined by the eigenvalues with modulus 1 of its transition matrix.

Lemma 1. Suppose that the state space S is finite with a transition probability matrix P ∈ R |S|×|S| . If P is an irreducible matrix with d eigenvalues of modulus 1 , then for any initial distribution µ 0 , P n µ 0 is asymptotically periodic with a period of d when d > 1 and asymptotically aperiodic when d = 1 .

Proof. According to the Perron-Frobenius theorem for irreducible non-negative matrices, all eigenvalues of P of modulus 1 are exactly the d complex roots of the equation λ d -1 = 0 . They can be formulated as λ 0 = 1 , λ 1 = ξ 1 , . . . , λ d -1 = ξ d -1 , where ξ = e 2 πj d . Each of them is a simple root of the characteristic polynomial of the matrix P . Since P is a transition probability matrix, the remaining eigenvalues λ d , . . . , λ s satisfy | λ r | < 1 . Therefore, the Jordan matrix of P has the form

We refer to J k as Jordan cells .

Let | S | = D , we can rewrite P in its Jordan canonical form P = XJX -1 where

Note that for k < d , x k is the eigenvector corresponding to λ k . Since the column vectors of X are linearly dependent, there exist ⃗ c = [ c 0 , c 1 , . . . , c D -1 ] not all zero, such that µ 0 = D -1 ∑ k =0 c k ⃗ x k = X⃗ c . Thus, we have

For any Jordan cell J k , let α k be the multiplicity of λ k , then

Since α k is fixed for matrix P , we have lim n →∞ J n k = 0 for each k = d, . . . , D -1 . Then the limiting vector of (9), denoted by P ∞ µ 0 , satisfies:

where we denote µ ( n ) = d -1 ∑ k =0 c k ( e j 2 πk d ) n ⃗ x k . Let r = n (mod d ) , then we have

Therefore, the probability sequence { P n µ 0 } n ≥ 1 will split into d converging subsequences and has d cyclic limiting probability distributions when n →∞ , denoted as

Thus, P n µ 0 is asymptotically periodic with period d if d > 1 and asymptotically aperiodic if d = 1 .

We now consider a more general state space that may not necessarily be irreducible. According to the Decomposition theorem of the Markov chain [30], the finite state space S can be partitioned uniquely as a set of transient states and one or several irreducible closed sets of recurrent states. According to [31], after performing an appropriate permutation of rows and columns, we can rewrite the transition probability matrix P in its canonical form:

where R 1 , . . . , R α represent the probability submatrices corresponding to the recurrent classes, Q represents the probability submatrix corresponding to the transient states, and T 1 , . . . , T α represent the probability submatrices corresponding to the transitions between transient and recurrent classes R 1 , . . . , R α respectively.

Theorem 3. Suppose that the state space S is finite with a transition probability matrix P ∈ R |S|×|S| and S has α recurrent classes. Let R 1 , R 2 , . . . , R α be the probability submatrices corresponding to the recurrent classes and let d 1 , d 2 , . . . , d α be the number of the eigenvalues of modulus 1 that the submatrices R 1 , R 2 , . . . , R α has. Then for any initial distribution µ 0 , P n µ 0 is asymptotically periodic with period d = lcm( d 1 , d 2 , . . . , d α ) when d > 1 and asymptotically aperiodic when d = 1 .

Proof. Since P is a block upper-triangular, it can be shown that the eigenvalues of P are equal to the union of the eigenvalues of the diagonal blocks R 1 , . . . , R α , Q . Note that the n th-power of P satisfies the following expression:

where T ( n ) r is related to the ( n -1) -th or the lower power of R r and Q . From Theorem 4.3 of [31], we obtain that lim n →∞ Q n = 0 , which implies that all eigenvalues of Q have modulus less than 1 .

On the other hand, note that the sum of every row in matrix R r is equal to 1 , which means λ = 1 is an eigenvalue of R r and all eigenvalues of R r satisfy | λ | ≤ 1 . Thus, the spectral radius of P is equal to 1 .

Note that the proof of Lemma 1 implies that the asymptotic periodicity of P n µ 0 depends on the eigenvalues of P that have modulus 1 . Since R r is non-negative irreducible with spectral radius 1 , based on the Perron-Frobenius theorem used in Lemma 1, we can express the eigenvalues of R r in modulus 1 as:

Based on the above discussion, it is easy to check that α ⋃ r =1 { λ r, 0 , . . . , λ r,d r -1 } is the set of all eigenvalues of modulus 1 of P . Rewrite P in its Jordan canonical form P = XJX -1 , where

and X = [ ⃗ x 0 , ⃗ x 1 , . . . , ⃗ x D -1 ] is an invertible matrix. Similar to the proof in Lemma 1, we get

Let d = lcm( d 1 , d 2 , . . . , d α ) and r = n (mod d ) , then we have

Therefore, the probability sequence { P n µ 0 } n ≥ 1 will split into d converging subsequences and has d cyclic limiting probability distributions when n →∞ , denoted as

Thus, P n µ 0 is asymptotically periodic with period d if d > 1 and asymptotically aperiodic if d = 1 . This completes the proof.

A.3 Proof of the Convergence of Our Auxiliary Loss

In this section, we provide a detailed derivation of the learning objective of SPF. As the DTFT of discrete-time state sequences is a continuous function that is difficult to compute, we practically sample the DTFT at L equally-spaced points.

As a result, the prediction target takes the form of a matrix with dimensions of L ∗ D , where D denotes the dimension of the state space. The auxiliary task is designed to encourage the representation to predict the Fourier transform of the state sequences using the current state-action pair as input. Specifically, we define the prediction target F π,p ( s t , a t ) as follows:

For simplicity of notation, we substitute F ( s t , a t ) for F π,p ( s t , a t ) in the following. We can derive that the DTFT functions at successive time steps are related to each other in a recursive form:

We can further express the above equation as a matrix-form recursive formula as follows:

Similar to the TD-learning of value functions, we can prove that the above recursive relationship (12) can be reformulated as a contraction mapping T . Due to the properties of contraction mappings, we can iteratively apply the operator T to compute the target DTFT function until convergence in tabular settings.

Theorem 4. Let F denote the set of all functions F : S × A → C L ∗ D and define the norm on F as

where [ F ( s, a ) ] k represents the kth row vector of F ( s, a ) . We show that the mapping T : F → F defined as

is a contraction mapping, where ˜ S t and Γ are defined as above.

where

Proof. For any F 1 , F 2 ∈ F , we have

Note that γ ∈ [0 , 1) , which implies that T is a contraction mapping.

B Pseudo-code of SPF

The training procedure of SPF is shown in the pseudo-code as follows:

Algorithm 1

Denote parameters of the online encoder ( ϕ s , ϕ s,a ) , predictor F , and projection ψ as θ aux Denote parameters of the target encoder ( ̂ ϕ s , ̂ ϕ s,a ) , predictor ̂ F , and projection ̂ ψ as ̂ θ aux Denote parameters of actor model π and critic model Q for RL agents as θ RL Denote the smoothing coefficient and update interval for target network updates as τ and K Initialize replay buffer D and parameters θ aux , θ RL

update the RL agent parameters θ RL with the representations s t , z s t ,a t update parameters of target networks with ̂ θ aux ← τθ aux +(1 -τ ) ̂ θ aux every K steps

C Network Details

The encoders ϕ s and ϕ s,a share the same architecture. Each layer of the encoders uses MLPDenseNet [19], a slightly modified version of DenseNet. For each MuJoCo task, the incremental number of hidden units per layer is selected from { 30 , 40 } , while the number of layers is selected from { 6 , 8 } (see Table 2). Both the predictor F and the projection ψ apply a 2-layer MLP. We divide the last layer of the predictor into two heads as the real part F Re and the imaginary part F Im, respectively, since the prediction target of our auxiliary task is complex-valued. With respect to the projection module, we add an additional 2-layer MLP (referred to as Projection2 ) after the original online projection to perform a dimension-invariant nonlinear transformation on the predicted DTFT that has been projected to a lower-dimensional space. We do not apply this nonlinear operation to the target projection. This additional step is carried out to prevent the projection from collapsing to a constant value in the case where the online and target projections share the same architecture.

In Fourier analysis, the low-frequency components of the DTFT contain information about the long-term trends of the signal, with higher signal energy, while the high-frequency components of the

DTFT reflect the amount of short-term variation present in the state sequences. Therefore, we attempt to preserve the overall information of the low and high-frequency components of the predicted DTFT by directly computing the cosine similarity distance without undergoing the dimensionality reduction process. For the remaining frequency components of the predicted DTFT, we first utilize projection layers to perform dimensionality reduction, followed by calculating the cosine similarity distance. The sum of these three distances is used as the final loss function, which we call freqloss .

D Hyperparameters

Weselect L = 128 as the number of discrete points sampled over one period of DTFT. In practice, due to the symmetry conjugate of DTFT, the predictor F only predicts L 2 +1 points on the left half of our frequency map, as mentioned in Section 5.2. The projection module described in Section 5.3 projects the predicted value, a matrix with the dimension of L ∗ D , into a 512 -dimensional vector. To update target networks, we overwrite the target network parameters with an exponential moving average of the online network parameters, with a smoothing coefficient of τ = 0 . 01 for every K = 1000 steps.

In order to eliminate dependency on the initial parameters of the policy, we use a random policy to collect transitions into the replay buffer [32] for the first 10K time steps for SAC, and 4K time steps for PPO. We also pretrain the representations with the aforementioned random collected samples to stabilize inputs to each RL algorithm, as described in [19].

The network architectures, optimizers, and hyperparameters of SAC and PPO are the same as those used in their original papers, except that we use mini-batches of size 256 instead of 100. As for PPO, we perform K 2 gradient updates of θ aux for every K 2 steps of data sampling. The update interval K 2 is set differently for six MuJoCo tasks and can be found in Table 3.

E Visualization

To demonstrate that the representations learned by SPF effectively capture the structural information contained in infinite-step state sequences, we compare the true state sequences with the states recovered from the predicted DTFT via the inverse DTFT.

Specifically, we first generate a state sequence from the trained policy and select a goal state s t at a certain time step. Next, we choose a historical state s t -k located k steps past the goal state and select an action a t -k based on the trained policy π ( ·| s t -k ) as the inputs of our trained predictor. We then obtain the DTFT F t -k := F π ( s t -k , a t -k ) of state sequences starting from the state s t -k +1 . Next, we compute the kth element of the inverse DTFT of F t -k and obtain a recovered state ̂ s t , which represents that we predict the future goal state using the historical state located k steps past the goal state. By selecting a sequence of states over a specific time interval as the goal states and repeating the aforementioned procedures, we will obtain a state sequence recovered by k-step prediction. In Figure 5(b), 6(b), 7(b), 8(b), 9(b) and 10(b), we visualize the true state sequence (the blue line) and the recovered state sequences (the red lines) via k-step predictions for k = 1 , 2 , 3 , 4 , 5 . Note that the lighter red line corresponds to predictions made by historical states from a more distant time step. We conduct the visualization experiment on six MuJoCo tasks using the representations and predictors trained by SPF-SAC or SPF-PPO. Due to the large dimensionality of the states in Ant-v2 and Humanoid-v2, which contain many zero values, we have chosen to visualize only six dimensions of their states, respectively. The fine distinctions between the true state sequences and the recovered state sequences from our trained representations and predicted FT indicates that our representation effectively captures the inherent structures of future state sequences.

Furthermore, we provide a visualization that compares the true DTFT and the predicted DTFT in Figure 5(a), 6(a), 7(a), 8(a), 9(a) and 10(a). To accomplish this, we use our trained policies to interact with the environments and select the state sequences of the 200 last steps of an episode. The blue lines represent the true DTFT of these state sequences, while the orange line represents the predicted DTFT using the online encoder and predictor trained by our learned policies. It is evident that the true DTFT and the predicted DTFT exhibit significant differences. These results demonstrate the ability of SPF to effectively extract the underlying structural information in infinite-step state sequences without relying on high prediction accuracy.

F Cosine Similarity Results

We present a comparison table between the real discrete-time Fourier transform (DTFT) and the predicted DTFT. To obtain these results, we use our trained policies to interact with six MuJoCo environments and record the states and the actions for the 201 last steps of an episode, denoted as { s t } 200 t =0 and { a t } 200 t =0 , respectively. The real DTFT of the state sequence { s 1 , s 2 , . . . , s 200 } over a 200-step horizon is computed using the formulation of the discrete-time Fourier transform. For the predicted DTFT, we employ our trained encoder and prediction module, with the state s 0 and the action a 0 serving as inputs. The output of the prediction module represents the predicted DTFT of the state sequence { s 1 , s 2 , . . . , s 200 } . We compute the cosine similarity between A and B with the

formula -A · B ∥ A ∥∥ B ∥ , where values closer to -1 indicate greater similarity while the values closer to 1 indicate greater dissimilarity. The cosine similarity between the real DTFT and the predicted DTFT is listed in Table 4 below.

G Code

Walker2d's state 14

Codes for the proposed method are available at https://github.com/MIRALab-USTC/RL-SPF/ . Future states

0

50

100

Horizon

(a) True and predicted DTFT

true

1-step pred

2-step pred

3-step pred

4-step pred

5-step pred

(b) True and recovered state sequences

HalfCheetah's state0 FT module

true

HalfCheetah's state1 FT module

true

pred

tate2's fourier module

HalfCheetah's state2 FT module

true

HalfCheetah's state 0

HalfCheetah's state 1

HalfCheetah's state 2

jauno, s,gagesss

WN

Future states

sas as

Future states

Frequency w

Frequency w

NV/2

Frequency w

HalfCheetah's state3 FT module

HalfCheetah’s state4 FT module

HalfCheetah's state5 FT module

150

00

Horizon

150

200

Horzon

150

anpou

true

red

HalfCheetah's state 3

HalfCheetah's state 4

HalfCheetah's state 5

Uu

Future states

Future states

Frequency w

2n/3

Frequency w

2n/3

n/2

Frequency w

2n/3

HalfCheetah’s state6 FT module

HalfCheetah's state7 FT module

HalfCheetah's state8 FT module

Horizon

100

150

200

Horizon

100

150

200

Horizon

100

150

te6's fourer module

ate7sfouriermodule

state8's fourier module

二

HalfCheetah's state 6

HalfCheetah's state7

HalfCheetah's state 8

Future states

sas as

Future states

HalfCheetah's state9 FT module

Frequency w

2n/3

HalfCheetah's state10 FT module

Frequency w

HalfCheetah's state11FT module

n/2

Frequency w

2n/3

2n

Horizon

100

150

Horizon

100

150

200

Horizon

100

150

a(npow ja(μnoj s,6a]

re

tre

rier module

二

re

HalfCheetah's state 9

HalfCheetah's state 10

HalfCheetah's state 11

Future states

Future states

Future states

HalfCheetah's state12FT module

V2

Frequency w

2n/3

true

pred

HalfCheetah's state13 FT module

Frequency w

2h/3

true

pred

HalfCheetah's state14FT module

module

I/2

Frequency w

2n/3

true

pred

Future states

HalfCheetah's state 12

Horizon

100

150

Future states

HalfCheetah's state 13

Horizon

100

150

Future states

HalfCheetah's state 14

Horizon

100

150

S.7T

Frequency

2n/3

Frequency

2R/3

Frequency w

2n/3

150

200

50

Horizon

100

150

Horizon

100

150

e15's fourier module

HalfCheetah's state15 FT module

(ate16sfouriermodule

HalfCheetah's state16 FT module

true

HalfCheetah's state 15

HalfCheetah's state 16

pau

Future states

states

Future

Frequency w

2n/3

Frequency w

Horizon

15

Horizon

100

150

200

(a) True and predicted DTFT

0

50

Walker2d's state 14

100

Horizon

150

200

true

1-step pred

2-step pred

3-step pred

4-step pred

5-step pred

(b) True and recovered state sequences

ajnpou.

true

pred

ajnpou

Walker2d’s state1 FT module

true

pred

Walker2d's state2 FT module

true

Walker2d's state 0

Walker2d's state 1

Walker2d's state 2

auno, s,gaesss

MwMhM

MMW

MmM

Future states

Frequency w

2n/3

Frequency

Frequency w

Walker2d’s state3FTmodule

state4's fourier module

Walker2d’s state4FT module

state5's fourier module

Walker2d’s state5FTmodule

tre

Walker2d’s state 3

50

Horizon

150

Walker2d’s state 4

Horizon

100

Walker2d’s state 5

Horizon

100

150

state3’s fourier

Future states

Frequency w

2n

Frequency w

2n/3

Frequency w

2n/3

Walker2d's state6 FT module

Walker2d's state7 FT module

Walker2d's state8 FT module

Horizon

100

150

Horizon

100

150

Horizon

100

150

true

state7's fourier module

state8's fourier module

re

Walker2d’s state 6

Walker2d's state 7

Walker2d's state 8

Future states

Future states

Wwwwww

WWW

Walker2d's state9 FT module

Frequency w

state10's fourier module

Walker2d’'s state10 FT module

/2

Frequency w

2/3

state11's fourier module

n/2

Frequency w

2π/3

Walker2d’s state 9

Horizon

100

150

Walker2d's state 10

Horizon

100

150

Walker2d's state 11

Horizon

100

150

e9’s fourier

Future states

Future states

WV

ainpou

Walker2d’s state12 FT module

Frequency w

2/3

二

2n

state13’s fourier module

Walker2d's state13 FT module

n/2

Frequency w

2/3

state14's fourier module

Walker2d's state14 FT module

n/2

Frequency w

2n/3

Walker2d's state 12

Horizon

100

150

Walker2d's state 13

Horizon

100

150

Walker2d's state 14

Horizon

100

150

Future states

Frequency w

2n/3

Z

Frequency w

2折/3

n/Z

Frequency w

2n/3

Horizon

100

150

Horizon

100

Horizon

100

150

Walker2d's state15 FT module

Walker2d's state16 FT module

Walker2d's state 15

Walker2d's state 16

inpou

true

state16's fourier module

true

jauno

Wwrdha

WWW

Frequency w

2n/3

Frequency w

2n/3

Horizon

15

Horizon

Walker2d's state 14

100

Horizon

(a) True and predicted DTFT

true

1-step pred

2-step pred

3-step pred

4-step pred

5-step pred

(b) True and recovered state sequences

state200's fourier module

Humanoid's state200 FT module

二

true

pred

state201's fourier module

Humanoid's state201 FT module

true

pred

state202's fourier module

Humanoid's state202 FT module

true

pred

Humanoid's state 200

Humanoid's state 201

Humanoid's state 202

Future states

Future states

Future states

state203’s fourier module

Humanoid's state203 FT module

n/2

Frequency w

I

2n/3

pred

2n

state204's fourier module

Humanoid’s state204 FT module

0

n/2

Frequency w

2n/3

anuq

pred

2n

state205's fourier module

Humanoid's state205 FT module

n/2

Frequency w

2n/3

true

pred

2n

Humanoid's state 203

0S

Horizon

100

150

200

Humanoid's state 204

50

Horizon

100

150

0

Humanoid's state 205

50

Horizon

100

150

200

Future states

Future states

RA

Future states

L

n/2

Frequency w

2n/3

2n

0

n/2

Frequency w

2n/3

2nt

0

n/2

Frequency w

2n/3

2n

0

50

Horizon

100

150

200

6

50

Horizon

100

150

200

0

50

Horizon

100

150

200

Future states

Future states

0

50

150

200

Future states

0

0

0

50

50

50

Walker2d's state 14

100

Horizon

(a) True and predicted DTFT

true

1-step pred

2-step pred

3-step pred

4-step pred

5-step pred

(b) True and recovered state sequences

Hopper's state0 FT module

true

pred

state1’s fourier module

Hopper's state1 FT module

true

pred

state2’'s fourier module

Hopper's state2 FT module

true

prep

Hopper's state 0

Hopper's state 1

Hopper's state 2

Future states

emh

n/2

Frequency w

2n/3

2n

n/2

Frequency w

2/3

n/2

Frequency w

2n/3

2

Hopper's state3 FT module

Hopper's state4 FT module

Hopper's state5 FT module

Horizon

100

150

200

Horizon

100

150

Horizon

100

150

200

anpo jauno, s,aess

—true

pred

state4's fourier module

true

pred

state5's fourier module

true

pred

Hopper's state 3

Hopper's state 4

Hopper's state 5

Future states

vV

Hopper's state6 FT module

I/2

Frequency w

2n/3

2n

Hopper's state7 FT module

n/2

Frequency w

2n/3

Hopper's state8 FT module

Frequency w

2n/3

Horizon

100

150

200

Horizon

100

150

002

50

Horizon

100

150

state6’s fourier module

true

pred

state7’s fourier module

true

pred

state8’s fourier module

true

Hopper's state 6

Hopper's state 7

Hopper's state8

Future states

Hopper's state9 FT module

n/2

Frequency w

2n

state10's fourier module

Hopper's state10 FT module

n/2

Frequency w

I

2n/3

true

2nt

n/2

Frequency w

2n/3

2n

Hopper's state 9

Horizon

100

150

Hopper's state 10

50

Horizon

100

150

002

50

Horizon

100

150

pred

Future states

n/2

Frequency w

2n/3

n/2

Horizon

100

oSt

Horizon

100

Future states

100

Horizon

(a) True and predicted DTFT

true

1-step pred

2-step pred

3-step pred

4-step pred

5-step pred

(b) True and recovered state sequences

state15's fourier module

Ant's state15 FT module

an

pred

state16's fourier module

Ant's state16 FT module

pred

state17's fourier module

Ant's state17 FT module

true

pred

Ant's state 15

Ant's state 16

Ant's state 17

Future states

r

Future states

Future states

state18's fourier module

Ant'sstatel8FT module

Frequency w

2n/3

true

pred

2n

state19's fourier module

0

Ant's statel9 FT module

n/2

Frequency w

2h/3

true

pred

2n

state20's fourier module

Ant's state20 FT module

Frequency w

2n/3

true

pred

2n

50

Ant's state 18

Horizon

100

150

200

50

Ant's state 19

Horizon

100

150

0

50

Ant's state 20

Horizon

100

150

200

Future states

Future states

Future

n/2

Frequency w

2n/3

2n

0

n/2

Frequency w

2n/3

2nt

0

n/2

Frequency w

2n/3

2n

50

Horizon

100

150

200

50

Horizon

100

150

200

0

50

Horizon

100

150

200

Future states

100

Horizon

(a) True and predicted DTFT

true

1-step pred

2-step pred

3-step pred

4-step pred

5-step pred

(b) True and recovered state sequences

Swimmer's state0 FT module

Swimmer's state1 FT module

Swimmer's state2 FT module

true

pred

statel’s fourier module

true

pred

state2’s fourier module

tue

pred

Swimmer's state 0

Swimmer's state 1

Swimmer's state 2

Future states

Future states

Future states

Swimmer's state3 FT module

n/2

Frequency w

2n

0

Swimmer's state4 FT module

n/2

Frequency w

2n/3

2n

0

Swimmer's state5 FT module

n/2

Frequency w

2n/3

2n

50

Horizon

100

150

200

0

50

Horizon

100

150

200

0

50

Horizon

100

150

200

ajnpow aun s,es

state4’s fourier module

一

true

pred

state5's fourier module

—true

pred

Swimmer'sstate3

Swimmer's state 4

Swimmer'sstate5

W

Future states

Future states

Future states

state6’s fourier module

Swimmer's state6 FT module

n/2

Frequency w

2n/3

true

pred

2n

state7’s fourier module

0

Swimmer's state7 FT module

n/2

Frequency w

2n/3

true

pred

2n

n/2

Frequency w

2n/3

2n

0

Swimmer's state 6

50

Horizon

100

150

200

Swimmer's state 7

50

Horizon

100

150

0

50

Horizon

100

150

200

Future states

Future states

rwr

n/2

Frequency w

2n/3

2n

0

n/2

Frequency w

2/3

2n

Horizon

100

150

SD

Horizon

00t

150

002

150

150

150

200

200

200

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors in this paper investigated the structural information in state sequences for reinforcement learning, and proposed the prediction of Fourier transformation for the input sequences, which is further used an auxiliary task for learning policies. Some theoritical results are provided to show the existence of structual information. Experimental results on MuJoKo tasks demonstrated the promise of the proposed method.\n\nSTRENGTHS:\n1. The idea of performing prediction on the Fourier spectrum is interesting. This approach may better extract information hidden in the time domain, which constitutes the main contribution of this paper.\n\n2. From the experimental results, the proposed auxiliary task indeed helped learn better policies for RL agents, w.r.t. both SAC and PPO.\n\nWEAKNESSES:\n1. There seems to be a gap between the motivation and the implementation. The introduction and some theoretical results, e.g., Theorem 3, imply that the prediciton is on the Fourier domain; however, the algorithm (SPF) simply used a parameterized prediction module. Consequently, it's unclear to me whether the proposed algorithm can extract the information in the Fourier spectrum.\n\n2. It's unclear to me whether the proposed method can capture the long-term information. From Figure 2, it only considers two adjacent states as inputs.\n\n3. The significance in Theorem 1 may not be sufficient, as the upper bound can be loose. Note that the use of l1 norm for the difference between two distributions will make the term in Equation (3) much greater than 1. Furthermore, it's unclear whether the states distributions for two policies can vary that much, given the use of Rmax / (1 - \\gamma).\n\nQUESTIONS:\n1. Some notations in Figure 2 are not defined (e.g., $P$) and cannot be located from the definition of the prediction loss (e.g., $\\mathcal{F}$) between L261 and L262.\n\n2. What does $\\Gamma_{Re}$ do in the prediction loss, given that $\\mathcal{F}_{Re}$ is already real?\n\nLIMITATIONS:\nYes, the authors mentioned that the current version may not handle the image inputs.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-01"}

comments: SUMMARY:
Data-efficient RL exploits representation learning which learns representations by predicting long-term future states. However, inherent structural information in sequential state signals is ignored in existing methods. The authors propose SPF which exploits the frequency domain of state sequences. The proposed method performs an auxiliary self-supervision task to improve the efficiency of representation learning. This method outperforms several state-of-the-art algorithms in terms of both sample efficiency and performance.

STRENGTHS:
* Notations are clear and methods are clearly proposed.

WEAKNESSES:
* Results are not promising compared to SAC-OFE. In Humanoid, SAC-OFE outperforms SAC-SPF, and in HalfCheetah, Walker2d, Ant, Swimmer, SAC-OFE is comparable to SAC-SPF.
* I could not find critical superior points to use Fourier transform. As the authors pointed out in the conclusion, experiments on visual-based RL settings are required to show the benefits of the Fourier transform.
* The inherent structural information in a sequential state is not clearly discussed in the paper.


**Minors**
* Typo in Figure2, There is no predictor F in the figure.

QUESTIONS:
* What is inherent structural information we can get from Fourier-transform? 

* The main concern is that results do not outperform the baseline. Also, in 6.3, the results of all the baselines and SPF are comparable.

====================================================================================================


The authors successfully handle my concerns. I raise my score to borderline accept.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 1
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nUnlike many prior model-based RL methods that predict one future state at a time, the authors present a Fourier-transform based approach that predicts future states over a fixed interval. They prove that a large difference in policy performance will cause a significant difference between the experienced states of a good and bad policy. This conclusion is important for utilizing frequency-space analysis of the states.\n\nTheir approach uses a self-supervised learning objective where they attempt to fit an online encoder given the current states, and a target encoder given the next states. Note that both the online and target encoder, given a single state, will predict frequencies for the rest of the trajectory. This is in contrast to prior methods that will just compute the next state. The TD-like error between predictions is then used to train the online encoder. I'm not sure I fully understand it, but it seems quite elegant in that their approach closely resembles the one-step TD error. As far as I can tell, they are not actually using the predicted state sequences for planning, as in model-based RL. Rather, they are learning the features necessary to predict future state sequences. These features are useful for a downstream policy.\n\nThey evaluate their method on the standard MuJoCo/Gym benchmarks which outperforms baselines. They compare their method using SAC/PPO to SAC/PPO using OFE or no prediction at all. They continue to ablate nearly all portions of their approach.\n\nSTRENGTHS:\n- The paper is very well written\n- Their approach is novel and elegant\n- Their experiments, specifically the ablation, is well-done\n- They show their approach outperforms baselines\n\nWEAKNESSES:\n- They only compared to one other feature-learning algorithm. It would be beneficial to have additional baselines.\n- Unless I'm misunderstanding, it seems like they've purposely limited the applications of their approach to fully observable, model-free RL\n    - They already predict future states, they might as do some online planning\n    - This could be applied to partially observable domains to learn Markov states\n- They do not list any limitations of their approach\n\nQUESTIONS:\n- \"Therefore, state sequences maximally preserve the influence of the transition intrinsic to the environment and the effect of actions generated from the current policy\"\n    - I'd be careful saying this. In the case of a deterministic policy, yes. But if you add stochasticity (e.g. epsilon greedy or policy-gradient sampling) I'm not sure you can claim you can predict a future state sequence. You can perhaps predict the distribution of states.\n- \"it is widely accepted that the frequency domain shows the regularity properties of the time-series data\"\n    - There are citations for this but I find it a bit hard to believe. For cyclical processes this makes sense, but there is no guarantee, for example, that you will see a specific feature every k timesteps. Could you add an intuitive explanation of why this should work for latent state sequences?\n    - This is explained later in the paper, but perhaps it makes sense to say you show/prove this in the introduction\n- Eq. 5: Why do you discount the future state using gamma? I understand dicounting the reward/return, but not effectively \"decaying\" the future state. Could you briefly explain the reasoning here? Is it to weight nearby states more heavily?\n\nLIMITATIONS:\n- The authors do not list any limitations", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper suggests a representation learning method that leverages the existing periodic structures in the state sequences of a policy. Specifically, they propose using the features obtained by the Fourier transform of the state sequence. To practically extract such features, they employ a Bellman equation-like recursive form of the Fourier transform (FT), which enables efficient online learning. The authors demonstrate that their representation learning method helps improve the performances of SAC and PPO in Gym MuJoCo environments.\n\nSTRENGTHS:\n- The idea of using frequency features in sequential decision-making problems seems novel and sensible, given that real-world tasks often inherently have periodic (sub-)structures in their states (e.g., robotic locomotion).\n- The recursive objective in Equation (7), which enables learning Fourier features in practice without the need to deal with the entire state sequences, is intriguing and potentially inspirational for future research.\n- The paper includes an informative ablation study.\n\nWEAKNESSES:\n- The Fourier features inherently depend on the policy and task, making them less reusable for multiple tasks, especially compared to previous self-supervised representation learning methods.\n- The theorems in the paper do not necessarily explain why Fourier features could be potentially useful for *representations*. Theorem 3 states a (very crude) bound for the performance difference between two policies expressed in the frequency domain. However, it is unclear as to why the features extracted in this manner are useful or sufficient for modeling policies/value functions as well, which is the way SPF utilizes the learned features. I would have expected a theorem that says the sub-optimality of using the learned Fourier representation decreases as the state sequence shows stronger periodicity.\n\nQUESTIONS:\n- Can the authors elaborate on why Fourier features are useful for representations? For example, let's say a state sequence from a policy forms a perfect sinusoidal curve so that its Fourier transform collapses to a single point. In such a case, its Fourier representation (which is a constant) may not contain sufficient information for control (i.e., for producing optimal actions to maintain that sinusoidal curve).\n- Could SPF be harmful in tasks that do not have periodicity? Have the authors tested SPF on such environments/tasks?\n\nLIMITATIONS:\nThe authors briefly mention the limitations of SPF in Section 7. However, it would be also important to include a discussion on the second question above in the limitation section as well.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}