SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We are further interested in bounding the performance difference between two policies by their state sequences in the frequency domain. Benefiting from the properties of the discrete-time Fourier transform (DTFT), we can constrain the performance difference using the Fourier transform over the interval  $[0, 2\pi]$ , instead of using the distribution functions of the state sequences in unbounded space.
Theorem 2. Suppose that  $\mathcal{S} \subset \mathbb{R}^D$  the reward function  $R(s, a, s') = R(s)$  is an  $n$ th-degree polynomial function with respect to  $s \in \mathcal{S}$ , then for any two policies  $\pi_1$  and  $\pi_2$ , their performance difference can be bounded as follows:
$$
\left| J \left(\pi_ {1}\right) - J \left(\pi_ {2}\right) \right| \leq \frac {\sqrt {D}}{1 - \gamma} \cdot \sum_ {k = 1} ^ {n} \frac {\left\| R ^ {(k)} (0) \right\| _ {D}}{k !} \cdot \max  _ {1 \leq i \leq D} \sup  _ {\omega_ {i} \in [ 0, 2 \pi ]} \left| F _ {\pi_ {1}} ^ {(k)} \left(\omega_ {i}\right) - F _ {\pi_ {2}} ^ {(k)} \left(\omega_ {i}\right) \right|, \tag {4}
$$
Proof. For sake of simplicity, we define  $p_t(s|\pi_i) = P(\mathbf{s_t} = s|\pi_i,\mathcal{M})$  for  $i = 1,2$ . We denote  $\varepsilon_{t}$  as
$$
\varepsilon_ {t} = \int_ {\mathcal {S}} R (s) \left[ p _ {t} \left(s \mid \pi_ {1}\right) - p _ {t} \left(s \mid \pi_ {2}\right)\right) \mathrm {d} s. \tag {5}
$$
Based on the Taylor series expansion, we can rewrite the reward function as  $R(s) = \sum_{k=0}^{n} \frac{R^{(k)}(0)^{\top}}{k!} s^k$ , then for any integer  $k \in [1, n]$ , we have
$$
\begin{array}{l} \left| \varepsilon_ {t} \right| \leq \sum_ {k = 0} ^ {n} \frac {\left\| R ^ {(k)} (0) \right\| _ {D}}{k !} \cdot \left\| \int_ {\mathcal {S}} \left[ s ^ {k} p _ {t} (s | \pi_ {1}) - s ^ {k} p _ {t} (s | \pi_ {2}) \right] d s \right\| _ {D} \\ = \sum_ {k = 0} ^ {n} \frac {\left\| R ^ {(k)} (0) \right\| _ {D}}{k !} \left\| _ {s \sim p _ {t} (\cdot | \pi_ {1})} E [ s ^ {k} ] - E _ {s \sim p _ {t} (\cdot | \pi_ {2})} [ s ^ {k} ] \right\| _ {D}. \tag {6} \\ \end{array}
$$
Since the inverse DTFT of  $F_{\pi}^{(k)}(\omega)$  is the original time series  $\mathbf{S}^{(k)}$ , we have
$$
E _ {s _ {i} \sim p _ {t} (\cdot | \pi)} \left[ s _ {i} ^ {k} \right] = \frac {1}{2 \pi} \int_ {0} ^ {2 \pi} F _ {\pi} ^ {(k)} \left(\omega_ {i}\right) e ^ {j \omega_ {i} t} d \omega_ {i}, \quad \forall i = 1, 2, \dots , D. \tag {7}
$$
$$
\begin{array}{l} \left| \frac {E}{s _ {i} \sim p _ {t} (\cdot | \pi_ {1})} [ s _ {i} ^ {k} ] - \frac {E}{s _ {i} \sim p _ {t} (\cdot | \pi_ {2})} [ s _ {i} ^ {k} ] \right| \leq \frac {1}{2 \pi} \int_ {0} ^ {2 \pi} \left| F _ {\pi_ {1}} ^ {(k)} (\omega_ {i}) - F _ {\pi_ {2}} ^ {(k)} (\omega_ {i}) \right| \cdot \left| e ^ {j \omega_ {i} t} \right| d \omega_ {i} \\ \leq \sup  _ {\omega_ {i} \in [ 0, 2 \pi ]} \left| F _ {\pi_ {1}} ^ {(k)} (\omega_ {i}) - F _ {\pi_ {2}} ^ {(k)} (\omega_ {i}) \right|. \tag {8} \\ \end{array}
$$
Substituting (8) into (6), then we obtain
$$
\left| \varepsilon_ {t} \right| \leq \sqrt {D} \cdot \sum_ {k = 1} ^ {n} \frac {\left\| R ^ {(k)} (0) \right\| _ {D}}{k !} \cdot \max  _ {1 \leq i \leq D} \sup  _ {\omega_ {i} \in [ 0, 2 \pi ]} \left| F _ {\pi_ {1}} ^ {(k)} (\omega_ {i}) - F _ {\pi_ {2}} ^ {(k)} (\omega_ {i}) \right|.
$$
For the sake of DTFT, the upper bound of  $\epsilon_{t}$  is independent of  $t$ , then we could derive the performance difference bound as follows.
$$
\begin{array}{l} \left| J \left(\pi_ {1}\right) - J \left(\pi_ {2}\right) \right| \leq \sum_ {t = 0} ^ {\infty} \gamma^ {t} \cdot \left| \varepsilon_ {t} \right| \\ \leq \frac {1}{1 - \gamma} \cdot \sqrt {D} \cdot \sum_ {k = 1} ^ {n} \frac {\left\| R ^ {(k)} (0) \right\| _ {D}}{k !} \cdot \max _ {1 \leq i \leq D} \sup _ {\omega_ {i} \in [ 0, 2 \pi ]} \left| F _ {\pi_ {1}} ^ {(k)} (\omega_ {i}) - F _ {\pi_ {2}} ^ {(k)} (\omega_ {i}) \right|, \\ \end{array}
$$
and so we immediately achieve the desired bound in (4).
A.2 Proof of the Asymptotic Periodicity of States in MDP
This section focuses on analyzing the asymptotic behavior of the state sequences generated from an MDP. We begin by discussing the limiting process of MDP with a finite state space  $S$ . Let  $P$  be the transition probability matrix and let  $\mu_{i}$  be the probability distribution of the states at time  $t_i$ . Then we have  $\mu_{i+1} = P\mu_{i}$  for any  $i \geq 0$ . If the sequence  $\{\mu_{i}\}_{i=0}^{\infty}$  splits into  $d$  subsequences with  $d$  cyclic limits  $\{\mu_{\infty}^{r}\}_{r=0}^{d-1}$  that follow the cycle:
$$
\mu_ {\infty} ^ {0} \rightarrow \mu_ {\infty} ^ {1} \rightarrow \dots \rightarrow \mu_ {\infty} ^ {d - 1} \rightarrow \mu_ {\infty} ^ {0},
$$
then we say that the states of the MDP exhibit asymptotic periodicity. Such cyclic asymptotic behavior implies that the limiting distribution of the states eventually repeats in a specific period after a certain number of steps.
We begin by providing some essential definitions in the field of stochastic processes [30], which will be utilized in the following proof. Let  $P$  be a transition probability matrix corresponding to  $n$  states ( $n \geq 1$ ). Two states  $i$  and  $j$  are said to intercommunicate if there exist paths from  $i$  to  $j$  as well as from  $j$  to  $i$ . The matrix  $P$  is called irreducible if any two states intercommunicate. A set of states is called irreducible if any two states in the set intercommunicate. Moreover, a state  $i$  is called recurrent if the probability of eventual return to  $i$ , having started from  $i$ , is 1. If this probability is strictly less than 1, the state  $i$  is called transient.
Note that if the whole state space  $S$  is irreducible, then its transition matrix  $P$  is also irreducible. The following lemma demonstrates that if the state space is irreducible, then its asymptotical periodicity is determined by the eigenvalues with modulus 1 of its transition matrix.
Lemma 1. Suppose that the state space  $\mathcal{S}$  is finite with a transition probability matrix  $P \in \mathbb{R}^{|S| \times |S|}$ . If  $P$  is an irreducible matrix with  $d$  eigenvalues of modulus 1, then for any initial distribution  $\mu_0$ ,  $P^n \mu_0$  is asymptotically periodic with a period of  $d$  when  $d > 1$  and asymptotically aperiodic when  $d = 1$ .
Proof. According to the Perron-Frobenius theorem for irreducible non-negative matrices, all eigenvalues of  $P$  of modulus 1 are exactly the  $d$  complex roots of the equation  $\lambda^d - 1 = 0$ . They can be formulated as  $\lambda_0 = 1, \lambda_1 = \xi^1, \ldots, \lambda_{d-1} = \xi^{d-1}$ , where  $\xi = e^{\frac{2\pi j}{d}}$ . Each of them is a simple root of the characteristic polynomial of the matrix  $P$ . Since  $P$  is a transition probability matrix, the remaining eigenvalues  $\lambda_d, \ldots, \lambda_s$  satisfy  $|\lambda_r| < 1$ . Therefore, the Jordan matrix of  $P$  has the form
$$
J = \left[ \begin{array}{c c c c c c c} \lambda_ {0} & & & & & & \\ & \lambda_ {1} & & & & & \\ & & \ddots & & & & \\ & & & \lambda_ {d - 1} & & & \\ & & & & J _ {d} & & \\ & & & & & \ddots & \\ & & & & & & J _ {s} \end{array} \right], \text {w h e r e} J _ {k} = \left[ \begin{array}{c c c c c c} \lambda_ {k} & 1 & & & \\ & \lambda_ {k} & 1 & & \\ & & \ddots & \ddots & \\ & & & \lambda_ {k} & 1 \\ & & & & \lambda_ {k} \end{array} \right].
$$
We refer to  $J_{k}$  as Jordan cells.
Let  $|S| = D$ , we can rewrite  $P$  in its Jordan canonical form  $P = XJX^{-1}$  where
$$
X = [ \vec {x} _ {0}, \vec {x} _ {1}, \dots , \vec {x} _ {D - 1} ].
$$
Note that for  $k < d$ ,  $x_{k}$  is the eigenvector corresponding to  $\lambda_{k}$ . Since the column vectors of  $X$  are linearly dependent, there exist  $\vec{c} = [c_0, c_1, \ldots, c_{D-1}]$  not all zero, such that  $\mu_{0} = \sum_{k=0}^{D-1} c_{k} \vec{x}_{k} = X \vec{c}$ . Thus, we have
$$
P ^ {n} \mu_ {0} = \sum_ {k = 0} ^ {d - 1} c _ {k} \lambda_ {k} ^ {n} \vec {x} _ {k} + \sum_ {k = d} ^ {D - 1} c _ {k} P ^ {n} \vec {x} _ {k}. \tag {9}
$$
For any Jordan cell  $J_{k}$ , let  $\alpha_{k}$  be the multiplicity of  $\lambda_{k}$ , then
$$
J _ {k} ^ {n} = \left[ \begin{array}{c c c c c} \lambda_ {k} & 1 & & & \\ & \lambda_ {k} & 1 & & \\ & & \ddots & \ddots & \\ & & & \lambda_ {k} & 1 \\ & & & & \lambda_ {k} \end{array} \right] _ {\alpha_ {k} \times \alpha_ {k}} ^ {n} = \left[ \begin{array}{c c c c c} \lambda_ {k} ^ {n} & C _ {n} ^ {n - 1} \lambda_ {k} ^ {n - 1} & \dots & C _ {n} ^ {n - \alpha_ {k} + 1} \lambda_ {k} ^ {n - \alpha_ {k} + 1} \\ & \lambda_ {k} ^ {n} & \dots & C _ {n} ^ {n - \alpha_ {k} + 2} \lambda_ {k} ^ {n - \alpha_ {k} + 2} \\ & & \ddots & \vdots \\ & & & \lambda_ {k} ^ {n} \end{array} \right].
$$
Since  $\alpha_{k}$  is fixed for matrix  $P$ , we have  $\lim_{n\to \infty}J_k^n = 0$  for each  $k = d,\ldots ,D - 1$ . Then the limiting vector of (9), denoted by  $P^{\infty}\mu_0$ , satisfies:
$$
P ^ {\infty} \mu_ {0} = \lim _ {n \to \infty} X J ^ {n} X ^ {- 1} X \vec {c} = \lim _ {n \to \infty} \sum_ {k = 0} ^ {d - 1} c _ {k} \lambda_ {k} ^ {n} \vec {x} _ {k} = \lim _ {n \to \infty} \mu^ {(n)},
$$
where we denote  $\mu^{(n)} = \sum_{k=0}^{d-1} c_k \left(e^j \frac{2\pi k}{d}\right)^n \vec{x}_k$ . Let  $r = n \pmod{d}$ , then we have
$$
\mu^ {(n)} = \mu^ {(r)} = \sum_ {k = 0} ^ {d - 1} c _ {k} \left(\xi^ {k}\right) ^ {r} \vec {x} _ {k}, \quad \forall n \geq 1.
$$
Therefore, the probability sequence  $\{P^n\mu_0\}_{n\geq 1}$  will split into  $d$  converging subsequences and has  $d$  cyclic limiting probability distributions when  $n\to \infty$ , denoted as
$$
\mu_ {\infty} ^ {r} = \sum_ {k = 0} ^ {d - 1} c _ {k} \left(\xi^ {k}\right) ^ {r} \vec {x} _ {k}, \quad r = 0, 1, \dots , d - 1.
$$
Thus,  $P^n \mu_0$  is asymptotically periodic with period  $d$  if  $d > 1$  and asymptotically aperiodic if  $d = 1$ .
We now consider a more general state space that may not necessarily be irreducible. According to the Decomposition theorem of the Markov chain [30], the finite state space  $S$  can be partitioned uniquely as a set of transient states and one or several irreducible closed sets of recurrent states. According to [31], after performing an appropriate permutation of rows and columns, we can rewrite the transition probability matrix  $P$  in its canonical form:
$$
P = \left[ \begin{array}{c c c c c} R _ {1} & \mathbf {0} & \dots & \mathbf {0} & \mathbf {0} \\ \mathbf {0} & R _ {2} & \dots & \mathbf {0} & \mathbf {0} \\ \vdots & \vdots & \ddots & \vdots & \vdots \\ \mathbf {0} & \mathbf {0} & \dots & R _ {\alpha} & \mathbf {0} \\ \hline T _ {1} & T _ {2} & \dots & T _ {\alpha} & Q \end{array} \right],
$$
where  $R_{1}, \ldots, R_{\alpha}$  represent the probability submatrices corresponding to the recurrent classes,  $Q$  represents the probability submatrix corresponding to the transient states, and  $T_{1}, \ldots, T_{\alpha}$  represent the probability submatrices corresponding to the transitions between transient and recurrent classes  $R_{1}, \ldots, R_{\alpha}$  respectively.
Theorem 3. Suppose that the state space  $\mathcal{S}$  is finite with a transition probability matrix  $P \in \mathbb{R}^{|S| \times |S|}$  and  $\mathcal{S}$  has  $\alpha$  recurrent classes. Let  $R_1, R_2, \ldots, R_\alpha$  be the probability submatrices corresponding to the recurrent classes and let  $d_1, d_2, \ldots, d_\alpha$  be the number of the eigenvalues of modulus 1 that the submatrices  $R_1, R_2, \ldots, R_\alpha$  has. Then for any initial distribution  $\mu_0$ ,  $P^n\mu_0$  is asymptotically periodic with period  $d = \operatorname{lcm}(d_1, d_2, \ldots, d_\alpha)$  when  $d > 1$  and asymptotically aperiodic when  $d = 1$ .
Proof. Since  $P$  is a block upper-triangular, it can be shown that the eigenvalues of  $P$  are equal to the union of the eigenvalues of the diagonal blocks  $R_{1},\ldots ,R_{\alpha},Q$ . Note that the nth-power of  $P$  satisfies the following expression:
$$
P ^ {n} = \left[ \begin{array}{c c c c c} R _ {1} ^ {n} & \mathbf {0} & \dots & \mathbf {0} & \mathbf {0} \\ \mathbf {0} & R _ {2} ^ {n} & \dots & \mathbf {0} & \mathbf {0} \\ \vdots & \vdots & \ddots & \vdots & \vdots \\ \mathbf {0} & \mathbf {0} & \dots & R _ {\alpha} ^ {n} & \mathbf {0} \\ \hline T _ {1} ^ {(n)} & T _ {2} ^ {(n)} & \dots & T _ {\alpha} ^ {(n)} & Q ^ {n} \end{array} \right],
$$
where  $T_r^{(n)}$  is related to the  $(n - 1)$ -th or the lower power of  $R_r$  and  $Q$ . From Theorem 4.3 of [31], we obtain that  $\lim_{n \to \infty} Q^n = 0$ , which implies that all eigenvalues of  $Q$  have modulus less than 1.
On the other hand, note that the sum of every row in matrix  $R_{r}$  is equal to 1, which means  $\lambda = 1$  is an eigenvalue of  $R_{r}$  and all eigenvalues of  $R_{r}$  satisfy  $|\lambda| \leq 1$ . Thus, the spectral radius of  $P$  is equal to 1.
Note that the proof of Lemma 1 implies that the asymptotic periodicity of  $P^n\mu_0$  depends on the eigenvalues of  $P$  that have modulus 1. Since  $R_{r}$  is non-negative irreducible with spectral radius 1, based on the Perron-Frobenius theorem used in Lemma 1, we can express the eigenvalues of  $R_{r}$  in modulus 1 as:
$$
\lambda_ {r, k} = e ^ {j \frac {2 \pi k}{d _ {r}}}, \quad , k = 0, 1, \ldots , d _ {r} - 1.
$$
Based on the above discussion, it is easy to check that  $\bigcup_{r=1}^{\alpha}\left\{\lambda_{r,0}, \ldots, \lambda_{r,d_r-1}\right\}$  is the set of all eigenvalues of modulus 1 of  $P$ . Rewrite  $P$  in its Jordan canonical form  $P = XJX^{-1}$ , where
$$
J = \left[ \begin{array}{c c c c c c c c c} \lambda_ {1, 0} & & & & & & & \\ & \ddots & & & & & & \\ & & \lambda_ {1, d _ {1} - 1} & & & & & \\ & & & \lambda_ {2, 0} & & & & \\ & & & & \ddots & & \\ & & & & & \lambda_ {\alpha , d _ {\alpha} - 1} & & \\ & & & & & & J _ {d _ {1} + \dots + d _ {\alpha}} & & \\ & & & & & & & \ddots \\ & & & & & & & & J _ {s} \end{array} \right]
$$
and  $X = [\vec{x}_0,\vec{x}_1,\dots ,\vec{x}_{D - 1}]$  is an invertible matrix. Similar to the proof in Lemma 1, we get
$$
P ^ {\infty} \mu_ {0} = \lim _ {n \to \infty} \sum_ {r = 1} ^ {\alpha} \sum_ {k = 0} ^ {d _ {r} - 1} c _ {k} (e ^ {j \frac {2 \pi k}{d _ {r}}}) ^ {n} \vec {x} _ {k} := \lim _ {n \to \infty} \mu^ {(n)}.
$$
Let  $d = \operatorname{lcm}(d_1, d_2, \ldots, d_\alpha)$  and  $r = n$  (mod  $d$ ), then we have
$$
\mu^ {(n)} = \mu^ {(r)}, \quad \forall n \geq 1.
$$
$$
\mu_ {\infty} ^ {r} = \sum_ {r = 1} ^ {\alpha} \sum_ {k = 0} ^ {d _ {r} - 1} c _ {k} e ^ {j \frac {2 \pi k r}{d _ {r}}} \vec {x} _ {k}, \quad r = 0, 1, \dots , d - 1.
$$
Thus,  $P^n \mu_0$  is asymptotically periodic with period  $d$  if  $d > 1$  and asymptotically aperiodic if  $d = 1$ . This completes the proof.
A.3 Proof of the Convergence of Our Auxiliary Loss
In this section, we provide a detailed derivation of the learning objective of SPF. As the DTFT of discrete-time state sequences is a continuous function that is difficult to compute, we practically sample the DTFT at  $L$  equally-spaced points.
$$
\left[ \mathcal {F} \widetilde {s} _ {t} \right] _ {k} = \sum_ {n = 0} ^ {+ \infty} \left[ \widetilde {s} _ {t} \right] _ {n} e ^ {- j \frac {2 \pi k}{L} n}, \quad k = 0, 1, \dots , L - 1. \tag {10}
$$
As a result, the prediction target takes the form of a matrix with dimensions of  $L * D$ , where  $D$  denotes the dimension of the state space. The auxiliary task is designed to encourage the representation to predict the Fourier transform of the state sequences using the current state-action pair as input. Specifically, we define the prediction target  $F_{\pi,p}(s_t, a_t)$  as follows:
$$
F _ {\pi , p} \left(s _ {t}, a _ {t}\right) = \mathcal {F} \widetilde {s} \left(s _ {t}, a _ {t}\right) = \left\{\sum_ {n = 0} ^ {+ \infty} \left[ \widetilde {s} \left(s _ {t}, a _ {t}\right) \right] _ {n} e ^ {- j \frac {2 \pi k}{L} n} \right\} _ {k = 0} ^ {L - 1}, \tag {11}
$$
For simplicity of notation, we substitute  $F(s_{t},a_{t})$  for  $F_{\pi ,p}(s_t,a_t)$  in the following. We can derive that the DTFT functions at successive time steps are related to each other in a recursive form:
$$
\begin{array}{l} [ F (s _ {t}, a _ {t}) ] _ {k} = \sum_ {n = 0} ^ {+ \infty} \gamma^ {n} \cdot e ^ {- j \frac {2 \pi k}{L} n} \cdot E _ {\pi , p} [ s _ {t + n + 1} | s _ {t} = s, a _ {t} = a ] \\ = E _ {p} \left[ s _ {t + 1} \mid s _ {t} = s, a _ {t} = a \right] + \gamma \cdot e ^ {- j \frac {2 \pi k}{L}}. \\ E _ {s _ {t + 1} \sim p, a _ {t + 1} \sim \pi} \left[ \sum_ {n = 0} ^ {+ \infty} \gamma^ {n} \cdot e ^ {- j \frac {2 \pi k}{L} n} \cdot E _ {p} \left[ s _ {t + n + 2} \mid s _ {t + 1}, a _ {t + 1} \right] \right] \\ = [ \widetilde {s} _ {t} ] _ {0} + \gamma \cdot e ^ {- j \frac {2 \pi k}{L}} \cdot E _ {\pi , p} [ [ F (s _ {t + 1}, a _ {t + 1}) ] _ {k} ], \forall k = 0, 1, \ldots L - 1. \\ \end{array}
$$
We can further express the above equation as a matrix-form recursive formula as follows:
$$
F \left(s _ {t}, a _ {t}\right) = \widetilde {\boldsymbol {S}} _ {t} + \Gamma E _ {\pi , p} \left[ F \left(s _ {t + 1}, a _ {t + 1}\right) \right], \tag {12}
$$
$$
\widetilde {\boldsymbol {S}} _ {t} = \left[ \left[ \widetilde {\boldsymbol {s}} _ {t} \right] _ {0}, \dots , \left[ \widetilde {\boldsymbol {s}} _ {t} \right] _ {0} \right] ^ {\intercal} \in \mathbb {R} ^ {L \times D},
$$
$$
\Gamma = \gamma \left[ \begin{array}{c c c c c} 1 & & & & \\ & e ^ {- j \frac {2 \pi}{L}} & & & \\ & & e ^ {- j \frac {4 \pi}{L}} & & \\ & & & \ddots & \\ & & & & e ^ {- j \frac {(L - 1) \pi}{L}} \end{array} \right].
$$
Similar to the TD-learning of value functions, we can prove that the above recursive relationship (12) can be reformulated as a contraction mapping  $\mathcal{T}$ . Due to the properties of contraction mappings, we can iteratively apply the operator  $\mathcal{T}$  to compute the target DTFT function until convergence in tabular settings.
Theorem 4. Let  $\mathcal{F}$  denote the set of all functions  $F:\mathcal{S}\times \mathcal{A}\to \mathbb{C}^{L*D}$  and define the norm on  $\mathcal{F}$  as
$$
\| F\|_{\mathcal{F}}:= \sup_{\substack{s\in \mathcal{S}\\ a\in \mathcal{A}}}\max_{0\leq k <   L}\left\| \left[F(s,a)\right]_{k}\right\|_{D},
$$
$$
\mathcal {T} F \left(s _ {t}, a _ {t}\right) = \widetilde {\boldsymbol {S}} _ {t} + \Gamma E _ {\pi , P} \left[ F \left(s _ {t + 1}, a _ {t + 1}\right) \right] \tag {13}
$$
is a contraction mapping, where  $\widetilde{S}_t$  and  $\Gamma$  are defined as above.
Proof. For any  $F_{1}, F_{2} \in \mathcal{F}$ , we have
$$
\begin{array}{l} \| \mathcal{T}F_{1} - \mathcal{T}F_{2}\|_{\mathcal{F}} = \sup_{\substack{s\in \mathcal{S}\\ a\in \mathcal{A}}}\max_{0\leq k <   L}\bigg\|s + \gamma e^{-j\frac{2\pi k}{L}}E_{s^{\prime}\sim P(\cdot |s,a)}\Big[ \begin{array}{c}\left[F_{1}(s^{\prime},a^{\prime})\right]_{k}|s,a\\ a^{\prime}\sim \pi (\cdot |s^{\prime}) \end{array} \Bigg] \\ - s - \gamma e^{-j\frac{2\pi k}{L}}_{\substack{s^{\prime}\sim P(\cdot |s,a)\\ a^{\prime}\sim \pi (\cdot |s^{\prime})}}\left[ \left[F_{2}(s^{\prime},a^{\prime})\right]_{k}|s,a\right]\bigg|\bigg|_{D} \\ \leq \gamma \cdot \max_{0\leq k <   L}\sup_{\substack{s\in \mathcal{S}\\ a\in \mathcal{A}}}\bigg\|E_{\substack{s^{\prime}\sim P(\cdot |s,a)\\ a^{\prime}\sim \pi (\cdot |s^{\prime})}}\bigg[\left[F_{1}(s^{\prime},a^{\prime})\right]_{k} - \left[F_{2}(s^{\prime},a^{\prime})\right]_{k}|s,a\bigg]{\|}_{D} \\ \leq \gamma \cdot \max_{0\leq k <   L}\sup_{\substack{s^{\prime}\in \mathcal{S}\\ a^{\prime}\in \mathcal{A}}}\bigl\|\big[F_{1}(s^{\prime},a^{\prime}) - F_{2}(s^{\prime},a^{\prime})\bigr ]_{k}\bigr\|_{D} \\ = \gamma \cdot \| F _ {1} - F _ {2} \| _ {\mathcal {F}}. \\ \end{array}
$$
Note that  $\gamma \in [0,1)$ , which implies that  $\mathcal{T}$  is a contraction mapping.
B Pseudo-code of SPF
The training procedure of SPF is shown in the pseudo-code as follows:
Algorithm 1 State Sequences Prediction via Fourier Transform (SPF)
Denote parameters of the online encoder  $(\phi_s,\phi_{s,a})$  , predictor  $\mathcal{F}$  , and projection  $\psi$  as  $\theta_{\mathrm{aux}}$    
Denote parameters of the target encoder  $(\widehat{\phi}_s,\widehat{\phi}_{s,a})$  , predictor  $\widehat{\mathcal{F}}$  , and projection  $\widehat{\psi}$  as  $\widehat{\theta}_{\mathrm{aux}}$    
Denote parameters of actor model  $\pi$  and critic model  $Q$  for RL agents as  $\theta_{\mathrm{RL}}$    
Denote the smoothing coefficient and update interval for target network updates as  $\tau$  and  $K$  Initialize replay buffer  $\mathcal{D}$  and parameters  $\theta_{\mathrm{aux}},\theta_{\mathrm{RL}}$    
for each environment step  $t$  do   
 $a_{t}\sim \pi (\cdot |\phi_{s}(s_{t}))$ $s_{t + 1},r_{t + 1}\sim p(\cdot |s_t,a_t)$ $\mathcal{D}\gets \mathcal{D}\cup (s_t,a_t,s_{t + 1},r_{t + 1})$    
sample a minibatch of  $\{(s_t,a_t,s_{t + 1},r_{t + 1})\}$  from  $\mathcal{D}$ $\theta_{\mathrm{aux}}\gets \theta_{\mathrm{aux}} - \alpha_{\mathrm{aux}}\nabla_{\theta_{\mathrm{aux}}}L_{\mathrm{pred}}(\theta_{\mathrm{aux}},\widehat{\theta}_{\mathrm{aux}})$    
resampling a minibatch of  $\{(s_t,a_t,s_{t + 1},r_{t + 1})\}$  from  $\mathcal{D}$ $\overline{s_t}\gets \phi_s(s_t)$ $z_{s_t,a_t}\gets \phi_{s,a}(\phi_s(s_t),a_t)$    
update the RL agent parameters  $\theta_{\mathrm{RL}}$  with the representations  $\overline{s_t}$ $z_{s_t,a_t}$    
update parameters of target networks with  $\widehat{\theta}_{\mathrm{aux}}\gets \tau \theta_{\mathrm{aux}} + (1 - \tau)\widehat{\theta}_{\mathrm{aux}}$  every  $K$  steps   
end for
C Network Details
The encoders  $\phi_s$  and  $\phi_{s,a}$  share the same architecture. Each layer of the encoders uses MLP-DenseNet [19], a slightly modified version of DenseNet. For each MuJoCo task, the incremental number of hidden units per layer is selected from  $\{30,40\}$ , while the number of layers is selected from  $\{6,8\}$  (see Table 2). Both the predictor  $\mathcal{F}$  and the projection  $\psi$  apply a 2-layer MLP. We divide the last layer of the predictor into two heads as the real part  $\mathcal{F}_{\mathrm{Re}}$  and the imaginary part  $\mathcal{F}_{\mathrm{Im}}$ , respectively, since the prediction target of our auxiliary task is complex-valued. With respect to the projection module, we add an additional 2-layer MLP (referred to as Projection2) after the original online projection to perform a dimension-invariant nonlinear transformation on the predicted DTFT that has been projected to a lower-dimensional space. We do not apply this nonlinear operation to the target projection. This additional step is carried out to prevent the projection from collapsing to a constant value in the case where the online and target projections share the same architecture.
In Fourier analysis, the low-frequency components of the DTFT contain information about the long-term trends of the signal, with higher signal energy, while the high-frequency components of the
Table 2: Detailed setting of the encoder for six MuJoCo tasks.
Environment	Number of Layers	Number of Units per Layer	Activation Function
HalfCheetah-v2	8	30	Swish
Walker2d-v2	6	40	Swish
Hopper-v2	6	40	Swish
Ant-v2	6	40	Swish
Swimmer-v2	6	40	Swish
Humanoid-v2	8	40	Swish
DTFT reflect the amount of short-term variation present in the state sequences. Therefore, we attempt to preserve the overall information of the low and high-frequency components of the predicted DTFT by directly computing the cosine similarity distance without undergoing the dimensionality reduction process. For the remaining frequency components of the predicted DTFT, we first utilize projection layers to perform dimensionality reduction, followed by calculating the cosine similarity distance. The sum of these three distances is used as the final loss function, which we call freqloss.