SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 3: Hyperparameters of auxiliary prediction tasks.
Hyperparameter	Setting
Optimizer	Adam
Discount γ	0.99
Learning rate	0.0003
Number of batch size	256
Predictor: Number of hidden layers	1
Predictor: Number of hidden units per layer	1024
Predictor: Activation function	ReLU
Projection: Number of hidden layers	1
Projection: Number of hidden units per layer	512
Projection: Activation function	ReLU
Projection2: Number of hidden layers	1
Projection2: Number of hidden units per layer	512
Projection2: Activation function	ReLU
Number of discrete points for sampling the DTFT L	128
The dimensionality of the output of projection	512
Replay buffer size	100,000
Pre-training steps	10000
Target smoothing coefficient τ	0.01
Target update interval K	1000
Hyperparameters of SPF-SAC
Each module: Normalization Layer	BatchNormalization
Random collection steps before pre-training	10,000
Hyperparameters of SPF-PPO
Each module: Normalization Layer	LayerNormalization
Random collection steps before pre-training	4,000
θaux update interval K2
HalfCheetah-v2	5
Walker2d-v2	2
Hopper-v2	150
Ant-v2	150
Swimmer-v2	200
Humanoid-v2	1
We select  $L = 128$  as the number of discrete points sampled over one period of DTFT. In practice, due to the symmetry conjugate of DTFT, the predictor  $\mathcal{F}$  only predicts  $\frac{L}{2} + 1$  points on the left half of our frequency map, as mentioned in Section 5.2. The projection module described in Section 5.3 projects the predicted value, a matrix with the dimension of  $L * D$ , into a 512-dimensional vector. To update target networks, we overwrite the target network parameters with an exponential moving average of the online network parameters, with a smoothing coefficient of  $\tau = 0.01$  for every  $K = 1000$  steps.
In order to eliminate dependency on the initial parameters of the policy, we use a random policy to collect transitions into the replay buffer [32] for the first 10K time steps for SAC, and 4K time steps for PPO. We also pretrain the representations with the aforementioned random collected samples to stabilize inputs to each RL algorithm, as described in [19].
The network architectures, optimizers, and hyperparameters of SAC and PPO are the same as those used in their original papers, except that we use mini-batches of size 256 instead of 100. As for PPO, we perform  $K_{2}$  gradient updates of  $\theta_{\mathrm{aux}}$  for every  $K_{2}$  steps of data sampling. The update interval  $K_{2}$  is set differently for six MuJoCo tasks and can be found in Table 3.
E Visualization
To demonstrate that the representations learned by SPF effectively capture the structural information contained in infinite-step state sequences, we compare the true state sequences with the states recovered from the predicted DTFT via the inverse DTFT.
Specifically, we first generate a state sequence from the trained policy and select a goal state  $s_t$  at a certain time step. Next, we choose a historical state  $s_{t - k}$  located  $\mathbf{k}$  steps past the goal state and select an action  $a_{t - k}$  based on the trained policy  $\pi (\cdot |s_{t - k})$  as the inputs of our trained predictor. We then obtain the DTFT  $F_{t - k}\coloneqq F_{\pi}(s_{t - k},a_{t - k})$  of state sequences starting from the state  $s_{t - k + 1}$ . Next, we compute the kth element of the inverse DTFT of  $F_{t - k}$  and obtain a recovered state  $\widehat{s}_t$ , which represents that we predict the future goal state using the historical state located  $\mathrm{k}$  steps past the goal state. By selecting a sequence of states over a specific time interval as the goal states and repeating the aforementioned procedures, we will obtain a state sequence recovered by k-step prediction. In Figure 5(b), 6(b), 7(b), 8(b), 9(b) and 10(b), we visualize the true state sequence (the blue line) and the recovered state sequences (the red lines) via k-step predictions for  $k = 1,2,3,4,5$ . Note that the lighter red line corresponds to predictions made by historical states from a more distant time step. We conduct the visualization experiment on six MuJoCo tasks using the representations and predictors trained by SPF-SAC or SPF-PPO. Due to the large dimensionality of the states in Ant-v2 and Humanoid-v2, which contain many zero values, we have chosen to visualize only six dimensions of their states, respectively. The fine distinctions between the true state sequences and the recovered state sequences from our trained representations and predicted FT indicates that our representation effectively captures the inherent structures of future state sequences.
Furthermore, we provide a visualization that compares the true DTFT and the predicted DTFT in Figure 5(a), 6(a), 7(a), 8(a), 9(a) and 10(a). To accomplish this, we use our trained policies to interact with the environments and select the state sequences of the 200 last steps of an episode. The blue lines represent the true DTFT of these state sequences, while the orange line represents the predicted DTFT using the online encoder and predictor trained by our learned policies. It is evident that the true DTFT and the predicted DTFT exhibit significant differences. These results demonstrate the ability of SPF to effectively extract the underlying structural information in infinite-step state sequences without relying on high prediction accuracy.
F Cosine Similarity Results
We present a comparison table between the real discrete-time Fourier transform (DTFT) and the predicted DTFT. To obtain these results, we use our trained policies to interact with six MuJoCo environments and record the states and the actions for the 201 last steps of an episode, denoted as  $\{s_t\}_{t=0}^{200}$  and  $\{a_t\}_{t=0}^{200}$ , respectively. The real DTFT of the state sequence  $\{s_1, s_2, \ldots, s_{200}\}$  over a 200-step horizon is computed using the formulation of the discrete-time Fourier transform. For the predicted DTFT, we employ our trained encoder and prediction module, with the state  $s_0$  and the action  $a_0$  serving as inputs. The output of the prediction module represents the predicted DTFT of the state sequence  $\{s_1, s_2, \ldots, s_{200}\}$ . We compute the cosine similarity between  $A$  and  $B$  with the
formula  $-\frac{A \cdot B}{\|A\| \|B\|}$ , where values closer to  $-1$  indicate greater similarity while the values closer to  $1$  indicate greater dissimilarity. The cosine similarity between the real DTFT and the predicted DTFT is listed in Table 4 below.
Table 4: Cosine similarity distance between the real DTFT and the predicted DTFT.
Environment	Cosine similarity distance
HalfCheetah	-0.655
Hopper	-0.660
Walker2d	-0.651
Ant	-0.351
Swimmer	-0.557
Humanoid	-0.651
G Code
Codes for the proposed method are available at https://github.com/MIRALab-USTC/RL-SPF/.
Figure 5: Predicted values via representations trained by SPF-SAC on HalfCheetah-v2
(b) True and recovered state sequences
(a) True and predicted DTFT
Figure 6: Predicted values via representations trained by SPF-SAC on Walker2d-v2
(a) True and predicted DTFT
(b) True and recovered state sequences
Figure 7: Predicted values via representations trained by SPF-SAC on Humanoid-v2
Figure 8: Predicted values via representations trained by SPF-PPO on Hopper-v2
Figure 9: Predicted values via representations trained by SPF-PPO on Ant-v2
Figure 10: Predicted values via representations trained by SPF-PPO on Swimmer-v2