SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The learning rate tables and the corresponding performance for the full continual RL experiments are shown in Fig. C.16 and Tables C.11, C.12, C.17, C.18, C.19, and C.20. We use 30 seeds to find the best hyperparameters for the tabular task, but we use 3 seeds for JBW experiment and MinAtar.
Figure C.11: Mean AUC of RMSVE against learning rate for various algorithms.
Algorithm	LRs
Deep TD (without reset)	{1e-1, 3e-2, 1e-2, 3e-3, 1e-3}
Deep TD (with reset)	{1e-1, 3e-2, 1e-2, 3e-3, 1e-3}
Deep PT-TD	PV: {1e-3, 3e-4, 1e-4, 3e-5}
TV: {1e-2, 5e-3, 3e-3, 1e-3}
Table C.7: LRs considered to pick the best one for the Deep prediction experiment with discrete grid.
We find the best learning rate for each  $k$ ,  $\lambda$  pair by searching over the learning rates listed in Table C.11.
Observations: We see that our algorithm performs better than their vanilla counterparts on all the problems we tested for a broad range of hyperparameters. There are a handful of  $(\overline{\alpha}, \alpha)$  pairs that results in a better performance in each setting, which indicates that our method is not sensitive to hyperparameter selection. We also observe that in all the cases  $\overline{\alpha}$  is smaller than  $\alpha$ , which fits into our intuition. Because, the permanent value function has to capture regularities in the value function estimate, it needs to update slowly, whereas, the transient value function should be updated quickly to facilitate fast adaptation. The sensitivity of two new hyperparameters:  $\lambda, k$  for the fully continual RL setting is analyzed in the main paper (see Sec. 6.1) where we concluded that our algorithm is robust to these values.
Figure C.12: Mean AUC of RMSVE against learning rate for various algorithms on image grid.
Algorithm	LRs
TD (without reset)	{1e-1, 3e-2, 1e-2, 3e-3, 1e-3}
TD (with reset)	{1e-1, 3e-2, 1e-2, 3e-3, 1e-3}
permanent value function	{3e-2, 1e-2, 3e-3, 1e-3, 3e-4}
transient value function	{3e-3, 1e-3, 3e-4, 1e-4, 3e-5, 1e-5, 3e-6, 1e-6}
Table C.8: Search space of learning rate for the policy evaluation minigrid task.
Figure C.13: Mean AUC of RMSVE against learning rate for various algorithms on minigrid grid.
Algorithm	LRs
Q-learning	{8e-1, 5e-1, 1e-1, 5e-2, 1e-2, 5e-3, 1e-3}
Q-learning (with reset)	{8e-1, 5e-1, 1e-1, 5e-2, 1e-2, 5e-3, 1e-3}
Permanent Value Function	{8e-1, 5e-1, 3e-1, 1e-1, 5e-2, 1e-2, 5e-3, 1e-3}
Transient Value Function	{8e-1, 5e-1, 3e-1, 1e-1, 5e-2, 1e-2}
Table C.9: Search space of learning rates for the tabular control task.
Figure C.14: Mean returns for various learning rates for the tabular control task.
Algorithm	LRs
Q-learning	{3e-3, 1e-3, 3e-4, 1e-4, 3e-5}
Q-learning (with reset)	{1e-2, 3e-3, 1e-3, 3e-4, 1e-4}
Permanent Value Function	{1e-5, 3e-6, 1e-6, 3e-7}
Transient Value Function	{3e-4, 1e-4, 3e-5, 1e-5}
Table C.10: Search space of learning rates.
Figure C.15: Mean returns for various learning rates for the Minigrid control task.
Figure C.16: Performance plot of PT-Q-learning against Q-learning for  $k, \lambda$  pairs
Algorithm	LRs
Q-learning	{8e-1, 5e-1, 1e-1, 5e-2, 1e-2, 5e-3, 1e-3}
Permanent Value Function (0.5x)	{1e-4, 1e-5, 1e-6, 1e-7, 1e-8}
Transient Value Function (0.5x)	{1e-2, 1e-3, 1e-4, 1e-5}
Table C.11: Search space of learning rates for the main JBW experiment.
Algorithm	LRs
Q-learning	{1e-2, 1e-3, 1e-4, 1e-5}
Permanent Value Function	{1e-4, 1e-5, 1e-6, 1e-7}
Transient Value Function	{1e-2, 1e-3, 1e-4, 1e-5}
Table C.12: Search space of learning rates for the main JBW ablation experiment.
TV-LR → PV-LR ↓	λ=0.55	λ=0.75	λ=0.95
1e-2	1e-3	1e-4	1e-5	1e-2	1e-3	1e-4	1e-5	1e-2	1e-3	1e-4	1e-5
1e-4	0.031	0.223	0.126	0.103	0.035	0.257	0.154	0.154	0.038	0.217	0.199	0.159
1e-5	0.031	0.252	0.071	0.051	0.030	0.261	0.077	0.068	0.040	0.220	0.201	0.096
1e-6	0.041	0.242	0.070	0.061	0.034	0.260	0.090	0.083	0.030	0.234	0.207	0.084
1e-7	0.036	0.256	0.056	0.067	0.040	0.268	0.062	0.070	0.028	0.227	0.216	0.124
1e-8	0.031	0.247	0.051	0.069	0.041	0.261	0.069	0.069	0.037	0.232	0.213	0.080
Table C.13: JBW: Reward per step for various hyperparameters for PT-DQN. We highlight the values that is larger than DQN's best performance.
LRs	1e-2	1e-3	1e-4	1e-5
Returns	0.03	0.127	0.192	0.093
Table C.14: JBW: Reward per step for various hyperparameters for DQN.
LRs	1e-2	1e-3	1e-4	1e-5
Returns	0.037	0.135	0.247	0.245
Table C.15: JBW: Reward per step for various hyperparameters for DQN-multi-head.
LRs	1e-2	1e-3	1e-4	1e-5
Returns	0.025	0.055	0.055	0.031
Table C.16: JBW: Reward per step for various hyperparameters for DQN with large experience replay.
TV-LR → PV-LR ↓	λ=0.55	λ=0.75	λ=0.95
1e-3	1e-4	1e-5	1e-3	1e-4	1e-5	1e-3	1e-4	1e-5
1e-7	14.41	17.38	7.89	13.87	19.07	10.55	9.50	16.12	12.16
1e-8	15.29	17.35	7.79	14.02	20.54	10.72	9.79	16.71	12.48
1e-9	15.54	17.51	8.02	12.63	19.54	10.90	8.57	16.48	11.76
Table C.17: MinAtar: AUC of the average return (in the previous 100 episodes) per step for various hyperparameters for PT-DQN. We highlight the values that is larger than DQN's best performance.
LRs	1e-3	1e-4	1e-5	1e-6
Returns	7.10	12.90	13.14	5.90
Table C.18: MinAtar: AUC of the average return (in the previous 100 episodes) per step for various hyperparameters for DQN.
LRs	1e-3	1e-4	1e-5	1e-6
Returns	8.72	14.93	15.68	8.54
Table C.19: MinAtar: AUC of the average return (in the previous 100 episodes) per step for various hyperparameters for DQN-multi-head.
LRs	1e-3	1e-4	1e-5	1e-6
Returns	6.78	11.44	10.70	4.56
Table C.20: MinAtar: AUC of the average return (in the previous 100 episodes) per step for various hyperparameters for DQN with large experience replay.