SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Experimental Setup. For all experiments, we use a finite  $m$ -horizon undiscounted return setup, with each domain specific horizon choice given by Table 3 in [20]. The CTW depth parameter to both the environment model ( $\hat{\xi} \coloneqq \mathrm{CTW}_d$ ) and the self-prediction model ( $\hat{\zeta} \coloneqq \mathrm{CTW}_d$ ) was also chosen to match Table 3 in [20]. We evaluated across 5 stochastic, partially observable and history dependent domains: Cheeze Maze, Kuhn Poker, 4x4 Grid, Tiger and Biased Scissor/Paper/Rock. The description for each of these domains can once again be found in [20]. Each environment was evaluated by performing a single online run across  $10^4$  timesteps; here a smaller number of timesteps was used than in [20] so that we could better assess the transient effects on performance improvement. At each timestep  $t$ , both agents pick either a random action with probability  $\epsilon_t \coloneqq 0.2 \times 0.999^t$ , or otherwise return the estimated best action according to action-value estimates computed with 500 simulations.
Results. Figure 1 shows learning curves for the Cheeze Maze, Tiger and 4x4 Grid domains respectively. The vertical axis shows the average reward per timestep obtained over the previous 2000 time steps. In these 3 domains we see that self-modelling gives a qualitative advantage over the original MC-AIXI agent given the restricted data and computational budget described above, approaching optimal performance in each domain; note that optimal performance can be obtained too with the original MC-AIXI agent, albeit with more compute and data, as described in [20]. This matches the theory which states that both agents will converge to optimal performance, but shows that self-modelling can, in some cases, converge to the optimal behavior more quickly. In two of the easier domains, ScissorPaperRock and Kuhn Poker, there was no significant difference in performance between the two methods. The final performance (as evaluated by the average reward per step over the final 2000 timesteps) of each agent configuration is shown in Table 2.
Agent	Kuhn Poker	Biased Scissor Paper Rock	Cheese Maze	Tiger	4×4 Grid
MC-AIXI	1.98	0.17	-1.0	-0.29	0.0
MC-Self-AIXI	1.93	0.22	1.08	0.5	0.24
Table 2: Final performance of MC-AIXI-CTW and MC-Self-AIXI-CTW after  ${10}^{4}$  time steps.
Cheese Maze: Rolling mean reward over time
Tiger: Rolling mean reward over time
4x4 Grid: Rolling mean reward over time
Figure 1: Rolling average reward per step for MC-AIXI (green) and MC-Self-AIXI (red).