SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left| \mathbb {E} \left[ Z ^ {(k + 1)} (s, a) \right] - \mathbb {E} \left[ Z ^ {(k)} (s, a) \right] \right| \\ \leq \gamma \sup  _ {s ^ {\prime}} \left| \mathbb {E} \left[ Z ^ {(k)} \left(s ^ {\prime}, a ^ {*} \left(\xi_ {k + 1}\right)\right) \right] - \mathbb {E} \left[ Z ^ {(k - 1)} \left(s ^ {\prime}, a ^ {*} \left(\xi_ {k}\right)\right) \right] \right| \\ \leq \gamma \sup  _ {s ^ {\prime}} \left(\left| \mathbb {E} [ Z ^ {(k)} (s ^ {\prime}, a ^ {*} (\xi_ {k + 1}) ] - \max  _ {a ^ {\prime}} \mathbb {E} [ Z ^ {(k)} (s ^ {\prime}, a ^ {\prime}) ] \right| + \left| \max  _ {a ^ {\prime}} \mathbb {E} [ Z ^ {(k)} (s ^ {\prime}, a ^ {\prime}) ] - \max  _ {a ^ {\prime}} \mathbb {E} [ Z ^ {(k - 1)} (s ^ {\prime}, a ^ {\prime}) ] \right| \right. \\ \left. + \left| \max  _ {a ^ {\prime}} \mathbb {E} \left[ Z ^ {(k - 1)} \left(s ^ {\prime}, a ^ {\prime}\right) \right] - \mathbb {E} \left[ Z ^ {(k - 1)} \left(s ^ {\prime}, a ^ {*} \left(\xi_ {k}\right)\right) \right] \right|\right) \\ \leq \gamma \sup  _ {s ^ {\prime}, a ^ {\prime}} \left| \mathbb {E} [ Z ^ {(k)} (s ^ {\prime}, a ^ {\prime}) ] - \mathbb {E} [ Z ^ {(k - 1)} (s ^ {\prime}, a ^ {\prime}) ] \right| + \gamma \sum_ {i = k - 1} ^ {k} \sup  _ {s ^ {\prime}} \left| \mathbb {E} [ Z ^ {(i)} (s ^ {\prime}, a ^ {*} (\xi_ {i + 1})) ] - \max  _ {a ^ {\prime}} \mathbb {E} [ Z ^ {(i)} (s ^ {\prime}, a ^ {\prime}) ] \right| \\ \leq \gamma \left\| \mathbb {E} [ Z ^ {(k)} ] - \mathbb {E} [ Z ^ {(k - 1)} ] \right\| _ {s a} + \gamma \sum_ {i = k - 1} ^ {k} \left[ \right. \sup  _ {s ^ {\prime}} \left(\left| \mathbb {E} [ Z ^ {(i)} (s ^ {\prime}, a ^ {*} (\xi_ {i + 1})) ] - \mathbb {E} _ {\xi_ {i + 1}} [ Z ^ {(i)} (s ^ {\prime}, a ^ {*} (\xi_ {i + 1})) ] \right|\right. \\ \left. + \left| \max  _ {a ^ {\prime}} \mathbb {E} _ {\xi_ {i + 1}} \left[ Z ^ {(i)} \left(s ^ {\prime}, a ^ {\prime}\right) \right] - \max  _ {a ^ {\prime \prime}} \mathbb {E} \left[ Z ^ {(i)} \left(s ^ {\prime}, a ^ {\prime \prime}\right) \right] \right|\right) \Bigg) \\ \leq \gamma \left\| \mathbb {E} [ Z ^ {(k)} ] - \mathbb {E} [ Z ^ {(k - 1)} ] \right\| _ {s a} + 2 \gamma \sum_ {i = k - 1} ^ {k} \sup  _ {s ^ {\prime}, a ^ {\prime}} \left(\left| \mathbb {E} [ Z ^ {(i)} (s ^ {\prime}, a ^ {\prime}) ] - \mathbb {E} _ {\xi_ {i + 1}} [ Z ^ {(i)} (s ^ {\prime}, a ^ {\prime}) ] \right|\right) \\ \leq \gamma \left\| \mathbb {E} [ Z ^ {(k)} ] - \mathbb {E} [ Z ^ {(k - 1)} ] \right\| _ {s a} + 2 \gamma \sum_ {i = k - 1} ^ {k} \Delta_ {i + 1} \\ \end{array}
$$
where we use A.1.1 in third and fifth line and A.1.2 in sixth line.
Taking a supremum over  $s$  and  $a$ , then for all  $k > 0$ ,
$$
\begin{array}{l} \left\| \mathbb {E} [ Z ^ {(k + 1)} ] - \mathbb {E} [ Z ^ {(k)} ] \right\| _ {s a} \leq \gamma \left\| \mathbb {E} [ Z ^ {(k)} ] - \mathbb {E} [ Z ^ {(k - 1)} ] \right\| _ {s a} + 2 \sum_ {i = k - 1} ^ {k} \gamma \Delta_ {i + 1} \\ \leq \gamma^ {2} \left\| \mathbb {E} [ Z ^ {(k - 1)} ] - \mathbb {E} [ Z ^ {(k - 2)} ] \right\| _ {s a} + 2 \sum_ {i = k - 2} ^ {k - 1} \gamma^ {2} \Delta_ {i + 1} + 2 \sum_ {i = k - 1} ^ {k} \gamma \Delta_ {i + 1} \\ \begin{array}{c} \vdots \\ \vdots \\ \vdots \end{array} \\ \leq \gamma^ {k} \left\| \mathbb {E} [ Z ^ {(1)} ] - \mathbb {E} [ Z ] \right\| _ {s a} + 2 \sum_ {i = 1} ^ {k} \gamma^ {i} \left(\Delta_ {k + 2 - i} + \Delta_ {k + 1 - i}\right) \\ \leq 2 \gamma^ {k} V _ {\max } + 2 \sum_ {i = 1} ^ {k} \gamma^ {i} \left(\Delta_ {k + 2 - i} + \Delta_ {k + 1 - i}\right) \\ \end{array}
$$
Since  $\sum_{i=1}^{\infty} \gamma^{i} = \frac{\gamma}{1 - \gamma} < \infty$  and  $\sum_{i=1}^{\infty} \Delta_{i} < \infty$  by assumption, we have
$$
\sum_ {i = 1} ^ {k} \gamma^ {i} \Delta_ {k + 1 - i} \rightarrow 0
$$
which is resulted from the convergence of Cauchy product of two sequences  $\{\gamma^i\}$  and  $\{\Delta_i\}$ . Hence,  $\{\mathbb{E}[Z^{(k)}]\}$  is a Cauchy sequence and therefore converges for every  $Z \in \mathcal{Z}$ .
Let  $\mathbb{E}[Z^{*}]$  be the limit point of the sequence  $\{\mathbb{E}[Z^{(n)}]\}$ . Then,
$$
\begin{array}{l} \left\| \mathbb {E} [ Z ^ {*} ] - \mathbb {E} [ Z ^ {(n)} ] \right\| _ {s a} = \lim  _ {l \rightarrow \infty} \left\| \mathbb {E} [ Z ^ {(n + l)} ] - \mathbb {E} [ Z ^ {(n)} ] \right\| _ {s a} \\ \leq \sum_ {k = n} ^ {\infty} \left\| \mathbb {E} \left[ Z ^ {(k + 1)} \right] - \mathbb {E} \left[ Z ^ {(k)} \right] \right\| _ {s a} \\ = \sum_ {k = n} ^ {\infty} \left(2 \gamma^ {k} V _ {\max } + 2 \sum_ {i = 1} ^ {k} \gamma^ {i} \left(\Delta_ {k + 2 - i} + \Delta_ {k + 1 - i}\right)\right). \\ \end{array}
$$
A.4 Proof of Theorem 3.4
Theorem 3.4. If  $\{\Delta_n\}$  follows the assumption in Theorem 3.3, then  $\mathbb{E}[Z^*]$  is the unique solution of Bellman optimality equation.
Proof. The proof follows by linearity of expectation. Denote the Q-value based operator as  $\widetilde{\tau}$ . Note that  $\Delta_{n}$  converges to 0 with regularity of  $\mathcal{Z}$  implies that  $\xi_{n}$  converges to 1 in probability on  $\Omega$ , i.e.,
$$
\begin{array}{l} \lim  _ {n \rightarrow \infty} \sup  _ {s, a} \left| \int_ {w \in \Omega} Z ^ {(n)} (w; s, a) (1 - \xi_ {n} (w)) \mathbb {P} (d w) \right| = 0 \\ \Rightarrow \lim  _ {n \rightarrow \infty} \mathbb {P} \left(\left\{w \in \Omega : | 1 - \xi_ {n} (w) | \geq \epsilon \right\}\right) = 0 \\ \end{array}
$$
By Theorem A.3, for a given  $\epsilon > 0$ , there exists a constant  $K = \max(K_1, K_2)$  such that for every  $k \geq K_1$ ,
$$
\sup  _ {Z \in \mathcal {Z}} \| \bar {\mathcal {T}} _ {\xi_ {k}} \mathbb {E} [ Z ] - \bar {\mathcal {T}} \mathbb {E} [ Z ] \| _ {s a} \leq \frac {\epsilon}{2}.
$$
Since  $\bar{T}$  is continuous, for every  $k\geq K_{2}$
$$
\| \bar {\mathcal {T}} \mathbb {E} [ Z ^ {(k)} ] - \bar {\mathcal {T}} \mathbb {E} [ Z ^ {*} ] \| _ {s a} \leq \frac {\epsilon}{2}.
$$
Thus, it holds that
$$
\begin{array}{l} \| \bar {\mathcal {T}} _ {\xi_ {k + 1}} \mathbb {E} [ Z ^ {(k)} ] - \bar {\mathcal {T}} \mathbb {E} [ Z ^ {*} ] \| _ {s a} \leq \| \bar {\mathcal {T}} _ {\xi_ {k + 1}} \mathbb {E} [ Z ^ {(k)} ] - \bar {\mathcal {T}} \mathbb {E} [ Z ^ {(k)} ] \| _ {s a} + \| \bar {\mathcal {T}} \mathbb {E} [ Z ^ {(k)} ] - \bar {\mathcal {T}} \mathbb {E} [ Z ^ {*} ] \| _ {s a} \\ \leq \sup  _ {Z \in \mathcal {Z}} \| \bar {\mathcal {T}} _ {\xi_ {k + 1}} \mathbb {E} [ Z ] - \bar {\mathcal {T}} \mathbb {E} [ Z ] \| _ {s a} + \| \bar {\mathcal {T}} \mathbb {E} [ Z ^ {(k)} ] - \bar {\mathcal {T}} \mathbb {E} [ Z ^ {*} ] \| _ {s a} \\ \leq \frac {\epsilon}{2} + \frac {\epsilon}{2} \\ = \epsilon . \\ \end{array}
$$
$$
\mathbb {E} [ Z ^ {*} ] = \lim  _ {k \rightarrow \infty} \mathbb {E} [ Z ^ {(k)} ] = \lim  _ {k \rightarrow \infty} \mathbb {E} [ Z ^ {(k + 1)} ] = \lim  _ {k \rightarrow \infty} \mathbb {E} [ \mathcal {T} _ {\xi_ {k + 1}} Z ^ {(k)} ] = \lim  _ {k \rightarrow \infty} \bar {\mathcal {T}} _ {\xi_ {k + 1}} \mathbb {E} [ Z ^ {(k)} ] = \bar {\mathcal {T}} \mathbb {E} [ Z ^ {*} ]
$$
Since the standard Bellman optimality operator has a unique solution, we derived the desired result.
B Algorithm Pipeline
Figure 8 shows the pipeline of our algorithm. With the schedule of perturbation bound  $\{\Delta_n\}$ , the ambiguity set  $\mathcal{U}_{\Delta_n}(Z_{n - 1})$  can be defined by previous  $Z_{n - 1}$ . For each step, (distributional) perturbation  $\xi_{n}$  is sampled from  $\mathcal{U}_{\Delta_n}(Z_{n - 1})$  by the symmetric Dirichlet distribution and then PDBOO  $\mathcal{T}_{\xi_n}$  can be performed.
Figure 8: Pipeline of PDBOO.
Except for each own hyperparameter, our algorithms and DLTV shares the same hyperparameter and network architecture with QR-DQN [13] for a fair comparison. Also, we set up p-DLTV by only multiplying a gaussian noise  $\mathcal{N}(0,1)$  to the coefficient of DLTV. We do not combine any additional improvements of Rainbow such as double Q-learning,ueling network, prioritized replay, and  $n$ -step update. Experiments on LunarLander-v2 and Atari games were performed with 3 random seeds. The training process is  $0 - 2\%$  slower than QR-DQN due to the sampling  $\xi$  and reweighting procedures.
C.1 Hyperparameter Setting
We report the hyperparameters for each environments we used in our experiments.
Table 2: Table of hyperparameter setting
Hyperparameters	N-Chain	LunarLander	Atari Games
Batch size	64	128	32
Number of quantiles	200	170	200
n-step updates	1
Network optimizer	Adam
β	Grid search[0.05, 0.1, 0.5, 1] × 1^N
κ	1
Memory size	1e6	1e5	1e6
Learning rate	5e-5	1.5e-3	5e-5
γ	0.9	0.99	0.99
Update interval	1	1	4
Target update interval	25	1	1e4
Start steps	5e2	1e4	5e4
ε (train)	LinearAnnealer(1 → 1e-2)
ε (test)	1e-3	1e-3	1e-3
ε decay steps	2.5e3	1e5	2.5e5
Coefficient c	Grid search[1e0, 5e0, 1e1, 5e1, 1e2, 5e2, 1e3, 5e3]
Δ0	5e2	5e4	1e6
Number of seeds	10	3	3
C.2 Pseudocode of p-DLTV
Algorithm 2 Perturbed DLTV (p-DLTV)
Input: transition  $(s,a,r,s^{\prime})$  , discount  $\gamma \in [0,1)$ $Q(s^{\prime},a^{\prime}) = \frac{1}{N}\sum_{j}\theta_{j}(s^{\prime},a^{\prime})$ $c_{t}\sim c\mathcal{N}(0,\frac{\ln t}{t})$  //Randomize the coefficient  $a^{*}\gets \mathrm{argmax}_{a^{\prime}}(Q(s^{\prime},a^{\prime}) + c_{t}\sqrt{\sigma_{+}^{2}(s^{\prime},a^{\prime}))}$ $\mathcal{T}\theta_{j}\leftarrow r + \gamma \theta_{j}(s^{\prime},a^{*}),\quad \forall j$    
Output:  $\begin{array}{r}\sum_{i = 1}^{N}\mathbb{E}_{j}[\rho_{\hat{\tau}_{i}}^{\kappa}(\mathcal{T}\theta_{j} - \theta_{i}(s,a))] \end{array}$
D Further experimental results & Discussion
D.1 N-Chain
To explore the effect of intrinsic uncertainty, we run multiple experiments with various reward settings for the rightmost state as keeping their mean at 9. As the distance between two Gaussians was increased, the performance of DLTV decrease gradually, while other algorithms show consistent results. The result implies the interference of one-sided tendency on risk is proportional to the magnitude of the intrinsic uncertainty and the randomized criterion is effective in escaping from the issue.
Table 3: Total counts of performing true optimal action with 4 different seeds.
Total Count	(8,10)	(7,11)	(6,12)	(5,13)	(4,14)	(3,15)	(2,16)	(1,17)
QR-DQN	12293	11381	11827	12108	10041	11419	9696	11619
DLTV	9997	9172	9646	9251	7941	6964	7896	7257
p-DLTV	14344	14497	13769	15507	14469	14034	14068	13404
PQR	14546	15018	14693	15142	15361	13859	14602	14354
D.2 LunarLander-v2
Figure 9: (Left) Three main environmental factors causing high intrinsic uncertainty on LunarLander-v2. (Right) Performance on LunarLander-v2
To verify the effectiveness of the proposed algorithm in the complex environment with high intrinsic uncertainty, we conduct the experiment on LunarLander-v2. We have focused on three main factors that increase the intrinsic uncertainty from the structural design of LunarLander environment:
- Random initial force: The lander starts at the top center with an random initial force.
- Action stochasticity: The noise of engines causes different transitions with same action.
- Extreme reward system: If the lander crashes, it receives -100 points. If the lander comes to rest, it receives +100 points.
Therefore, several returns with a fixed policy have a high variance. As previously discussed about the fixedness from N-Chain environment, we can demonstrate that randomized approaches, PQR and p-DLTV, outperform other baselines in LunarLander-v2.
D.3 Atari games
Figure 10: Evaluation curves on Atari games. All curves are smoothed over 10 consecutive steps with three random seeds. In case of Pong-v4, we resize the x-axis, since it can easily obtain the optimal policy with few interactions due to its environmental simplicity.
We test our algorithm under 30 no-op settings to align with previous works. We compare our baseline results with results from the DQN Zoo framework [29], which provides the full benchmark results on 55 Atari games at 50M and 200M frames. We report the average of the best scores over 5 seeds for each baseline algorithms up to 50M frames.
However, recent studies tried to follow the setting proposed by Machado et al. [20] for reproducibility, where they recommended using sticky actions. Hence, we provide all human normalized scores results across 55 Atari games for 50M frames including previous report of Dopamine and DQN Zoo framework to help the follow-up researchers as a reference. We exclude Defender and Surround which is not reported on Yang et al. [38] because of reliability issues in the Dopamine framework. In summary,
- DQN Zoo framework corresponds to 30 no-op settings (version v4).
- Dopamine framework corresponds to sticky actions protocol (version v0).
D.3.1 Ablation on PQR schedules
Figure 11: Evaluation curves on Pong-v4 environments.
To investigate the effect of the schedule of  $\Delta_t$ , we run the experiment on Pong-v4 and set up several baselines as follows:
-  $1 / t^0$ : A fixed size ambiguity set.  $\Delta_t = O(1)$
- PQR: Our main algorithm.  $\Delta_t = O(1 / t^{1 + \epsilon})$
- OPT: We fix the output vector, sampled from the Dirichlet distribution, as  $[0, 0, \dots, 1]$ , forcing the agent to estimate only optimistically.
-  $\sqrt{\log t / t}$ : We imitate the schedule of p-DLTV (which does not satisfy the sufficient condition we presented).  $\Delta_t = O(\sqrt{\log t / t})$
-  $\sqrt{\log t / t} + \mathrm{OPT}$ : We imitate the schedule of DLTV (which does not satisfy the sufficient condition we presented). We fixed the output vector, sampled from the Dirichlet distribution, as  $[0, 0, \dots, 1]$ , forcing the agent to estimate only optimistically.  $\Delta_t = |O(\sqrt{\log t / t})|$
In this experiment, our proposed PQR is the only method that stably achieves the maximum score with low variance. In the case of optimism (purple, orange curve), the agent learns quickly in the early stages, but converges without reaching the maximum score. In the case of fixed ambiguity set (blue curve), it converges to suboptimal and eventually shows low performance. This result implies the necessity of time-varying schedule of  $\Delta_t$ . Finally, when imitating the schedule of p-DLTV (green curve), the performance also degrades implying that the proposed sufficient condition is quite tight.
D.4 Comparison with QUOTA
Zhang and Yao [40] have proposed Quantile Option Architecture(QUOTA) which derives different policies corresponding to different risk levels and consider them as options. By using an option-based framework, the agent learns a high-level policy that adaptively selects a pessimistic or optimistic exploration strategy. While QUOTA has a similar approach in high-level idea, PQR gives a lot of improvements in both theoretical analysis and experimental results.
- Theoretical guarantees of convergence toward risk-neutrality.
Since the agent selects via randomized risk criterion, the natural question is "How should we control the injected randomness without sacrificing the original purpose of risk-neutrality?" In this work, we provide the sufficient condition for convergence without sacrificing risk-neutral perspective. Although QUOTA explores by using optimism or pessimism of a value distribution, there is no discussion whether the convergence is guaranteed toward a risk-neutral objective.
- Explaining the effectiveness of randomized strategy.
QUOTA tested on two Markov chains to illustrate the inefficiency of expectation-based RL. It assumed that each task has an inherent, but unknown, preferred risk strategy, so agents should learn hidden preference. In contrast, we point out that the amount of inherent (intrinsic) uncertainty causes the inefficiency of fixed optimism or pessimism based exploration.
- Significant performance difference in experimental results.
QUOTA is based on option-based learning which requires an additional option-value network. While QUOTA aims to control risk-sensitivity by transforming into an option  $O$ , the introduction of an option-value network requires the agent to explore an action space  $|O| \times |A|$ . This opposes the idea of efficient exploration as a factor that increases the complexity of learning. In contrast, PQR does not require a additional network and explores over the original action space. In addition, PQR does not artificially discretize the ambiguity set of risk measurement. Another main reason is that PQR does not depend on an greedy schedule which is well-known for inefficient exploration strategies in tabular episodic MDP [18]. PQR solely explores its own strategies which is a simple yet effective approach. However, QUOTA depends on a greedy schedule in both quantile and option networks.
D.5 Reproducibility issues on DLTV
For the expected concerns about the comparison with DLTV, we address some technical issues to correct misconceptions of their performance. Before we reproduce the empirical results of DLTV, Mavrin et al. [21] did not report each raw scores of Atari games, but only the relative performance with cumulative rewards comparing with QR-DQN. While DLTV was reported to have a cumulative reward 4.8 times greater than QR-DQN, such gain mainly comes from VENTURE which is evaluated as  $22,700\%$  from their metric (i.e.,  $463\%$  performance gain solely). However, the approximate raw score of VENTURE was 900 which is lower than our score of 993.3. Hence, the report with cumulative rewards causes a severe misconception that can be overestimated where the human-normalized score is commonly used for evaluation metrics. For a fair comparison, we computed based on mean and median of human-normalized scores and obtained results of  $603.66\%$  and  $109.90\%$ . Due to the absence of public results, however, DLTV was inevitably excluded from the comparison with human-normalized score in the main paper for reliability. In Table 4 and 7, we report our raw scores and human-normalized score of DLTV based on QR-DQN_zoo performance.
Table 4: Performance comparison on QUOTA, DLTV, and PQR.
QUOTA >QR-DQN_Zhang	QR-DQN_zoo >QR-DQN_Zhang	PQR >QUOTA	PQR >QR-DQN_Zhang	PQR >DLTV
30	34	42	42	39
Avg HN Score(QR-DQN_zoo)	Avg HN Score(QR-DQN_Zhang)	Avg HN Score(QUOTA)	Avg HN Score(DLTV)	Avg HN Score(PQR)
505.02	463.47	383.70	603.66	1078.00
Med HN Score(QR-DQN_zoo)	Med HN Score(QR-DQN_Zhang)	Med HN Score(QUOTA)	Med HN Score(DLTV)	Med HN Score(PQR)
120.74	78.07	91.08	109.90	129.25
Table 7: Raw scores across all 49 games, starting with 30 no-op actions. We report the best scores for QR-DQN_zoo[29], QR-DQN_Zhang[40](implemented by QUOTA to evaluate the relative improvement) for a fair comparison and QUOTA[40], DLTV[21] on 40M frames, averaged by 3 seeds. **Bold** are wins against QUOTA and DLTV.
Games	Random	Human	QR-DQN_zoo(40M)	QR-DQN_Zhang(40M)	QUOTA(40M)	DLTV(40M)	PQR(40M)
Alien	227.8	7127.7	1645.7	1760.0	1821.9	2280.9	2406.9
Amidar	5.8	1719.5	552.9	567.9	571.4	1042.7	644.1
Assault	222.4	742	9880.4	3308.7	3511.1	5896.2	10759.2
Asterix	210	8503.3	13157.2	6176.0	6112.1	6336.6	8431.0
Asteroids	719.1	47388.7	1503.9	1305.3	1497.6	1268.7	1416.00
Atlantis	12850	29028.1	750190.1	978385.3	965193.0	845324.9	897640.0
BankHeist	14.2	753.1	1146.1	644.7	735.2	1183.7	1038.8
BattleZone	2360	37187.5	17788.4	22725.0	25321.6	23315.8	28470.5
BeamRider	363.9	16926.5	10684.2	5007.8	5522.6	6490.1	10224.9
Bowling	23.1	160.7	44.3	27.6	34.0	29.8	86.9
Boxing	0.1	12.1	98.2	95.0	96.1	112.8	97.1
Breakout	1.7	30.5	401.5	322.1	316.7	260.9	357.7
Centipede	2090.9	12017.0	6633.0	4330.3	3537.9	4676.7	6803.6
ChopperCommand	811.0	7387.8	1133.1	3421.1	3793.0	2586.3	1500.0
CrazyClimber	10780.5	35829.4	93499.1	107371.6	113051.7	92769.1	83900.0
DemonAttack	152.1	1971.0	98063.6	80026.6	61005.1	146928.9	73794.0
DoubleDunk	-18.6	-16.4	-10.5	-21.6	-21.5	-23.3	-10.5
Enduro	0.0	860.5	2105.7	1220.0	1162.3	5665.9	2252.8
FishingDerby	-91.7	-38.7	25.7	-9.6	-59.0	-8.2	31.7
Freeway	0.0	29.6	30.9	30.6	31.0	34.0	34.0
Frostbite	65.2	4334.7	3822.7	2046.3	2208.5	3867.6	4051.2
Gopher	257.6	2412.5	4191.2	9443.8	6824.3	10199.4	47054.5
Gravitar	173.0	3351.4	477.4	414.3	457.6	357.9	583.6
IceHockey	-11.2	0.9	-2.4	-9.8	-9.9	-14.3	-2.1
Jamesbond	29.0	302.8	907.1	601.7	495.5	779.8	1747.1
Kangaroo	52.0	3035	14171	2364.6	2555.8	4596.7	14385.1
Krull	1598.0	2665.5	9618.2	7725.4	7747.5	10012.21	9537.0
KungFuMaster	258.5	22736.3	27576.5	17807.4	20992.5	23078.4	38074.1
MontezumaRevenge	0.0	4753.3	0.0	0.0	0.0	0.0	0.0
MsPacman	307.3	6951.6	2561.0	2273.3	2423.5	3191.7	2895.6
NameThisGame	2292.3	8049.0	11770.0	7748.2	7327.5	8368.1	10298.2
Pitfall	-229.4	6463.7	0.0	-32.9	-30.7	-	0.0
Pong	-20.7	14.6	20.9	19.6	20.0	21.0	21.0
PrivateEye	24.9	69571.3	100.0	419.3	114.1	1358.6	372.4
Qbert	163.9	13455.0	8348.2	10875.3	11790.2	15856.2	14593.0
Riverraid	1338.5	17118.0	8814.1	9710.4	10169.8	10487.3	9374.7
RoadRunner	11.5	7845.0	52575.7	27640.7	27872.2	49255.7	44341.0
Robotank	2.2	11.9	50.4	45.1	37.6	58.4	53.9
Sequest	68.4	42054.7	5854.6	1690.5	2628.6	3103.8	16011.2
SpaceInvaders	148.0	1668.7	1281.8	1387.6	1553.8	1498.6	1562.6
StarGunner	664.0	10250.0	53624.7	49286.6	52920.0	53229.5	55475.0
Tennis	-23.8	-8.3	0.0	-22.7	-23.7	-18.4	-1.0
TimePilot	3568.0	5229.2	6243.4	6417.7	5125.1	6931.1	6506.4
Tutankham	11.4	167.6	200.0	173.2	195.4	130.9	213.3
UpNDown	533.4	11693.2	22248.8	30443.6	24912.7	44386.7	33786.3
Venture	0.0	1187.5	12.5	5.3	26.5	1305.0	0.0
VideoPinball	16256.9	17667.9	104227.2	123425.4	44919.1	93309.6	443870.0
WizardOfWor	563.5	4756.5	13133.8	5219.0	4582.0	9582.0	6132.8
Zaxxon	32.5	9173.3	7222.7	6855.1	8252.8	6293.0	10250.0