Table 6: Raw scores across 55 games. We report the best scores for DQN, QR-DQN, IQN*, and Rainbow on 50M frames, averaged by 5 seeds. Reference values were provided by Dopamine framework [4]. **Bolds** are wins against DQN, QR-DQN, and *asterisk* are wins over IQN* and Rainbow. Note that IQN* and Rainbow implemented in Dopamine framework applied  $n$ -step updates with  $n = 3$  which improves performance.
GAMES	RANDOM	HUMAN	DQN(50M)	QR-DQN(50M)	IQN*(50M)	RAINBOW(50M)	PQR(50M)
Alien	227.8	7127.7	1688.1	2754.2	4016.3	2076.2	3173.9
Amidar	5.8	1719.5	888.2	841.6	1642.8	1669.6	*2814.7
Assault	222.4	742.0	1615.9	2233.1	4305.6	2535.9	*8456.5
Asterix	210.0	8503.3	3326.1	3540.1	7038.4	5862.3	*19004.6
Asteroids	719.1	47388.7	828.2	1333.4	1336.3	1345.1	851.8
Atlantis	12850.0	29028.1	388466.7	879022.0	897558.0	870896.0	880303.7
BankHeist	14.2	753.1	720.2	964.1	1082.8	1104.9	1050.1
BattleZone	2360.0	37187.5	15110.3	25845.6	29959.7	32862.1	*61494.4
BeamRider	343.9	16926.5	4771.3	7143.0	7113.7	6331.9	*12217.6
Berzerk	123.7	2630.4	529.2	603.2	627.3	697.8	*2707.2
Bowling	23.1	160.7	38.5	55.3	33.6	55.0	*174.1
Boxing	0.1	12.1	80.0	96.6	97.8	96.3	96.7
Breakout	1.7	30.5	113.5	40.7	164.4	69.8	48.5
Centipede	2090.9	12017.0	3403.7	3562.5	3746.1	5087.6	*31079.8
ChopperCommand	811.0	7387.8	1615.3	1600.3	6654.1	5982.0	4653.9
CrazyClimber	10780.5	35829.4	111493.8	108493.9	131645.8	135786.1	105526.0
DemonAttack	152.1	1971.0	4396.7	3182.6	7715.5	6346.4	*19530.2
DoubleDunk	-18.6	-16.4	-16.7	7.4	20.2	17.4	15.0
Enduro	0.0	860.5	2268.1	2062.5	766.5	2255.6	1765.5
FishingDerby	-91.7	-38.7	12.3	48.4	41.9	37.6	46.8
Freeway	0.0	29.6	25.8	33.5	33.5	33.2	33.0
Frostbite	65.2	4334.7	760.2	8022.8	7824.9	5697.2	*8401.5
Gopher	257.6	2412.5	3495.8	3917.1	11192.6	7102.1	*12252.9
Gravitar	173.0	3351.4	250.7	821.3	1083.5	926.2	703.5
Hero	1027.0	30826.4	12316.4	14980.0	18754.0	31254.8	15655.8
IceHockey	-11.2	0.9	-6.7	-4.5	0.0	2.3	0.0
Jamesbond	29.0	302.8	500.0	802.3	1118.8	656.7	*1454.9
Kangaroo	52.0	3035.0	6768.2	4727.3	11385.4	13133.1	*13894.0
Krull	1598	2665.5	6181.1	8073.9	8661.7	6292.5	*31927.4
KungFuMaster	258.5	22736.3	20418.8	20988.3	33099.9	26707.0	22040.4
MontezumaRevenge	0.0	4753.3	2.6	300.5	0.7	501.2	0.0
MsPacman	307.3	6951.6	2727.2	3313.9	4714.4	3406.4	*5426.5
NameThisGame	2292.3	8049.0	5697.3	7307.9	9432.8	9389.5	*9891.3
Phoenix	761.4	7245.6	5833.7	4641.1	5147.2	8272.9	5260
Pitfall	-229.4	6463.7	-16.8	-3.4	-0.4	0.0	*0.0
Pong	-20.7	14.6	13.2	19.2	19.9	19.4	19.7
PrivateEye	24.9	69571.3	1884.6	680.7	1287.3	4298.8	*12806.1
Qbert	163.9	13455.0	8216.2	17228.0	15045.5	17121.4	15806.9
Riverraid	1338.5	17118.0	9077.8	13389.4	14868.6	15748.9	14101.3
RoadRunner	11.5	7845.0	39703.1	44619.2	50534.1	51442.4	48339.7
Robotank	2.2	11.9	25.8	53.6	65.9	63.6	48.7
Seaquest	68.4	42054.7	1585.9	4667.9	20081.3	3916.2	5038.1
Skiing	-17098.1	-4336.9	-17038.2	-14401.6	-13755.6	-17960.1	*-9021.2
Solaris	1236.3	12326.7	2029.5	2361.7	2234.5	2922.2	*7145.3
SpaceInvaders	148.0	1668.7	1361.1	940.2	3115.0	1908.0	1602.4
StarGunner	664.0	10250.0	1676.5	23593.3	60090.0	39456.3	59404.6
Tennis	-23.8	-9.3	-0.1	19.2	3.5	0.0	*15.4
TimePilot	3568.0	5229.2	3200.9	6622.8	9820.6	9324.4	5597.0
Tutankham	11.4	167.6	138.8	209.9	250.4	252.2	147.3
UpNDown	533.4	11693.2	10405.6	29890.1	44327.6	18790.7	32155.5
Venture	0.0	1187.5	50.8	1099.6	1134.5	1488.9	1000.0
VideoPinball	16256.9	17667.9	216042.7	250650.0	486111.5	536364.4	460860.9
WizardOfWor	563.5	4756.5	2664.9	2841.8	6791.4	7562.7	5738.2
YarsRevenge	3092.9	54576.9	20375.7	66055.9	57960.3	31864.4	*67545.8
Zaxxon	32.5	9173.3	1928.6	8177.2	12048.6	14117.5	9531.8