SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 5: Raw Atari scores for the three offline RL approaches. The scores are average results over three trials. The Human Avg and Random scores are adopted from [6].
Games	Human Avg	Random	EDT-ONE (80M)	DT-ONE (80M)	IQL-IQM (80M)
Asterix	8503.3	210	12089.6	7731.0	994.6
Assault	742.0	222.4	1849.2	1260.0	964.3
BankHeist	753.1	14.2	5.0	9.3	21.3
BeamRider	16926.5	363.9	6469.9	4864.6	3865.3
Breakout	30.5	1.7	220.1	180.8	70.9
Centipede	12017.0	2090.9	2164.0	2290.4	2940.4
ChopperCommand	7387.8	811	3491.6	2774.8	331.7
Enduro	860.5	0	823.6	557.1	877.8
FishingDerby	-38.7	-91.7	-1.7	-43.3	30.6
Freeway	29.6	0	26.0	13.5	37.8
Frostbite	4334.7	65.2	1837.6	1558.1	488.5
Gravitar	3351.4	173	70.5	119.8	-37.1
NameThisGame	8049.0	2292.3	7529.4	5939.7	3954.3
Phoenix	7242.0	761.4	4280.5	3369.6	1893.1
Qbert	13455.0	163.9	11739.8	7978.4	11035.5
Seaquest	42054.0	68.4	4410.9	2628.3	756.7
TimePilot	5229.2	3568.0	2737.7	3036.7	2311.3
VideoPinball	17667.9	0	875.7	476.0	0
WizardOfWor	4756.5	563.5	222.2	384.8	498.2
Zaxxon	9173.3	32.5	233.4	141.7	-6.5
B.5 Image Encoder
To ensure a fair comparison, we standardize the image encoder across EDT and the two baseline approaches. We adopt the image encoder from DrQ-v2 [60] for this purpose.
The architecture of the image encoder is as followed:
- 1 convolution with stride 2, output channels 32, kernel size 3. (ReLU).
- 3 convolution with stride 1, output channels 32, kernel size 3. (ReLU).
1 fully connected layer and  $H$  output dimensions.
Figure 11: All Atari games used for our experiments. We adopted gray scale and reshape observations to images of size  $84 \times 84$ .
C Inter-Quartile Mean of the Atari results
Alongside the main paper's results, we present the Inter-Quartile Mean (IQM) of our findings. Notably, IQL outperforms DT in terms of IQM. This occurs due to the variance in task difficulties, despite our use of HNS to balance scales across tasks. The difficulty disparity among games leads to significant variance. However, despite this, our proposed EDT still significantly surpasses both baselines, demonstrating its robust performance.
In the original Decision Transformer model, estimating the remaining return-to-go is dependent on a preset expected return-to-go value. This requirement presents a challenge when applied to multi-task environments, where return-to-go values can significantly vary across different tasks.
The issue is particularly pronounced in Atari games where high scores often necessitate long trajectories.
These long trajectories may consequently yield negative
estimates for the return-to-go, which presents an issue for the model's performance and application.
Figure 12: Results of learning from 20 Atari games in terms of IQM.
Thus, the unique characteristics of different tasks highlight the limitations of a one-size-fits-all approach in the context of return-to-go estimation in the Decision Transformer model.
D Architecture Details
We adopt the causal transformer decoder architecture and process the sequences as follows.
- Transformer Blocks: Composed of masked causal attention and a multilayer perceptron (MLP), with layer normalization and residual connections. Activation function used is GELU (Gaussian Error Linear Unit).
- Embedding Layers:
- State Embedding: A linear layer followed by positional (time) embeddings addition.
- Action Embedding: A linear layer followed by positional (time) embeddings addition.
- Return to Go Embedding: A linear layer followed by positional (time) embeddings addition.
- Timestep Embedding: An embedding layer for encoding timesteps.
- Prediction Heads:
- State Prediction: A linear layer taking as input the concatenated action and state embeddings.
- Action Prediction: A linear layer followed by a Tanh activation function.
- Return-to-go Prediction: A linear layer.
Table 6: Hper-parameters. We list all hyper-parameters for D4RL and Atari games below.
Parameter	Setting (D4RL)	Setting (Atari)
Observation down-sampling	-	84 x 84
Frames stacked	-	4
Frames skip	-	4
Terminal on loss of life	-	True
Augmentation (random cropping)	-	True
Augmentation (random rotation)	-	True
Discount factor	1	0.9974
Gradient clipping	True	True
Reward clipping	True	True
Optimizer	AdamW	AdamW
Optimizer (learning rate)	0.0001	0.0001
Optimizer (weight decay)	0.0001	0.0001
Maximum history length	20	30
inverse temperature (κ)	10	10
Expectile level	0.99	0.99
Mix precision	True	True
Evaluation episodes	100	32
Action loss coefficient (Laction)	1	0.01
Return loss coefficient (Lreturn)	0.001	0.001
State loss coefficient (Lstate)	1	1
Return maximizer loss coefficient (Lmax)	0.5	0.5
Number of return output bins	60	120
Observation normalization	True	True
Batch size	256	256
Sample from the top 85th percentile logits (return)	True	True
Sample from the top 85th percentile logits (action)	-	True
Step size (δ)	2	2
GPU	3090 Ti	V100