SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We employ a 2-hidden-layer neural network with a hidden size of 512 and ReLU activation as the classifier. To extract features from images, we utilize the pretrained ResNet-18 model (trained on ImageNet), which has a feature dimension of 512. The ResNet-18 model can be accessed at https://pytorch.org/vision/main/models/generated/torchvision.models.resnet18.html. We opted for this approach as training such a large convolutional neural network directly on the DomainNet dataset proved to be ineffective. The training objective is to minimize the cross-entropy loss. To optimize the network parameters, we use the stochastic gradient descent (SGD) optimizer with a learning rate of 0.01 and momentum of 0.9. Additionally, we apply
${}^{9}$  Loading 200M frames requires over 500GB memory.
weight decay with a coefficient of 0.0005. The models are trained for 100 epochs with a batch size of 100, following the standard practice.
The discriminators used in ISW-BC and DWBC are implemented as 2-hidden-layer neural networks with ReLU activation. It's important to note that these discriminators take both the image and label as inputs. The image input is processed by the pre-trained and fixed ResNet-18, while the label input is projected to the same dimension (512) by a random projection matrix. The hidden size for the discriminator is set to 1024 for ISW-BC and 1025 for DWBC, as the discriminator in DWBC also takes the log-likelihood as an input. For ISW-BC, the discriminator is trained independently for 100 epochs with the same optimization configuration as the classifier. Afterward, the discriminator is fixed, and its output is used to compute the importance sampling ratio, which is then used to train the classifier.
H.2 Additional Results
H.2.1 Training Curves
The training curves on the robotic locomotion control tasks are displayed in Figure 6, Figure 7 and Figure 8. The training curves on Atari video games are displayed in Figure 9 and Figure 10. The training curves on the image classification task are displayed in Figure 11.
Figure 6: Training curves of algorithms on the locomotion control task in the full replay setting. Solid lines correspond to the mean performance and shaded regions correspond to the  $95\%$  confidence interval. Same as other figures.
H.2.2 Ablation Study
In this section, we present ablation studies conducted on Atari games, aiming to provide insights into the underlying working scheme of our method. We specifically emphasize Atari games due to their high-dimensional image inputs, making these tasks particularly challenging. In contrast, the other two tasks, locomotion control and image classification, involve informative vector inputs, setting them apart from the unique characteristics of Atari games.
Ablation Study on Feature Representations of Discriminator Network. Our study reveals that employing a separate CNN for the discriminator yields inferior results compared to utilizing the feature extractor of the policy network. Please refer to Figure 12. Our conjecture is that training the discriminator independently may cause it to fit noise information (e.g., background). In contrast, the policy CNN network is capable of learning decision-centric information, enabling an effective approach to building the discriminator network through the feature extractor of the policy network.
Figure 7: Training curves of algorithms on the locomotion control task in the noisy expert setting.
Figure 8: Training curves of algorithms on the locomotion control task in the expert&random setting.
Figure 9: Training curves of algorithms on the Atari games in the full replay setting.
Figure 10: Training curves of algorithms on the Atari games in the noisy expert setting.
Figure 11: Training curves of algorithms on the image classification task using the DomainNet dataset.
Figure 12: Training curves of ISW-BC on the Atari games in the full replay setting. We test the performance with different feature extractors of the discriminator.
Ablation Study on Depth of Discriminator Network. We have discovered that the number of discriminator layers plays a crucial role in the performance of Atari games. The training curves, depicted in both Figure 13 and Figure 14, illustrate the performance variation based on the number of layers in the discriminator network. Notably, a 1-hidden-layer neural network yields the best results for the full replay setting, while a 3-hidden-layer neural network performs optimally in the noisy expert setting. It is important to note that this phenomenon is specific to Atari games. We do not have a good explanation yet. We believe this deserves further investigation in the future work.
Figure 13: Training curves of ISW-BC on the Atari games in the full replay setting. We test the performance with different number of layers for the discriminator network.
H.2.3 Results on the Mix of Expert and Random Data
In the main text, we consider two types of supplementary datasets: full replay and noisy expert. Here we consider another type of supplementary datasets which consists of expert trajectories and trajectories generated by a random policy [25, 47]. We evaluate all methods on this new type of supplementary datasets on the robotic locomotion control tasks. In particular, the supplementary dataset contains 10 expert trajectories and 10 random trajectories. We report the experimental results on Table 5. The results show that ISW-BC outperforms NBCU significantly, demonstrating
Figure 14: Training curves of ISW-BC on the Atari games with the noisy expert setting. We test the performance with different number of layers for the discriminator network.
the robustness of ISW-BC to distribution shift. Furthermore, ISW-BC also performs better than DemoDICE and DWBC.
Table 5: Environment return of algorithms on the mixture of expert and random dataset on 4 locomotion control tasks
Ant	HalfCheetah	Hopper	Walker	Avg
Random	-326	-280	-20	2	0%
Expert	5229	11115	3589	5082	100%
BC	1759±287	931±273	2468±164	1738±311	38%
NBCU	4316±215	-14±0	3401±68	1189±100	51%
DemoDICE	4097±237	7401±356	3556±27	4451±141	83%
DWBC	3447±226	5307±319	3286±32	1451±118	59%
ISW-BC	4467±134	8697±365	3584±6	4528±226	88%