SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 4: VGG Architecture.
Layer Type	Hyperparameters	Activation Function	Dropout rate
Conv2d	Input Channels:3 ; Output Channels:64 ; Kernel Size:3 ; Stride:1 ; Padding:1	BN & ReLU	0.3
Conv2d	Input Channels:64 ; Output Channels:64 ; Kernel Size:3 ; Stride:1 ; Padding:1	BN & ReLU	0
MaxPool2d	Kernel Size:2 ; Padding:2
Conv2d	Input Channels:64 ; Output Channels:128 ; Kernel Size:3 ; Stride:1 ; Padding:1	BN & ReLU	0.4
Conv2d	Input Channels:128 ; Output Channels:128 ; Kernel Size:3 ; Stride:1 ; Padding:1	BN & ReLU	0
MaxPool2d	Kernel Size:2 ; Padding:2
Conv2d	Input Channels:128 ; Output Channels:256 ; Kernel Size:3 ; Stride:1 ; Padding:1	BN & ReLU	0.4
Conv2d	Input Channels:256 ; Output Channels:256 ; Kernel Size:3 ; Stride:1 ; Padding:1	BN & ReLU	0.4
Conv2d	Input Channels:256 ; Output Channels:256 ; Kernel Size:3 ; Stride:1 ; Padding:1	BN & ReLU	0
MaxPool2d	Kernel Size:2 ; Padding:2
Conv2d	Input Channels:256 ; Output Channels:512 ; Kernel Size:3 ; Stride:1 ; Padding:1	BN & ReLU	0.4
Conv2d	Input Channels:512 ; Output Channels:512 ; Kernel Size:3 ; Stride:1 ; Padding:1	BN & ReLU	0.4
Conv2d	Input Channels:512 ; Output Channels:512 ; Kernel Size:3 ; Stride:1 ; Padding:1	BN & ReLU	0
MaxPool2d	Kernel Size:2 ; Padding:2
Conv2d	Input Channels:512 ; Output Channels:512 ; Kernel Size:3 ; Stride:1 ; Padding:1	BN & ReLU	0.4
Conv2d	Input Channels:512 ; Output Channels:512 ; Kernel Size:3 ; Stride:1 ; Padding:1	BN & ReLU	0.4
Conv2d	Input Channels:512 ; Output Channels:512 ; Kernel Size:3 ;stride:1 ; Padding:1	BN & ReLU	0
MaxPool2d	Kernel Size:2 ; Padding:2	NA
Flatten	Start Dimension:1	0.5
Linear	Input Dimension: 512 ; Output Dimension: 512	BN & ReLU	0.5
Linear	Input Dimension: 512 ; Output Dimension: 10	0
Figure 16: VGG ensembles on SVHN data. The same phenomenon emerges when we consider an alternative VGG style architecture on the SVHN dataset.
G.7 Learner dropout as a resolution
In this section we provide a negative result in which we attempt to resolve learner collusion by randomly removing a subset of base learners from the ensemble at each batch during training. The hypothesis is that if we drop a sufficiently large portion of base learners but still perform joint training on the remainder, this should remove the ability to deterministically collude and, therefore, cause inflated diversity to be actively harmful even on the training data. One might hope that this would cause the ensemble to avoid this degeneracy. In what follows we provide the details of this experiment.
We repeat the setup on CIFAR-10 with ResNet-18 architecture as described earlier in the paper. We train each model using the joint objective, but at each batch we drop a proportion  $p \in [0, 0.2, 0.4, 0.6]$  of randomly selected learners from the ensemble. We then investigate if this (a) reduces collusion and (b) improves ensemble performance. The results of this experiment are included in Figure 18. We find that dropping learners does indeed significantly reduce the diversity score indicating a reduction in learner collusion (although a reasonably large proportion of learners is required to be dropped). Unfortunately, the improvement in performance by reducing collusion is negated by a decrease in individual performance due to the base learners becoming weaker on average. The reason why is apparent once we notice that by dropping a base learner from some proportion of batches, this is exactly equivalent to bootstrapping. While bootstrapping has been effective for ensembles of weak learners (e.g. random forest) it has already been shown to be harmful for deep ens
Figure 17: Loss plots. Corresponding ensemble loss (ERR) for the results presented in Figure 6 (left).
bles [12]. Unfortunately, this resolution appears to only reduce the effect of learner collusion by simultaneously harming the performance of the ensemble due to bootstrapping.
Proposing methods to overcome learner collusion is a valuable but non-trivial direction for future work that we and, we hope, others in the research community will pursue. We believe that the comprehensive diagnosis of the issue we have presented in this work will provide a foundation for future methodological progress.
Figure 18: Investigating learner dropout. We evaluate randomly dropping a subset of learners during training to remedy the learner collusion effect. We repeat the setup on CIFAR-10 with ResNet-18 architecture from Figure 6 with  $\beta = 1$  but at each training batch, we drop a proportion of learners in  $[0.0, 0.2, 0.4, 0.6]$ . We find that, while this seemingly does reduce learner collusion, the benefit is negated by a decrease in individual performance due to base learners becoming weaker on average. Results are reported over 5 runs.