SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We also note that although our objective function has favorable computational complexity compared to existing methods, the evaluation of our objective is not straightforward to distribute across machines. This is because we need to compute the SVD over the centroid matrix, which requires gathering the outputs of a distributed forward pass onto a single machine. Efficient methods for the distributed computation of the SVD could help alleviate this issue in the future.
O Impact of Momentum Encoder
With all other settings fixed, we find that the use of a momentum encoder confers a small advantage in terms of classification performance. Below we report frozen linear evaluation accuracy for MMCR training 2, 4, and 8 views with and without a momentum encoder.
2 Views	4 Views	8 Views
With Momentum Encoder	69.5	71.4	72.1
Without Momentum Encoder	68.4	70.2	71.5
Table 7: Shown are Top-1 accuracies under the linear evaluation protocol for representations trained with or without a momentum encoder using the settings described above. It appears that as the number of views increases the added diversity of positive sample representations conferred by the momentum encoder diminishes.
P MMCR Benefits from Longer Pretraining
To verify that MMCR benefits from longer pretraining we modified our setup to pretrain with two views and reduced our base learning rate to 0.4 and train for 1000 epochs. We additionally changed the hyperparameters for pretraining the linear classifier when evaluating this model (the classifier was trained with an learning rate of 0.3 and batch size of 2048 for 100 epochs). We did not perform any hyperparameter tuning for this setting, so these results represent a performance floor for 2-view MMCR.
Method	Top-1 Accuracy
SimCLR	69.3
MoCov2	71.1
SimSiam	71.3
SwAV (without multicrop)	71.3
MMCR (2 views)	73.0
Barlow Twins	73.2
BYOL	74.3
SwAV (with multicrop)	75.3
NNCLR	75.6
ReLUCV2 (69)	77.1
Table 8: Shown are Top-1 accuracies under the linear evaluation protocol for representations trained for 1000 epochs using various self supervised learning frameworks.