SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 9: ImageNet linear probing settings. Settings in the first section were grid-searched for each model, settings in the last section are unexplored good defaults.
Linear probe at 224 px
ViT-L/16	SoViT-400m/14	ViT-g/14
Learning rate	0.001	0.0003	0.001
Weight decay	0.0001	-	-
Training duration	24.7 k steps (10 epochs)
Image crop	resize(256) |randomCrop(224)
Random augment	-
Mixup	0.1
Learning rate decay	-
Polyak averaging (EMA)	-
Optimizer	SGD with 0.9 Momentum
Gradient clipping	-
Batch size	512
Learning rate schedule	Cosine with 10% linear warmup
Random flip	Horizontal
Loss	Sigmoid cross-entropy [6]
Head init	kernel=0, bias=-6.9
Train and minival splits	train[:99%] and train[99%:]