SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 12: Hyperparameters for all the experiments.
Model	Dropout	Stoch Depth	Augmentations	Optimizer	β1	β2	lr	λ
Train from scratch on ImageNet
ResNet-50	-	-	-	AdamW	0.9	0.999	3e-3	0.1
Lion	0.9	0.99	3e-4	1.0
Mixer-S/16	-	0.1	-	AdamW	0.9	0.999	1e-2	0.3
Lion	0.9	0.99	3e-3	1.0
Mixer-B/16	-	0.1	-	AdamW	0.9	0.999	1e-2	0.3
Lion	0.9	0.99	3e-3	3.0
ViT-S/16	0.1	0.1	-	AdamW	0.9	0.999	1e-2	0.1
Lion	0.9	0.99	1e-3	1.0
-	-	RandAug: 2, 15 Mixup: 0.5	AdamW	0.9	0.999	3e-3	0.1
Lion	0.9	0.99	3e-4	1.0
ViT-B/16	0.1	0.1	-	AdamW	0.9	0.999	3e-3	0.3
Lion	0.9	0.99	1e-3	1.0
-	-	RandAug: 2, 15 Mixup: 0.5	AdamW	0.9	0.999	1e-3	1.0
Lion	0.9	0.99	1e-4	10.0
CoAtNet-1	-	0.3	RandAug: 2, 15 Mixup: 0.8	AdamW	0.9	0.999	1e-3	0.05
Lion	0.9	0.99	2e-4	1.0
CoAtNet-3	-	0.7	RandAug: 2, 15 Mixup: 0.8	AdamW	0.9	0.999	1e-3	0.05
Lion	0.9	0.99	2e-4	1.0
Pre-train on ImageNet-21K
ViT-B/16	0.1	0.1	-	AdamW	0.9	0.999	1e-3	0.1
Lion	0.9	0.99	1e-4	0.3
ViT-L/16	0.1	0.1	-	AdamW	0.9	0.999	1e-3	0.3
Lion	0.9	0.99	1e-4	1.0
Pre-train on JFT
ViT-B/16	-	-	-	AdamW	0.9	0.999	6e-4	0.1
Lion	0.9	0.99	1e-4	0.3
ViT-L/16	-	-	-	AdamW	0.9	0.999	3e-4	0.1
Lion	0.9	0.99	1e-4	0.3
ViT-H/14	-	-	-	AdamW	0.9	0.999	3e-4	0.1
Lion	0.9	0.99	3e-5	0.3
ViT-g/14 & ViT-G/14	-	-	-	Adafactor	0.9	0.999	8e-4	0.03
Lion	0.9	0.99	3e-5	0.3
Vision-language contrastive learning
LiT-B/*-B	-	-	-	AdamW	0.9	0.999	1e-3	-
Lion	0.9	0.99	3e-4	-
LiT-g/14-L	-	-	-	AdamW	0.9	0.999	1e-3	0.1
Lion	0.9	0.99	2e-4	0.5
BASIC-L	-	-	-	Adafactor	0.9	0.999	5e-4	0.01
Lion	0.9	0.99	2e-4	0.1
Diffusion model
Imagen base & super-resolution	-	-	-	AdamW	0.9	0.999	1e-3	-
Lion	0.9	0.99	1e-4	-
Image generation on ImageNet	64 × 64: 0.1128 × 128 & 256 × 256: 0.2	-	-	AdamW	0.9	0.999	3e-4	0.01
Lion	0.9	0.99	3e-5	0.1
Autoregressive & masked language modeling
Small & Medium (PG-19, C4) & Large	-	-	-	AdamW	0.9	0.99	3e-3	-
Lion	0.95	0.98	3e-4	-
Medium (Wiki-40B)	-	-	-	AdamW	0.9	0.99	3e-3	0.001
Lion	0.95	0.98	3e-4	0.01
1.1B & 2.1B	-	-	-	Adafactor	0.9	0.99	2e-3	0.0005
Lion	0.95	0.98	2e-4	0.005
7.5B	-	-	-	Adafactor	0.9	0.99	1e-3	0.001
Lion	0.95	0.98	1e-4	0.01
Language model fine-tuning
T5-Base & Large & 11B	0.1	-	-	AdamW	0.9	0.99	3e-5	-
Lion	0.95	0.98	3e-6	-