SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table F.3: Hyperparameters of VGG16 with the shifted Tanh
Convention	Swap
CIFAR-10	CIFAR-100	Tiny ImageNet	ImageNet	CIFAR-10	CIFAR-100	Tiny ImageNet	ImageNet
Training Epochs	200	200	200	90	200	200	200	90
Learning Rate	0.1	0.1	0.1	0.1	0.1	0.1	0.1	0.1
Learning Rate Drop	100, 150	100, 150	100, 150	30, 60	100, 150	100, 150	100, 150	30, 60
Weight Decay	0.0001	0.0005	0.0001	0.0001	0.0005	0.0005	0.0005	0.0001
Batch Size	128	128	128	256	128	128	128	256
Table F.4: Hyperparameters of VGG16 with the ReLU
Convention	Swap
CIFAR-10	CIFAR-100	Tiny ImageNet	ImageNet	CIFAR-10	CIFAR-100	Tiny ImageNet	ImageNet
Training Epochs	200	200	200	90	200	200	200	90
Learning Rate	0.01	0.01	0.1	0.1	0.01	0.01	0.01	0.01
Learning Rate Drop	100, 150	100, 150	100, 150	30, 60	100, 150	100, 150	100, 150	30, 60
Weight Decay	0.001	0.005	0.0001	0.0001	0.001	0.005	0.001	0.0005
Batch Size	128	128	128	256	128	128	128	256
E.2 MobileNet
Table F.5: Hyperparameters for MobileNet with Tanh
Convention	Swap
CIFAR-10	CIFAR-100	Tiny ImageNet	ImageNet	CIFAR-10	CIFAR-100	Tiny ImageNet	ImageNet
Training Epochs	200	200	200	90	200	200	200	90
Learning Rate	0.1	0.1	0.01	0.1	0.1	0.1	0.01	0.1
Learning Rate Drop	100, 150	100, 150	100, 150	30, 60	100, 150	100, 150	100, 150	30, 60
Weight Decay	0.0001	0.0005	0.005	0.0001	0.0001	0.0005	0.005	0.0001
Batch Size	128	128	128	256	128	128	128	256
Table F.6: Hyperparameters for MobileNet with the Shifted Tanh
Convention	Swap
CIFAR-10	CIFAR-100	Tiny ImageNet	ImageNet	CIFAR-10	CIFAR-100	Tiny ImageNet	ImageNet
Training Epochs	200	200	200	90	200	200	200	90
Learning Rate	0.1	0.1	0.01	0.1	0.1	0.1	0.01	0.01
Learning Rate Drop	100, 150	100, 150	100, 150	30, 60	100, 150	100, 150	100, 150	30, 60
Weight Decay	0.0001	0.0005	0.005	0.0001	0.0005	0.0005	0.005	0.0005
Batch Size	128	128	128	256	128	128	128	256
Table F.7: Hyperparameters for MobileNet with ReLU
Convention	Swap
CIFAR-10	CIFAR-100	Tiny ImageNet	ImageNet	CIFAR-10	CIFAR-100	Tiny ImageNet	ImageNet
Training Epochs	200	200	200	90	200	200	200	90
Learning Rate	0.01	0.01	0.1	0.01	0.01	0.01	0.1	0.1
Learning Rate Drop	100, 150	100, 150	100, 150	30, 60	100, 150	100, 150	100, 150	30, 60
Weight Decay	0.001	0.005	0.0001	0.0001	0.001	0.005	0.001	0.0001
Batch Size	128	128	128	256	128	128	128	256
E.3 PreAct-ResNet
Table F.8: Hyperparameters for PreAct-ResNet18 and PreAct-ResNet34 with Tanh. PreAct-ResNet18 is for the CIFAR dataset, and PreAct-ResNet34 is for the Tiny ImageNet dataset.
Convention	Swap
CIFAR-10	CIFAR-100	Tiny ImageNet	CIFAR-10	CIFAR-100	Tiny ImageNet
Training Epochs	200	200	200	200	200	200
Learning Rate	0.1	0.1	0.1	0.1	0.1	0.1
Learning Rate Drop	100, 150	100, 150	100, 150	100, 150	100, 150	100, 150
Weight Decay	0.0001	0.0001	0.0001	0.0005	0.0005	0.0005
Batch Size	128	128	128	128	128	128
Table F.9: Hyperparameters for PreAct-ResNet18 and PreAct-ResNet34 with the Shifted Tanh. PreAct-ResNet18 is for the CIFAR dataset, and PreAct-ResNet34 is for the Tiny ImageNet dataset.
Convention	Swap
CIFAR-10	CIFAR-100	Tiny ImageNet	CIFAR-10	CIFAR-100	Tiny ImageNet
Training Epochs	200	200	200	200	200	200
Learning Rate	0.1	0.1	0.01	0.1	0.1	0.1
Learning Rate Drop	100, 150	100, 150	100, 150	100, 150	100, 150	100, 150
Weight Decay	0.0001	0.0005	0.001	0.0005	0.0005	0.0005
Batch Size	128	128	128	128	128	128
Table F.10: Hyperparameters for PreAct-ResNet18 and PreAct-ResNet34 with ReLU. PreAct-ResNet18 is for the CIFAR dataset, and PreAct-ResNet34 is for the Tiny ImageNet dataset.
Convention	Swap
CIFAR-10	CIFAR-100	Tiny ImageNet	CIFAR-10	CIFAR-100	Tiny ImageNet
Training Epochs	200	200	200	200	200	200
Learning Rate	0.01	0.01	0.1	0.01	0.01	0.1
Learning Rate Drop	100, 150	100, 150	100, 150	100, 150	100, 150	100, 150
Weight Decay	0.005	0.005	0.0005	0.005	0.005	0.0005
Batch Size	128	128	128	128	128	128