beit-base-patch16-224-dmae-va-U5-42B

This model is a fine-tuned version of microsoft/beit-base-patch16-224 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.1619
• Accuracy: 0.5167

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0015
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 42

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
No log	0.9	7	1.2389	0.45
1.5474	1.94	15	1.2573	0.4667
1.4502	2.97	23	1.4267	0.2667
1.343	4.0	31	1.3320	0.1833
1.343	4.9	38	1.3385	0.4
1.2875	5.94	46	1.3633	0.3167
1.3121	6.97	54	1.4097	0.2
1.3669	8.0	62	1.3222	0.3167
1.3669	8.9	69	1.2321	0.4333
1.294	9.94	77	1.2933	0.4
1.2646	10.97	85	1.2691	0.4
1.2352	12.0	93	1.2749	0.4333
1.1953	12.9	100	1.1975	0.4833
1.1953	13.94	108	1.3215	0.45
1.2519	14.97	116	1.2561	0.3833
1.1971	16.0	124	1.3446	0.3667
1.204	16.9	131	1.2449	0.35
1.204	17.94	139	1.2131	0.4333
1.1334	18.97	147	1.2282	0.3667
1.1345	20.0	155	1.2902	0.3
1.1168	20.9	162	1.1619	0.5167
1.0866	21.94	170	1.1868	0.4667
1.0866	22.97	178	1.2290	0.4333
1.0751	24.0	186	1.1980	0.4167
1.0122	24.9	193	1.1744	0.45
0.9719	25.94	201	1.2401	0.4
0.9719	26.97	209	1.1908	0.4333
0.9418	28.0	217	1.2830	0.4167
0.8875	28.9	224	1.2310	0.4333
0.8961	29.94	232	1.2579	0.4667
0.7788	30.97	240	1.4248	0.4833
0.7788	32.0	248	1.3387	0.45
0.7828	32.9	255	1.3085	0.4667
0.696	33.94	263	1.3884	0.4833
0.6859	34.97	271	1.3446	0.5
0.6859	36.0	279	1.4850	0.45
0.6036	36.9	286	1.4519	0.4667
0.5565	37.94	294	1.4544	0.4667

Framework versions

• Transformers 4.38.2
• Pytorch 2.2.1+cu121
• Datasets 2.19.0
• Tokenizers 0.15.2