bert-base-uncased-gpqa

This model is a fine-tuned version of google-bert/bert-base-uncased on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.3847
• Accuracy: 0.8939

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 16
• eval_batch_size: 16
• seed: 321
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
No log	1.0	13	1.3835	0.2525
No log	2.0	26	1.3694	0.3838
No log	3.0	39	1.3625	0.3838
No log	4.0	52	1.3641	0.3485
No log	5.0	65	1.3295	0.5404
No log	6.0	78	1.3407	0.4949
No log	7.0	91	1.2322	0.6162
No log	8.0	104	1.1000	0.5758
No log	9.0	117	1.1184	0.6717
No log	10.0	130	0.8604	0.7525
No log	11.0	143	0.7279	0.7727
No log	12.0	156	0.6254	0.7929
No log	13.0	169	0.5025	0.8636
No log	14.0	182	0.8694	0.7071
No log	15.0	195	0.4373	0.8535
No log	16.0	208	0.4071	0.9141
No log	17.0	221	0.6762	0.8131
No log	18.0	234	0.6295	0.8333
No log	19.0	247	0.3242	0.8687
No log	20.0	260	0.3847	0.8939

Framework versions

• Transformers 4.34.0.dev0
• Pytorch 2.0.1+cu117
• Datasets 2.14.5
• Tokenizers 0.14.0