ChemBERTa-77M-MLM-finetuned-bace

This model is a fine-tuned version of DeepChem/ChemBERTa-77M-MLM on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.0592

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 50

Training results

Training Loss	Epoch	Step	Validation Loss
No log	1.0	152	0.1396
No log	2.0	304	0.1761
No log	3.0	456	0.1527
0.2304	4.0	608	0.1260
0.2304	5.0	760	0.1307
0.2304	6.0	912	0.1256
0.178	7.0	1064	0.1016
0.178	8.0	1216	0.0890
0.178	9.0	1368	0.1486
0.1588	10.0	1520	0.1047
0.1588	11.0	1672	0.1072
0.1588	12.0	1824	0.1336
0.1588	13.0	1976	0.0955
0.1473	14.0	2128	0.0946
0.1473	15.0	2280	0.0940
0.1473	16.0	2432	0.0756
0.1415	17.0	2584	0.1275
0.1415	18.0	2736	0.1175
0.1415	19.0	2888	0.0743
0.1237	20.0	3040	0.0811
0.1237	21.0	3192	0.1033
0.1237	22.0	3344	0.0877
0.1237	23.0	3496	0.0835
0.1254	24.0	3648	0.1107
0.1254	25.0	3800	0.1098
0.1254	26.0	3952	0.1138
0.1238	27.0	4104	0.0896
0.1238	28.0	4256	0.0923
0.1238	29.0	4408	0.1019
0.1161	30.0	4560	0.1115
0.1161	31.0	4712	0.0716
0.1161	32.0	4864	0.1300
0.114	33.0	5016	0.1017
0.114	34.0	5168	0.0922
0.114	35.0	5320	0.0458
0.114	36.0	5472	0.1335
0.1116	37.0	5624	0.0924
0.1116	38.0	5776	0.0828
0.1116	39.0	5928	0.0857
0.1082	40.0	6080	0.0847
0.1082	41.0	6232	0.0629
0.1082	42.0	6384	0.0685
0.11	43.0	6536	0.0918
0.11	44.0	6688	0.0829
0.11	45.0	6840	0.1118
0.11	46.0	6992	0.0561
0.1041	47.0	7144	0.0834
0.1041	48.0	7296	0.0940
0.1041	49.0	7448	0.1024
0.1103	50.0	7600	0.0809

Framework versions

• Transformers 4.35.0
• Pytorch 2.1.0+cu118
• Datasets 2.14.6
• Tokenizers 0.14.1