dna_bert_3_1000seq-finetuned

This model is a fine-tuned version of armheb/DNA_bert_3 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.4684

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 100
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
0.8607	1.0	62	0.6257
0.6177	2.0	124	0.6120
0.6098	3.0	186	0.6062
0.604	4.0	248	0.6052
0.5999	5.0	310	0.6040
0.5982	6.0	372	0.5996
0.5985	7.0	434	0.5985
0.5956	8.0	496	0.5968
0.5936	9.0	558	0.5950
0.5908	10.0	620	0.5941
0.5904	11.0	682	0.5932
0.59	12.0	744	0.5917
0.5877	13.0	806	0.5921
0.5847	14.0	868	0.5903
0.5831	15.0	930	0.5887
0.5852	16.0	992	0.5878
0.5805	17.0	1054	0.5872
0.5795	18.0	1116	0.5853
0.5754	19.0	1178	0.5869
0.5757	20.0	1240	0.5839
0.5722	21.0	1302	0.5831
0.5693	22.0	1364	0.5811
0.5667	23.0	1426	0.5802
0.5652	24.0	1488	0.5775
0.5608	25.0	1550	0.5788
0.5591	26.0	1612	0.5724
0.5538	27.0	1674	0.5736
0.552	28.0	1736	0.5689
0.5483	29.0	1798	0.5689
0.5442	30.0	1860	0.5671
0.5405	31.0	1922	0.5658
0.537	32.0	1984	0.5605
0.5349	33.0	2046	0.5575
0.5275	34.0	2108	0.5569
0.5227	35.0	2170	0.5537
0.52	36.0	2232	0.5509
0.5173	37.0	2294	0.5504
0.5123	38.0	2356	0.5435
0.5088	39.0	2418	0.5472
0.5037	40.0	2480	0.5383
0.501	41.0	2542	0.5379
0.4931	42.0	2604	0.5365
0.4923	43.0	2666	0.5328
0.4879	44.0	2728	0.5301
0.482	45.0	2790	0.5295
0.4805	46.0	2852	0.5261
0.4772	47.0	2914	0.5221
0.4738	48.0	2976	0.5234
0.4674	49.0	3038	0.5210
0.4646	50.0	3100	0.5169
0.4621	51.0	3162	0.5142
0.4574	52.0	3224	0.5129
0.4552	53.0	3286	0.5127
0.4539	54.0	3348	0.5124
0.4506	55.0	3410	0.5076
0.4457	56.0	3472	0.5082
0.4454	57.0	3534	0.5027
0.4398	58.0	3596	0.5019
0.4386	59.0	3658	0.4998
0.4332	60.0	3720	0.4970
0.4277	61.0	3782	0.4995
0.4273	62.0	3844	0.4962
0.4235	63.0	3906	0.4909
0.4201	64.0	3968	0.4913
0.4198	65.0	4030	0.4899
0.4182	66.0	4092	0.4919
0.4157	67.0	4154	0.4902
0.4104	68.0	4216	0.4881
0.4095	69.0	4278	0.4881
0.4077	70.0	4340	0.4861
0.4064	71.0	4402	0.4868
0.4041	72.0	4464	0.4826
0.4029	73.0	4526	0.4833
0.3976	74.0	4588	0.4819
0.3997	75.0	4650	0.4809
0.3974	76.0	4712	0.4801
0.3953	77.0	4774	0.4783
0.3938	78.0	4836	0.4775
0.3934	79.0	4898	0.4762
0.3923	80.0	4960	0.4742
0.3893	81.0	5022	0.4742
0.3909	82.0	5084	0.4740
0.3856	83.0	5146	0.4739
0.3904	84.0	5208	0.4740
0.3883	85.0	5270	0.4701
0.3865	86.0	5332	0.4727
0.3809	87.0	5394	0.4736
0.3853	88.0	5456	0.4704
0.3821	89.0	5518	0.4704
0.3809	90.0	5580	0.4701
0.3814	91.0	5642	0.4698
0.3795	92.0	5704	0.4702
0.3804	93.0	5766	0.4692
0.377	94.0	5828	0.4683
0.3812	95.0	5890	0.4692
0.3806	96.0	5952	0.4683
0.3745	97.0	6014	0.4690
0.3825	98.0	6076	0.4684
0.374	99.0	6138	0.4687
0.3795	100.0	6200	0.4684

Framework versions

• Transformers 4.21.1
• Pytorch 1.12.0+cu113
• Datasets 2.4.0
• Tokenizers 0.12.1