sembr2023-distilbert-base-cased

This model is a fine-tuned version of distilbert-base-cased on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.2214
• Precision: 0.7952
• Recall: 0.8261
• F1: 0.8104
• Iou: 0.6812
• Accuracy: 0.9665
• Balanced Accuracy: 0.9030
• Overall Accuracy: 0.9525

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 64
• eval_batch_size: 128
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• training_steps: 1000

Training results

Training Loss	Epoch	Step	Validation Loss	Precision	Recall	F1	Iou	Accuracy	Balanced Accuracy	Overall Accuracy
0.3717	0.06	10	0.3886	0	0.0	0.0	0.0	0.9133	0.5	0.9133
0.3692	0.12	20	0.3552	0	0.0	0.0	0.0	0.9133	0.5	0.9133
0.2579	0.17	30	0.2734	0.8923	0.1841	0.3052	0.1801	0.9274	0.5910	0.9255
0.2135	0.23	40	0.2224	0.7632	0.6267	0.6882	0.5247	0.9508	0.8041	0.9348
0.225	0.29	50	0.1935	0.8426	0.6378	0.7260	0.5699	0.9583	0.8132	0.9427
0.1637	0.35	60	0.1785	0.8115	0.6951	0.7488	0.5985	0.9596	0.8399	0.9470
0.1497	0.4	70	0.1837	0.8159	0.6961	0.7513	0.6016	0.9601	0.8406	0.9434
0.1248	0.46	80	0.1758	0.7920	0.7523	0.7717	0.6282	0.9614	0.8668	0.9446
0.1297	0.52	90	0.1796	0.7740	0.7933	0.7835	0.6441	0.9620	0.8857	0.9430
0.1321	0.58	100	0.1721	0.8616	0.7178	0.7832	0.6436	0.9656	0.8534	0.9496
0.1058	0.64	110	0.1572	0.8132	0.7766	0.7945	0.6591	0.9652	0.8799	0.9494
0.1183	0.69	120	0.1734	0.8084	0.7792	0.7935	0.6578	0.9649	0.8809	0.9470
0.1195	0.75	130	0.1652	0.7753	0.7952	0.7851	0.6462	0.9623	0.8867	0.9463
0.0996	0.81	140	0.1433	0.8292	0.7684	0.7977	0.6634	0.9662	0.8767	0.9527
0.1009	0.87	150	0.1817	0.8181	0.7808	0.7990	0.6653	0.9660	0.8822	0.9480
0.0953	0.92	160	0.1554	0.8669	0.7245	0.7893	0.6519	0.9665	0.8570	0.9524
0.1077	0.98	170	0.1556	0.8261	0.7752	0.7998	0.6664	0.9664	0.8798	0.9512
0.0981	1.04	180	0.1526	0.8283	0.7703	0.7982	0.6642	0.9663	0.8776	0.9520
0.0982	1.1	190	0.1547	0.8001	0.7976	0.7989	0.6651	0.9652	0.8894	0.9504
0.0789	1.16	200	0.1606	0.8135	0.7947	0.8040	0.6722	0.9664	0.8887	0.9513
0.0829	1.21	210	0.1566	0.8244	0.7872	0.8054	0.6741	0.9670	0.8856	0.9517
0.0742	1.27	220	0.1680	0.8167	0.7895	0.8029	0.6707	0.9664	0.8864	0.9506
0.084	1.33	230	0.1680	0.8197	0.7824	0.8006	0.6675	0.9662	0.8830	0.9511
0.0702	1.39	240	0.1653	0.8184	0.7996	0.8089	0.6791	0.9673	0.8914	0.9510
0.0713	1.45	250	0.1675	0.7844	0.8184	0.8010	0.6681	0.9648	0.8985	0.9492
0.0763	1.5	260	0.1501	0.8239	0.7833	0.8031	0.6710	0.9667	0.8837	0.9532
0.0738	1.56	270	0.1518	0.8203	0.7962	0.8081	0.6780	0.9672	0.8898	0.9527
0.0736	1.62	280	0.1624	0.7849	0.8222	0.8031	0.6710	0.9651	0.9004	0.9508
0.0659	1.68	290	0.1735	0.7775	0.8308	0.8033	0.6712	0.9647	0.9041	0.9496
0.0653	1.73	300	0.1586	0.7828	0.8224	0.8022	0.6697	0.9648	0.9004	0.9503
0.0635	1.79	310	0.1720	0.8091	0.8033	0.8062	0.6753	0.9665	0.8927	0.9510
0.0724	1.85	320	0.1588	0.8057	0.8033	0.8045	0.6729	0.9662	0.8925	0.9531
0.0612	1.91	330	0.1818	0.7828	0.8222	0.8020	0.6695	0.9648	0.9003	0.9488
0.0612	1.97	340	0.1704	0.8235	0.7893	0.8060	0.6751	0.9671	0.8866	0.9526
0.0592	2.02	350	0.1634	0.8002	0.7929	0.7965	0.6618	0.9649	0.8870	0.9520
0.0474	2.08	360	0.1835	0.7931	0.8120	0.8025	0.6701	0.9654	0.8960	0.9506
0.0484	2.14	370	0.1790	0.8123	0.7941	0.8031	0.6710	0.9663	0.8883	0.9522
0.0524	2.2	380	0.1812	0.7702	0.8291	0.7985	0.6646	0.9637	0.9028	0.9499
0.052	2.25	390	0.1716	0.8041	0.7964	0.8002	0.6670	0.9655	0.8890	0.9533
0.0443	2.31	400	0.1676	0.8054	0.7976	0.8015	0.6687	0.9658	0.8897	0.9535
0.057	2.37	410	0.1836	0.8028	0.8084	0.8056	0.6745	0.9662	0.8948	0.9507
0.0414	2.43	420	0.1791	0.8049	0.8053	0.8051	0.6737	0.9662	0.8934	0.9527
0.0471	2.49	430	0.1771	0.7964	0.8126	0.8044	0.6728	0.9658	0.8965	0.9527
0.039	2.54	440	0.1773	0.8066	0.8021	0.8043	0.6727	0.9662	0.8919	0.9537
0.0543	2.6	450	0.1855	0.7887	0.8193	0.8037	0.6718	0.9653	0.8992	0.9511
0.0398	2.66	460	0.1959	0.7938	0.8147	0.8041	0.6724	0.9656	0.8973	0.9504
0.0419	2.72	470	0.1944	0.7847	0.8286	0.8060	0.6751	0.9654	0.9035	0.9498
0.0436	2.77	480	0.1869	0.8002	0.8109	0.8055	0.6744	0.9661	0.8958	0.9520
0.0497	2.83	490	0.1850	0.7736	0.8422	0.8065	0.6757	0.9650	0.9094	0.9501
0.0408	2.89	500	0.1883	0.8178	0.7962	0.8068	0.6762	0.9670	0.8897	0.9527
0.0332	2.95	510	0.1883	0.7913	0.8188	0.8048	0.6733	0.9656	0.8991	0.9516
0.0382	3.01	520	0.2008	0.7914	0.8307	0.8106	0.6815	0.9664	0.9049	0.9515
0.047	3.06	530	0.1913	0.8137	0.8013	0.8075	0.6771	0.9669	0.8920	0.9522
0.0327	3.12	540	0.1969	0.7993	0.8168	0.8080	0.6778	0.9664	0.8987	0.9518
0.0338	3.18	550	0.1989	0.7962	0.8173	0.8066	0.6759	0.9660	0.8987	0.9518
0.0332	3.24	560	0.2004	0.7999	0.8178	0.8087	0.6789	0.9665	0.8992	0.9518
0.0308	3.29	570	0.1964	0.8126	0.8092	0.8109	0.6819	0.9673	0.8957	0.9537
0.0348	3.35	580	0.2032	0.7902	0.8239	0.8067	0.6761	0.9658	0.9016	0.9515
0.0351	3.41	590	0.2064	0.7855	0.8218	0.8032	0.6712	0.9651	0.9003	0.9511
0.0301	3.47	600	0.2118	0.7872	0.8265	0.8063	0.6755	0.9656	0.9026	0.9505
0.0261	3.53	610	0.1997	0.7991	0.8194	0.8091	0.6794	0.9665	0.8999	0.9522
0.0282	3.58	620	0.1950	0.8029	0.8114	0.8071	0.6766	0.9664	0.8962	0.9527
0.0326	3.64	630	0.2038	0.7873	0.8290	0.8076	0.6773	0.9658	0.9039	0.9516
0.0353	3.7	640	0.2010	0.7930	0.8228	0.8076	0.6773	0.9660	0.9012	0.9514
0.0348	3.76	650	0.2043	0.7949	0.8243	0.8093	0.6797	0.9663	0.9021	0.9519
0.0296	3.82	660	0.2050	0.7976	0.8226	0.8099	0.6805	0.9665	0.9014	0.9529
0.0287	3.87	670	0.2158	0.7820	0.8318	0.8061	0.6752	0.9653	0.9049	0.9504
0.024	3.93	680	0.2110	0.7847	0.8294	0.8065	0.6757	0.9655	0.9039	0.9512
0.0274	3.99	690	0.2075	0.7937	0.8254	0.8092	0.6796	0.9663	0.9025	0.9523
0.0247	4.05	700	0.2130	0.7995	0.8210	0.8101	0.6808	0.9666	0.9007	0.9525
0.0202	4.1	710	0.2142	0.7955	0.8215	0.8083	0.6782	0.9662	0.9007	0.9518
0.0245	4.16	720	0.2120	0.7965	0.8195	0.8078	0.6776	0.9662	0.8998	0.9516
0.0214	4.22	730	0.2151	0.7899	0.8256	0.8074	0.6770	0.9659	0.9024	0.9515
0.0202	4.28	740	0.2145	0.7963	0.8220	0.8089	0.6792	0.9664	0.9010	0.9520
0.0257	4.34	750	0.2181	0.7960	0.8217	0.8087	0.6788	0.9663	0.9009	0.9520
0.0271	4.39	760	0.2151	0.7953	0.8232	0.8090	0.6793	0.9663	0.9015	0.9518
0.0279	4.45	770	0.2196	0.7955	0.8237	0.8094	0.6798	0.9664	0.9018	0.9521
0.0273	4.51	780	0.2194	0.7984	0.8256	0.8118	0.6832	0.9668	0.9029	0.9523
0.018	4.57	790	0.2201	0.7985	0.8247	0.8114	0.6826	0.9668	0.9025	0.9526
0.0275	4.62	800	0.2204	0.7893	0.8358	0.8119	0.6834	0.9664	0.9073	0.9519
0.0198	4.68	810	0.2160	0.7983	0.8232	0.8105	0.6814	0.9666	0.9017	0.9526
0.019	4.74	820	0.2109	0.7961	0.8243	0.8100	0.6806	0.9665	0.9021	0.9527
0.0236	4.8	830	0.2208	0.7956	0.8238	0.8094	0.6799	0.9664	0.9019	0.9521
0.0177	4.86	840	0.2217	0.7900	0.8301	0.8095	0.6800	0.9661	0.9046	0.9519
0.0209	4.91	850	0.2226	0.7927	0.8285	0.8102	0.6810	0.9664	0.9040	0.9522
0.0241	4.97	860	0.2215	0.7915	0.8276	0.8091	0.6794	0.9662	0.9035	0.9521
0.0211	5.03	870	0.2181	0.7957	0.8242	0.8097	0.6802	0.9664	0.9020	0.9525
0.0234	5.09	880	0.2171	0.7975	0.8224	0.8098	0.6803	0.9665	0.9013	0.9526
0.0201	5.14	890	0.2191	0.7925	0.8265	0.8092	0.6795	0.9662	0.9030	0.9523
0.0211	5.2	900	0.2175	0.7957	0.8238	0.8095	0.6799	0.9664	0.9019	0.9526
0.0234	5.26	910	0.2207	0.7913	0.8291	0.8097	0.6803	0.9662	0.9042	0.9522
0.023	5.32	920	0.2202	0.7965	0.8234	0.8098	0.6803	0.9665	0.9017	0.9524
0.0192	5.38	930	0.2203	0.7969	0.8239	0.8102	0.6809	0.9665	0.9020	0.9525
0.0217	5.43	940	0.2206	0.7956	0.8255	0.8103	0.6811	0.9665	0.9027	0.9524
0.0195	5.49	950	0.2213	0.7953	0.8259	0.8103	0.6811	0.9665	0.9029	0.9524
0.0285	5.55	960	0.2214	0.7955	0.8254	0.8102	0.6809	0.9665	0.9026	0.9524
0.0263	5.61	970	0.2213	0.7955	0.8254	0.8102	0.6809	0.9665	0.9026	0.9524
0.02	5.66	980	0.2214	0.7951	0.8258	0.8101	0.6809	0.9665	0.9028	0.9524
0.021	5.72	990	0.2214	0.7952	0.8261	0.8104	0.6812	0.9665	0.9030	0.9525
0.0233	5.78	1000	0.2214	0.7952	0.8261	0.8104	0.6812	0.9665	0.9030	0.9525

Framework versions

• Transformers 4.34.1
• Pytorch 2.0.1
• Datasets 2.14.6
• Tokenizers 0.14.1