BramVanroy/mbart-en-to-amr

MBART to "translate" English to AMR

To be used with MBART text-to-AMR pipeline (not yet available). Won't work with the default MBartTokenizer so you have to be patient until the pipeline can become available. :-)

This model is a fine-tuned version of facebook/mbart-large-cc25 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.2811
• Accuracy: 0.9361
• Bleu: 73.0667
• Smatch Precision: 0.8451
• Smatch Recall: 0.9039
• Smatch Fscore: 0.8735

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 8
• total_train_batch_size: 128
• total_eval_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 15

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	Bleu	Smatch Precision	Smatch Recall	Smatch Fscore	Ratio Invalid Amrs
1.5295	0.23	100	1.2218	0.7536	55.6802	0.2551	0.6184	0.3612	86.6434
0.978	0.46	200	0.7306	0.8311	63.4299	0.3785	0.7182	0.4957	87.0499
0.7187	0.69	300	0.5882	0.8484	66.1354	0.5547	0.8481	0.6707	93.9605
0.599	0.92	400	0.4560	0.8820	68.3221	0.6594	0.8046	0.7248	89.7213
0.4842	1.15	500	0.3876	0.8904	69.5185	0.7491	0.8648	0.8028	92.2764
0.4311	1.38	600	0.3653	0.8913	69.7931	0.7589	0.8922	0.8202	92.6249
0.4171	1.61	700	0.3244	0.9084	70.8940	0.7759	0.8615	0.8165	90.0697
0.3769	1.84	800	0.2995	0.9128	71.0933	0.7824	0.8688	0.8234	89.6051
0.3121	2.07	900	0.2869	0.9177	71.3434	0.8014	0.8831	0.8403	89.8955
0.2843	2.3	1000	0.2755	0.9190	71.2255	0.8021	0.8766	0.8377	89.3148
0.3013	2.53	1100	0.2876	0.9140	70.8907	0.7391	0.8503	0.7908	86.0627
0.3192	2.76	1200	0.2674	0.9206	71.4303	0.8179	0.8991	0.8566	89.7793
0.3032	2.99	1300	0.2597	0.9230	71.6003	0.7791	0.8794	0.8262	89.1405
0.2367	3.23	1400	0.2933	0.9148	71.7204	0.8318	0.8935	0.8615	91.4634
0.247	3.46	1500	0.2505	0.9272	72.1396	0.8224	0.8889	0.8543	89.0244
0.2326	3.69	1600	0.2467	0.9284	72.0828	0.8257	0.8992	0.8609	88.8502
0.2622	3.92	1700	0.2590	0.9236	71.9205	0.8231	0.8902	0.8553	90.0697
0.1935	4.15	1800	0.2528	0.9281	72.0722	0.8523	0.9075	0.8790	88.7340
0.2067	4.38	1900	0.2480	0.9287	72.2628	0.8322	0.9062	0.8677	89.1405
0.2248	4.61	2000	0.2520	0.9273	72.4493	0.8474	0.9023	0.8740	89.5470
0.2049	4.84	2100	0.2403	0.9316	72.3463	0.8231	0.8998	0.8598	88.0952
0.1942	5.07	2200	0.2482	0.9314	72.4402	0.8291	0.8987	0.8625	89.6051
0.1796	5.3	2300	0.2587	0.9319	72.6028	0.8349	0.8955	0.8642	88.0952
0.1852	5.53	2400	0.2550	0.9316	72.4129	0.8435	0.9002	0.8710	88.4437
0.1898	5.76	2500	0.2493	0.9321	72.5551	0.8269	0.8957	0.8599	87.7468
0.1861	5.99	2600	0.2459	0.9314	72.4291	0.8565	0.9085	0.8817	88.2695
0.1568	6.22	2700	0.2487	0.9321	72.5308	0.8582	0.9122	0.8844	88.1533
0.1491	6.45	2800	0.2461	0.9331	72.6714	0.8632	0.9154	0.8885	88.5598
0.1437	6.68	2900	0.2434	0.9330	72.6621	0.8699	0.9097	0.8893	88.2695
0.1504	6.91	3000	0.2496	0.9341	72.7762	0.8544	0.9021	0.8776	87.5726
0.1313	7.14	3100	0.2510	0.9339	72.7713	0.8674	0.9048	0.8857	88.0372
0.1501	7.37	3200	0.2502	0.9343	72.7488	0.8633	0.9016	0.8820	88.3275
0.1295	7.6	3300	0.2459	0.9348	72.6939	0.8365	0.8969	0.8657	87.9210
0.1262	7.83	3400	0.2524	0.9318	72.8235	0.8509	0.9077	0.8784	88.3275
0.1072	8.06	3500	0.2551	0.9346	72.7323	0.8566	0.9022	0.8788	88.2695
0.1198	8.29	3600	0.2549	0.9350	72.8186	0.8638	0.9099	0.8862	88.0372
0.1175	8.52	3700	0.2581	0.9331	72.6339	0.8624	0.9054	0.8834	88.5598
0.1173	8.75	3800	0.2508	0.9357	73.1089	0.8515	0.9057	0.8778	87.9791
0.1208	8.98	3900	0.2542	0.9335	72.8848	0.8416	0.8972	0.8685	87.9210
0.0874	9.22	4000	0.2580	0.9350	72.9432	0.8532	0.9052	0.8784	88.0372
0.1019	9.45	4100	0.2615	0.9351	72.8476	0.8704	0.9024	0.8862	87.9791
0.1039	9.68	4200	0.2635	0.9331	72.8678	0.8432	0.8900	0.8660	87.2242
0.0986	9.91	4300	0.2588	0.9352	72.9545	0.8548	0.9078	0.8805	87.7468
0.0867	10.14	4400	0.2659	0.9347	72.8253	0.8574	0.9025	0.8794	87.9210
0.1029	10.37	4500	0.2651	0.9350	72.9023	0.8480	0.9042	0.8752	87.8630
0.0935	10.6	4600	0.2669	0.9344	72.8549	0.8438	0.8981	0.8701	87.9791
0.0944	10.83	4700	0.2703	0.9334	72.8564	0.8460	0.9021	0.8732	87.3403
0.0724	11.06	4800	0.2712	0.9349	72.9326	0.8435	0.9010	0.8713	88.2695
0.0906	11.29	4900	0.2708	0.9351	72.8490	0.8513	0.9062	0.8779	87.8049
0.0819	11.52	5000	0.2683	0.9356	72.8973	0.8304	0.9056	0.8664	87.9791
0.0892	11.75	5100	0.2704	0.9361	72.9746	0.8463	0.9069	0.8755	87.9210
0.0791	11.98	5200	0.2705	0.9353	72.9050	0.8475	0.9064	0.8759	88.1533
0.0718	12.21	5300	0.2751	0.9361	73.0216	0.8546	0.9035	0.8783	87.9791
0.0744	12.44	5400	0.2769	0.9355	72.9041	0.8717	0.9063	0.8886	87.9210
0.081	12.67	5500	0.2735	0.9359	72.9850	0.8501	0.9092	0.8786	87.8630
0.0757	12.9	5600	0.2778	0.9359	72.9826	0.8639	0.9133	0.8879	88.2114
0.0648	13.13	5700	0.2876	0.9333	72.9175	0.8587	0.9111	0.8841	88.2114
0.0738	13.36	5800	0.2782	0.9360	73.0831	0.8647	0.9144	0.8888	88.1533
0.0653	13.59	5900	0.2803	0.9354	73.0048	0.8628	0.9120	0.8867	88.2695
0.0717	13.82	6000	0.2792	0.9359	73.0330	0.8387	0.9033	0.8698	87.8049
0.071	14.06	6100	0.2787	0.9363	73.0967	0.8527	0.9070	0.8790	87.9210
0.0661	14.29	6200	0.2828	0.9361	73.0762	0.8482	0.9068	0.8765	87.8630
0.062	14.52	6300	0.2812	0.9361	73.0716	0.8399	0.9070	0.8722	87.6887
0.0722	14.75	6400	0.2808	0.9361	73.0682	0.8377	0.9032	0.8692	87.5145
0.0633	14.98	6500	0.2811	0.9361	73.0667	0.8451	0.9039	0.8735	87.6307

Framework versions

• Transformers 4.26.0
• Pytorch 1.12.0
• Datasets 2.9.0
• Tokenizers 0.13.2