BramVanroy/mbart-en-es-nl-to-amr

MBART to "translate" English, Spanish and Dutch to AMR

To be used with MBART text-to-AMR pipeline (not yet available). Won't work with the default MBartTokenizer so you have to be patient until the pipeline can become available. :-)

This model is a fine-tuned version of facebook/mbart-large-cc25 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.3301
• Accuracy: 0.9180
• Bleu: 69.9161
• Smatch Precision: 0.8088
• Smatch Recall: 0.8878
• Smatch Fscore: 0.8465

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 8
• total_train_batch_size: 128
• total_eval_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 5

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	Bleu	Smatch Precision	Smatch Recall	Smatch Fscore	Ratio Invalid Amrs
1.8405	0.08	100	1.5336	0.6884	48.1355	0.2062	0.5313	0.2970	79.5780
1.2656	0.15	200	0.9595	0.7799	58.3189	0.3576	0.6405	0.4589	93.7863
0.996	0.23	300	0.7541	0.8240	62.2701	0.5021	0.7736	0.6089	94.4638
0.9236	0.31	400	0.6923	0.8298	63.3524	0.5014	0.8177	0.6216	94.1928
0.8074	0.38	500	0.5592	0.8643	65.1618	0.6030	0.7911	0.6843	91.1343
0.693	0.46	600	0.5499	0.8628	65.7325	0.5453	0.8207	0.6552	92.0441
0.6427	0.54	700	0.4840	0.8761	67.3134	0.6329	0.8262	0.7167	91.6957
0.6001	0.61	800	0.4613	0.8792	67.1435	0.6296	0.8464	0.7221	91.8893
0.6801	0.69	900	0.4265	0.8889	66.8917	0.6635	0.8356	0.7397	90.9408
0.5366	0.77	1000	0.4399	0.8899	66.8936	0.6430	0.8182	0.7201	89.1792
0.5446	0.84	1100	0.4050	0.8953	67.5782	0.6612	0.8492	0.7435	90.7859
0.4848	0.92	1200	0.4026	0.8955	68.3245	0.6778	0.8531	0.7554	90.2245
0.4988	1.0	1300	0.3909	0.8955	68.1950	0.6829	0.8673	0.7641	90.7859
0.4567	1.07	1400	0.3825	0.9013	68.2347	0.6899	0.8669	0.7683	89.7019
0.4563	1.15	1500	0.3800	0.9014	68.2520	0.7375	0.8664	0.7967	90.4181
0.4386	1.23	1600	0.3806	0.8996	68.1288	0.7334	0.8768	0.7987	91.1537
0.4377	1.3	1700	0.3814	0.8968	67.6520	0.7182	0.8380	0.7735	88.5792
0.4477	1.38	1800	0.3781	0.8986	68.1177	0.7376	0.8763	0.8010	91.8118
0.4254	1.46	1900	0.3578	0.9062	68.5803	0.7173	0.8635	0.7836	89.7406
0.451	1.53	2000	0.3569	0.9061	68.9853	0.7563	0.8708	0.8095	90.3020
0.3828	1.61	2100	0.3579	0.9050	68.7272	0.7712	0.8733	0.8191	90.3600
0.4147	1.69	2200	0.3545	0.9067	69.0921	0.7690	0.8786	0.8201	90.6504
0.3699	1.76	2300	0.3546	0.9059	69.2822	0.7562	0.8774	0.8123	90.6117
0.3651	1.84	2400	0.3468	0.9098	70.1585	0.7761	0.8737	0.8220	89.9148
0.3831	1.92	2500	0.3431	0.9101	69.0716	0.7619	0.8721	0.8133	89.8180
0.3676	1.99	2600	0.3447	0.9098	69.8364	0.7814	0.8765	0.8262	90.2245
0.3281	2.07	2700	0.3443	0.9097	69.1463	0.8037	0.8804	0.8403	90.4762
0.3471	2.15	2800	0.3407	0.9116	69.2910	0.7662	0.8763	0.8175	89.6245
0.327	2.22	2900	0.3414	0.9118	69.8713	0.7725	0.8752	0.8207	89.7213
0.3232	2.3	3000	0.3386	0.9129	69.4165	0.7666	0.8765	0.8179	89.5277
0.3168	2.38	3100	0.3593	0.9051	69.1672	0.7736	0.8824	0.8244	90.9408
0.2781	2.45	3200	0.3408	0.9127	69.2028	0.7767	0.8720	0.8216	89.1599
0.3135	2.53	3300	0.3374	0.9131	69.3979	0.7611	0.8760	0.8146	89.6438
0.326	2.61	3400	0.3336	0.9128	69.9662	0.7916	0.8815	0.8341	89.9923
0.3258	2.68	3500	0.3329	0.9139	69.7743	0.7925	0.8796	0.8338	89.8180
0.3159	2.76	3600	0.3377	0.9130	69.6988	0.7845	0.8877	0.8329	89.6825
0.3178	2.84	3700	0.3303	0.9142	69.6296	0.7787	0.8780	0.8254	88.8308
0.3144	2.91	3800	0.3310	0.9137	69.3942	0.7895	0.8815	0.8330	89.3728
0.3098	2.99	3900	0.3298	0.9151	69.8902	0.8011	0.8732	0.8356	89.1986
0.3005	3.07	4000	0.3334	0.9154	69.6235	0.7845	0.8819	0.8303	89.3922
0.2716	3.14	4100	0.3319	0.9154	69.4647	0.8098	0.8797	0.8433	89.4890
0.2801	3.22	4200	0.3329	0.9151	69.5338	0.8019	0.8851	0.8415	89.6825
0.2721	3.3	4300	0.3327	0.9153	69.6714	0.8028	0.8885	0.8435	89.6051
0.2607	3.37	4400	0.3310	0.9157	69.5581	0.7916	0.8797	0.8333	89.2760
0.2823	3.45	4500	0.3309	0.9156	69.6805	0.8123	0.8887	0.8488	89.3922
0.2675	3.53	4600	0.3313	0.9158	69.6664	0.8168	0.8844	0.8492	89.4115
0.2642	3.6	4700	0.3297	0.9166	69.6888	0.8147	0.8904	0.8509	89.2954
0.2842	3.68	4800	0.3299	0.9162	69.6175	0.8000	0.8870	0.8413	89.5858
0.2646	3.76	4900	0.3294	0.9168	69.6792	0.7889	0.8827	0.8332	89.1986
0.2624	3.83	5000	0.3276	0.9171	69.6874	0.8047	0.8906	0.8455	89.2760
0.2647	3.91	5100	0.3282	0.9166	69.6530	0.7998	0.8823	0.8390	89.3535
0.2525	3.99	5200	0.3269	0.9168	69.7478	0.8062	0.8853	0.8439	89.3535
0.2417	4.06	5300	0.3311	0.9168	69.6945	0.7978	0.8877	0.8404	89.4503
0.2608	4.14	5400	0.3311	0.9169	69.7272	0.7998	0.8865	0.8409	89.3535
0.2408	4.22	5500	0.3308	0.9172	69.9450	0.8111	0.8864	0.8471	89.3341
0.2268	4.29	5600	0.3319	0.9176	69.8016	0.7923	0.8845	0.8359	89.0437
0.2158	4.37	5700	0.3315	0.9173	69.7748	0.8008	0.8849	0.8407	89.5083
0.2461	4.45	5800	0.3310	0.9174	69.9786	0.8030	0.8876	0.8432	89.2954
0.249	4.52	5900	0.3315	0.9176	69.9070	0.8066	0.8886	0.8456	89.4503
0.2428	4.6	6000	0.3312	0.9176	69.8172	0.8017	0.8859	0.8417	89.2760
0.2266	4.68	6100	0.3306	0.9178	69.8095	0.8016	0.8893	0.8431	89.1599
0.2266	4.75	6200	0.3305	0.9178	69.8328	0.8090	0.8902	0.8476	89.4309
0.2341	4.83	6300	0.3305	0.9180	69.8605	0.8101	0.8883	0.8474	89.4309
0.2226	4.91	6400	0.3305	0.9179	69.9342	0.8101	0.8896	0.8480	89.3728
0.209	4.98	6500	0.3301	0.9180	69.9161	0.8088	0.8878	0.8465	89.3728

Framework versions

• Transformers 4.26.0
• Pytorch 1.12.0
• Datasets 2.9.0
• Tokenizers 0.13.2