iva_mt_wslot-m2m100_1.2B-en-pl (v0.1.0)

This model is a fine-tuned version of facebook/m2m100_1.2B on the iva_mt_wslot dataset. There is also smaller version of this model here: cartesinus/iva_mt_wslot-m2m100_418M-en-pl. This model (1.2B) achieves the following results:

1. On the test set (iva_mt):

• BLEU (plain text): (result in preparation)
• BLEU (with slots): (result in preparation)
• F1 score: (in preparation)

For reference BLEU for baseline m2m100-418M (plain text) was 21.9468 and for m2m100-1.2B was (result in preparation). Second result (BLEU with slots) is when tags are treated as "normal" words in sentence. Therefore that result might be a bit misleading. Please refer to plain text results if you are not sure how to interpret them.

2. WMT20 (en2pl):

• BLEU (lowercased, tags removed): (result in preparation)
• BLEU for baseline m2m100-1.2B (plain text): (result in preparation)

For reference WMT20 submission systems in en-pl direction had between 25 and 30 BLEU

3. BLEU on the evaluation set (same as in below table 'Training results'): 62.4604

4. On the training set (to see how it adjusted to train):

• BLEU (plain text): (result in preparation)
• BLEU (with slots): (result in preparation)

BLEU was measured with sacreBLEU library.

Model description, intended uses & limitations

Model is biased towards virtual assistant (IVA) sentences in prediction/translation. These sentences are short, imperatives with a lot of name entities (slots) and particular vocabulary (for example settings name). It can be observed in above results where WMT results are very low while in-domain test is very high.

This model will most probably force IVA translations on your text. As long as sentences that you are translating are more or less similar to massive and leyzer domains it will be ok. If you will translate out-of-domain sentenences (such as for example News, Medical) that are not very similar then results will drop significantly.

One last thing that needs to be mentioned is that BLEU is not particulary good metric to evaluate IVA sentences due to their length and it should be evalued with other metrices (e.g. GLEU).

How to use

First please make sure to install pip install transformers. First download model:

from transformers import M2M100ForConditionalGeneration, M2M100Tokenizer
import torch

def translate(input_text, lang):
    input_ids = tokenizer(input_text, return_tensors="pt")
    generated_tokens = model.generate(**input_ids, forced_bos_token_id=tokenizer.get_lang_id(lang))
    return tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)

model_name = "cartesinus/iva_mt_wslot-m2m100_1.2B-en-pl"
tokenizer = M2M100Tokenizer.from_pretrained(model_name, src_lang="en", tgt_lang="pl")
model = M2M100ForConditionalGeneration.from_pretrained(model_name)

Then you can translate either plan text like this:

print(translate("set the temperature on my thermostat", "pl"))

or you can translate with slot annotations that will be restored in tgt language:

print(translate("wake me up at <a>nine am<a> on <b>friday<b>", "pl")) #translation: obudź mnie o <a>piątej rano<a> <b>w tym tygodniu<b>

Limitations of translation with slot transfer:

1. Annotated words must be placed between semi-xml tags like this "this is <a>example<a>"
2. There is no closing tag for example "<\a>" in above example - this is done on purpose to ommit problems with backslash escape
3. If sentence consists of more than one slot then simply use next alphabet letter. For example "this is <a>example<a> with more than <b>one<b> slot"
4. Please do not add space before first or last annotated word because this particular model was trained this way and it most probably will lower it's results

Training and evaluation data

Dataset Composition (en-pl)

Corpus	Train	Dev	Test
Massive 1.1	11514	2033	2974
Leyzer 0.2.0	3974	701	1380
OpenSubtitles from OPUS	2329	411	500
KDE from OPUS	1154	241	241
CCMatrix from Opus	1096	232	237
Ubuntu from OPUS	281	60	59
Gnome from OPUS	14	3	3
total	20362	3681	5394

Training results

Training Loss	Epoch	Step	Validation Loss	Bleu	Gen Len
0.2744	1.0	5091	0.2555	58.5119	21.0728
0.1829	2.0	10182	0.2475	59.7364	21.0769
0.1124	3.0	15273	0.2499	61.3552	21.06
0.0783	4.0	20364	0.2597	61.6618	21.2402
0.0496	5.0	25455	0.2698	62.1942	21.2901
0.0318	6.0	30546	0.2798	61.9068	21.3399
0.0204	7.0	35637	0.2893	61.7753	21.3102
0.0138	8.0	40728	0.2979	62.3925	21.3238
0.009	9.0	45819	0.3034	62.4942	21.2516
0.0058	10.0	50910	0.3082	62.4604	21.2847

Framework versions

• Transformers 4.26.1
• Pytorch 1.13.1+cu116
• Datasets 2.10.1
• Tokenizers 0.13.2