cartesinus commited on
Commit
d14286a
1 Parent(s): f2a65ad

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +23 -20
README.md CHANGED
@@ -23,39 +23,42 @@ This model is a fine-tuned version of [facebook/m2m100_418M](https://huggingface
23
 
24
  It achieves the following results:
25
  1) On the test set (iva_mt):
26
- - Bleu (plain text): 39.1560
27
- - Bleu (with slots): 63.8767
28
- - Baseline m2m100-418M (plain text): TBD
29
 
30
  2) WMT20 (en2pl):
31
- - Bleu (lowercased, tags removed): 15.0863 (for reference WMT20 submission systems in en-pl direction had between 25 and 30 bleu)
32
- - Baseline m2m100-418M (plain text): TBD
33
 
34
  4) On the evaluation set:
35
- - Bleu: 61.6249
36
 
37
  3) On the training set (to see how it adjusted to train):
38
- - Bleu (plain text): 70.5597
39
- - Bleu (with slots): 93.8200
40
 
41
- Bleu was measured with [sacrebleu](https://github.com/mjpost/sacrebleu) library.
42
 
43
  ## Model description, intended uses & limitations
44
 
45
- Model is biased towards virtual assistant (IVA) sentences in prediction/translation. These sentences are short, most of them are short, imperatives. Both of this facts
46
- can be observed in above results. WMT results are very low while in-domain test is very high. One thing that needs to be mentioned is that BLEU is not particulary good
47
- metric to evaluate IVA sentences due to their length and it should be evalued with other metrices (e.g. [GLEU](https://aclanthology.org/P15-2097.pdf)).
48
 
49
  This model will most probably force IVA translations on your text. As long as sentences that you are translating are more or less similar to massive and leyzer domains it
50
- can work. If you will translate domains unseen in either of them results might drop significantly to the point where baseline m2m100-418M will be better than this model.
 
51
 
52
- This model will generate tags in output even if there is not tag in input sentence. Frequency of this depends on input text. When testing IVA utterances this occurs
53
- between 3 and 5% of all cases. When WMT20 was translated it happened in % cases (input text was from News domain).
54
- This is not very severe and can be fixed easily in post-processing (something like `sed 's/<[a-z]>//g'` should be enough in most cases).
55
 
56
- Translations with slots very often differ from same sentences when slots are removed. This is quite frequent and it happens between 30 and 50% of translated utterances.
57
- For example there will be a difference between "is it raining in barcelona" and "is it raining in \<a\>barcelona\<a\>". In second case model will more likely localize name of
58
- city to some Polish name (here Lublin, because such city was given in Massive train set). This might be useful if you want to generate more variants.
 
 
 
59
 
60
  ## How to use
61
 
@@ -120,7 +123,7 @@ The following hyperparameters were used during training:
120
 
121
  ### Training results
122
 
123
- | Training Loss | Epoch | Step | Validation Loss | Bleu | Gen Len |
124
  |:-------------:|:-----:|:-----:|:---------------:|:-------:|:-------:|
125
  | 0.0178 | 1.0 | 5091 | 0.0171 | 57.4439 | 21.1396 |
126
  | 0.013 | 2.0 | 10182 | 0.0159 | 58.886 | 21.2285 |
 
23
 
24
  It achieves the following results:
25
  1) On the test set (iva_mt):
26
+ - BLEU (plain text): 39.1560
27
+ - BLEU (with slots): 63.8767
28
+ - BLEU for baseline m2m100-418M (plain text): TBD
29
 
30
  2) WMT20 (en2pl):
31
+ - BLEU (lowercased, tags removed): 15.0863 (for reference WMT20 submission systems in en-pl direction had between 25 and 30 bleu)
32
+ - BLEU for baseline m2m100-418M (plain text): 20.2750
33
 
34
  4) On the evaluation set:
35
+ - BLEU: 61.6249
36
 
37
  3) On the training set (to see how it adjusted to train):
38
+ - BLEU (plain text): 70.5597
39
+ - BLEU (with slots): 93.8200
40
 
41
+ BLEU was measured with [sacreBLEU](https://github.com/mjpost/sacrebleu) library.
42
 
43
  ## Model description, intended uses & limitations
44
 
45
+ Model is biased towards virtual assistant (IVA) sentences in prediction/translation. These sentences are short, most of them are short, imperatives. It can be observed in
46
+ above results where WMT results are very low while in-domain test is very high.
 
47
 
48
  This model will most probably force IVA translations on your text. As long as sentences that you are translating are more or less similar to massive and leyzer domains it
49
+ will be ok. If you will translate out-of-domain sentenences (such as for example News, Medical) that are not very similar then results will drop significantly up to the
50
+ point where baseline m2m100-418M will be better than this model.
51
 
52
+ This model will generate tags in output even if there is not tag in input sentence. Frequency of this depends on input text origin. When testing IVA utterances this occurs
53
+ between 3 and 5%. When WMT20 was translated it happened in 40% cases (input text was from News domain).
54
+ This is not very severe problem and it can be fixed easily in post-processing (simple `sed 's/<[a-z]>//g'` should be enough in most cases).
55
 
56
+ Translations with slot annotation very often differ from same sentences when slots are removed. This is quite frequent and it happens between 30 and 50% of translated
57
+ utterances. For example there will be a difference between "is it raining in barcelona" and "is it raining in \<a\>barcelona\<a\>". In second case model will more likely
58
+ localize name of city to some Polish name (here Lublin, because such city was given in Massive train set). This might be useful if you want to generate more variants.
59
+
60
+ One last thing that needs to be mentioned is that BLEU is not particulary good metric to evaluate IVA sentences due to their length and it should be evalued with other
61
+ metrices (e.g. [GLEU](https://aclanthology.org/P15-2097.pdf)).
62
 
63
  ## How to use
64
 
 
123
 
124
  ### Training results
125
 
126
+ | Training Loss | Epoch | Step | Validation Loss | BLEU | Gen Len |
127
  |:-------------:|:-----:|:-----:|:---------------:|:-------:|:-------:|
128
  | 0.0178 | 1.0 | 5091 | 0.0171 | 57.4439 | 21.1396 |
129
  | 0.013 | 2.0 | 10182 | 0.0159 | 58.886 | 21.2285 |