Better regex expression for extracting language code in tokenization_marian.py
Author: soumya-ch
Pull request: yes
Labels: name: wontfix

name: marian

# Better regex expression for extracting language code in tokenization_marian.py

In `tokenization_marian.py`, the regex `>>.+<<` is used to extract the language token from the sentences leading to the following incorrect tokenization.

```
Example Sentence:
>>hin<< We use cout<< function to print a line in C++.

Current Tokenizer gives:
['<unk>', '▁function', '▁to', '▁print', '▁a', '▁line', '▁in', '▁C', '++', '.', '
']

Expected Tokenization:
['>>hin<<', '▁We', '▁use', '▁c', 'out', '<', '<', '▁function', '▁to', '▁print', '▁a', '▁line', '▁in', '▁C', '++', '.', '
']
```

This pull request changes the regex to `>>.{3}<<|>>.{3}\_.{4}<<` which covers the 194 language tags in the en-mul model.

@sshleifer 

Comments:
cc @patrickvonplaten 

No, 2 character language codes are not covered. However `>>.{2}<<|>>.{3}<<|>>.{3}\_.{4}<<` can be used.

I can try writing tests. Is these somewhere where I could find an exhaustive set of language codes? I had written the regex to cover the 194 language code in `'Helsinki-NLP/opus-mt-en-mul'`.

```python
def test_language_codes(self):
    tok = MarianTokenizer.from_pretrained(f"{ORG_NAME}opus-mt-en-mul")
    batch = tok.prepare_seq2seq_batch([">>hin<< I am a small frog", ">>zlm_Latn<< I am a small frog", ">>fr<< I am a small frog"], return_tensors=FRAMEWORK)
    
    expected = [[[888, 21, 437, 9,  2613, 37565, 0], [770, 21, 437, 9, 2613, 37565, 0], [1, 21, 437, 9, 2613, 37565, 0]]
    for i in range(3):
        self.assertListEqual(expected[i], batch.input_ids[i])
```

The above will need some changes since `>>fr<<` is not in the vocabulary and the sentences also do not have another `<<` in them (maybe a different test for that?).

> Is there somewhere where I could find an exhaustive set of language codes?

Not easily, you can find the supported codes for any model by looking for `tgt_languages` or `tgt_constituents` on a model card e.g. https://huggingface.co/Helsinki-NLP/opus-mt-en-ROMANCE or https://huggingface.co/Helsinki-NLP/opus-mt-en-roa?text=%3E%3Efra%3C%3C+My+name+is+Sarah+and+I+live+in+London


Here are some more:

```python
tgt_constituents= {'ita', 'cat', 'roh', 'spa', 'pap', 'lmo', 'mwl', 'lij', 'lad_Latn', 'ext', 'ron', 'ast', 'glg', 'pms', 'zsm_Latn', 'gcf_Latn', 'lld_Latn', 'min', 'tmw_Latn', 'cos', 'wln', 'zlm_Latn', 'por', 'egl', 'oci', 'vec', 'arg', 'ind', 'fra', 'hat', 'lad', 'max_Latn', 'frm_Latn', 'scn', 'mfe'}
tgt_constituents: {'cmn_Hans', 'nan', 'nan_Hani', 'gan', 'yue', 'cmn_Kana', 'yue_Hani', 'wuu_Bopo', 'cmn_Latn', 'yue_Hira', 'cmn_Hani', 'cjy_Hans', 'cmn', 'lzh_Hang', 'lzh_Hira', 'cmn_Hant', 'lzh_Bopo', 'zho', 'zho_Hans', 'zho_Hant', 'lzh_Hani', 'yue_Hang', 'wuu', 'yue_Kana', 'wuu_Latn', 'yue_Bopo', 'cjy_Hant', 'yue_Hans', 'lzh', 'cmn_Hira', 'lzh_Yiii', 'lzh_Hans', 'cmn_Bopo', 'cmn_Hang', 'hak_Hani', 'cmn_Yiii', 'yue_Hant', 'lzh_Kana', 'wuu_Hani'}
```


I think we should accept any number of alphabet characters or '_' inside the `>>` or at least up to 8.





This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.