tiedeman commited on
Commit
950f618
1 Parent(s): 9558d92

Initial commit

Browse files
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ *.spm filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,182 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ language:
4
+ - de
5
+ - en
6
+ - es
7
+ - fr
8
+ - pt
9
+ - zh
10
+
11
+ tags:
12
+ - translation
13
+ - opus-mt-tc-bible
14
+
15
+ license: apache-2.0
16
+ model-index:
17
+ - name: opus-mt-tc-bible-big-zhx-deu_eng_fra_por_spa
18
+ results:
19
+ - task:
20
+ name: Translation multi-multi
21
+ type: translation
22
+ args: multi-multi
23
+ dataset:
24
+ name: tatoeba-test-v2020-07-28-v2023-09-26
25
+ type: tatoeba_mt
26
+ args: multi-multi
27
+ metrics:
28
+ - name: BLEU
29
+ type: bleu
30
+ value: 34.1
31
+ - name: chr-F
32
+ type: chrf
33
+ value: 0.52761
34
+ ---
35
+ # opus-mt-tc-bible-big-zhx-deu_eng_fra_por_spa
36
+
37
+ ## Table of Contents
38
+ - [Model Details](#model-details)
39
+ - [Uses](#uses)
40
+ - [Risks, Limitations and Biases](#risks-limitations-and-biases)
41
+ - [How to Get Started With the Model](#how-to-get-started-with-the-model)
42
+ - [Training](#training)
43
+ - [Evaluation](#evaluation)
44
+ - [Citation Information](#citation-information)
45
+ - [Acknowledgements](#acknowledgements)
46
+
47
+ ## Model Details
48
+
49
+ Neural machine translation model for translating from Chinese (family) (zhx) to unknown (deu+eng+fra+por+spa).
50
+
51
+ This model is part of the [OPUS-MT project](https://github.com/Helsinki-NLP/Opus-MT), an effort to make neural machine translation models widely available and accessible for many languages in the world. All models are originally trained using the amazing framework of [Marian NMT](https://marian-nmt.github.io/), an efficient NMT implementation written in pure C++. The models have been converted to pyTorch using the transformers library by huggingface. Training data is taken from [OPUS](https://opus.nlpl.eu/) and training pipelines use the procedures of [OPUS-MT-train](https://github.com/Helsinki-NLP/Opus-MT-train).
52
+ **Model Description:**
53
+ - **Developed by:** Language Technology Research Group at the University of Helsinki
54
+ - **Model Type:** Translation (transformer-big)
55
+ - **Release**: 2024-05-30
56
+ - **License:** Apache-2.0
57
+ - **Language(s):**
58
+ - Source Language(s): cjy cmn hsn wuu yue
59
+ - Target Language(s): deu eng fra por spa
60
+ - Valid Target Language Labels: >>deu<< >>eng<< >>fra<< >>por<< >>spa<< >>xxx<<
61
+ - **Original Model**: [opusTCv20230926max50+bt+jhubc_transformer-big_2024-05-30.zip](https://object.pouta.csc.fi/Tatoeba-MT-models/zhx-deu+eng+fra+por+spa/opusTCv20230926max50+bt+jhubc_transformer-big_2024-05-30.zip)
62
+ - **Resources for more information:**
63
+ - [OPUS-MT dashboard](https://opus.nlpl.eu/dashboard/index.php?pkg=opusmt&test=all&scoreslang=all&chart=standard&model=Tatoeba-MT-models/zhx-deu%2Beng%2Bfra%2Bpor%2Bspa/opusTCv20230926max50%2Bbt%2Bjhubc_transformer-big_2024-05-30)
64
+ - [OPUS-MT-train GitHub Repo](https://github.com/Helsinki-NLP/OPUS-MT-train)
65
+ - [More information about MarianNMT models in the transformers library](https://huggingface.co/docs/transformers/model_doc/marian)
66
+ - [Tatoeba Translation Challenge](https://github.com/Helsinki-NLP/Tatoeba-Challenge/)
67
+ - [HPLT bilingual data v1 (as part of the Tatoeba Translation Challenge dataset)](https://hplt-project.org/datasets/v1)
68
+ - [A massively parallel Bible corpus](https://aclanthology.org/L14-1215/)
69
+
70
+ This is a multilingual translation model with multiple target languages. A sentence initial language token is required in the form of `>>id<<` (id = valid target language ID), e.g. `>>deu<<`
71
+
72
+ ## Uses
73
+
74
+ This model can be used for translation and text-to-text generation.
75
+
76
+ ## Risks, Limitations and Biases
77
+
78
+ **CONTENT WARNING: Readers should be aware that the model is trained on various public data sets that may contain content that is disturbing, offensive, and can propagate historical and current stereotypes.**
79
+
80
+ Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)).
81
+
82
+ ## How to Get Started With the Model
83
+
84
+ A short example code:
85
+
86
+ ```python
87
+ from transformers import MarianMTModel, MarianTokenizer
88
+
89
+ src_text = [
90
+ ">>deu<< Replace this with text in an accepted source language.",
91
+ ">>spa<< This is the second sentence."
92
+ ]
93
+
94
+ model_name = "pytorch-models/opus-mt-tc-bible-big-zhx-deu_eng_fra_por_spa"
95
+ tokenizer = MarianTokenizer.from_pretrained(model_name)
96
+ model = MarianMTModel.from_pretrained(model_name)
97
+ translated = model.generate(**tokenizer(src_text, return_tensors="pt", padding=True))
98
+
99
+ for t in translated:
100
+ print( tokenizer.decode(t, skip_special_tokens=True) )
101
+ ```
102
+
103
+ You can also use OPUS-MT models with the transformers pipelines, for example:
104
+
105
+ ```python
106
+ from transformers import pipeline
107
+ pipe = pipeline("translation", model="Helsinki-NLP/opus-mt-tc-bible-big-zhx-deu_eng_fra_por_spa")
108
+ print(pipe(">>deu<< Replace this with text in an accepted source language."))
109
+ ```
110
+
111
+ ## Training
112
+
113
+ - **Data**: opusTCv20230926max50+bt+jhubc ([source](https://github.com/Helsinki-NLP/Tatoeba-Challenge))
114
+ - **Pre-processing**: SentencePiece (spm32k,spm32k)
115
+ - **Model Type:** transformer-big
116
+ - **Original MarianNMT Model**: [opusTCv20230926max50+bt+jhubc_transformer-big_2024-05-30.zip](https://object.pouta.csc.fi/Tatoeba-MT-models/zhx-deu+eng+fra+por+spa/opusTCv20230926max50+bt+jhubc_transformer-big_2024-05-30.zip)
117
+ - **Training Scripts**: [GitHub Repo](https://github.com/Helsinki-NLP/OPUS-MT-train)
118
+
119
+ ## Evaluation
120
+
121
+ * [Model scores at the OPUS-MT dashboard](https://opus.nlpl.eu/dashboard/index.php?pkg=opusmt&test=all&scoreslang=all&chart=standard&model=Tatoeba-MT-models/zhx-deu%2Beng%2Bfra%2Bpor%2Bspa/opusTCv20230926max50%2Bbt%2Bjhubc_transformer-big_2024-05-30)
122
+ * test set translations: [opusTCv20230926max50+bt+jhubc_transformer-big_2024-05-29.test.txt](https://object.pouta.csc.fi/Tatoeba-MT-models/zhx-deu+eng+fra+por+spa/opusTCv20230926max50+bt+jhubc_transformer-big_2024-05-29.test.txt)
123
+ * test set scores: [opusTCv20230926max50+bt+jhubc_transformer-big_2024-05-29.eval.txt](https://object.pouta.csc.fi/Tatoeba-MT-models/zhx-deu+eng+fra+por+spa/opusTCv20230926max50+bt+jhubc_transformer-big_2024-05-29.eval.txt)
124
+ * benchmark results: [benchmark_results.txt](benchmark_results.txt)
125
+ * benchmark output: [benchmark_translations.zip](benchmark_translations.zip)
126
+
127
+ | langpair | testset | chr-F | BLEU | #sent | #words |
128
+ |----------|---------|-------|-------|-------|--------|
129
+ | multi-multi | tatoeba-test-v2020-07-28-v2023-09-26 | 0.52761 | 34.1 | 10000 | 82611 |
130
+
131
+ ## Citation Information
132
+
133
+ * Publications: [Democratizing neural machine translation with OPUS-MT](https://doi.org/10.1007/s10579-023-09704-w) and [OPUS-MT – Building open translation services for the World](https://aclanthology.org/2020.eamt-1.61/) and [The Tatoeba Translation Challenge – Realistic Data Sets for Low Resource and Multilingual MT](https://aclanthology.org/2020.wmt-1.139/) (Please, cite if you use this model.)
134
+
135
+ ```bibtex
136
+ @article{tiedemann2023democratizing,
137
+ title={Democratizing neural machine translation with {OPUS-MT}},
138
+ author={Tiedemann, J{\"o}rg and Aulamo, Mikko and Bakshandaeva, Daria and Boggia, Michele and Gr{\"o}nroos, Stig-Arne and Nieminen, Tommi and Raganato, Alessandro and Scherrer, Yves and Vazquez, Raul and Virpioja, Sami},
139
+ journal={Language Resources and Evaluation},
140
+ number={58},
141
+ pages={713--755},
142
+ year={2023},
143
+ publisher={Springer Nature},
144
+ issn={1574-0218},
145
+ doi={10.1007/s10579-023-09704-w}
146
+ }
147
+
148
+ @inproceedings{tiedemann-thottingal-2020-opus,
149
+ title = "{OPUS}-{MT} {--} Building open translation services for the World",
150
+ author = {Tiedemann, J{\"o}rg and Thottingal, Santhosh},
151
+ booktitle = "Proceedings of the 22nd Annual Conference of the European Association for Machine Translation",
152
+ month = nov,
153
+ year = "2020",
154
+ address = "Lisboa, Portugal",
155
+ publisher = "European Association for Machine Translation",
156
+ url = "https://aclanthology.org/2020.eamt-1.61",
157
+ pages = "479--480",
158
+ }
159
+
160
+ @inproceedings{tiedemann-2020-tatoeba,
161
+ title = "The Tatoeba Translation Challenge {--} Realistic Data Sets for Low Resource and Multilingual {MT}",
162
+ author = {Tiedemann, J{\"o}rg},
163
+ booktitle = "Proceedings of the Fifth Conference on Machine Translation",
164
+ month = nov,
165
+ year = "2020",
166
+ address = "Online",
167
+ publisher = "Association for Computational Linguistics",
168
+ url = "https://aclanthology.org/2020.wmt-1.139",
169
+ pages = "1174--1182",
170
+ }
171
+ ```
172
+
173
+ ## Acknowledgements
174
+
175
+ The work is supported by the [HPLT project](https://hplt-project.org/), funded by the European Union’s Horizon Europe research and innovation programme under grant agreement No 101070350. We are also grateful for the generous computational resources and IT infrastructure provided by [CSC -- IT Center for Science](https://www.csc.fi/), Finland, and the [EuroHPC supercomputer LUMI](https://www.lumi-supercomputer.eu/).
176
+
177
+ ## Model conversion info
178
+
179
+ * transformers version: 4.45.1
180
+ * OPUS-MT git hash: 0882077
181
+ * port time: Wed Oct 9 02:20:05 EEST 2024
182
+ * port machine: LM0-400-22516.local
benchmark_results.txt ADDED
@@ -0,0 +1 @@
 
 
1
+ multi-multi tatoeba-test-v2020-07-28-v2023-09-26 0.52761 34.1 10000 82611
benchmark_translations.zip ADDED
File without changes
config.json ADDED
@@ -0,0 +1,41 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "pytorch-models/opus-mt-tc-bible-big-zhx-deu_eng_fra_por_spa",
3
+ "activation_dropout": 0.0,
4
+ "activation_function": "relu",
5
+ "architectures": [
6
+ "MarianMTModel"
7
+ ],
8
+ "attention_dropout": 0.0,
9
+ "bos_token_id": 0,
10
+ "classifier_dropout": 0.0,
11
+ "d_model": 1024,
12
+ "decoder_attention_heads": 16,
13
+ "decoder_ffn_dim": 4096,
14
+ "decoder_layerdrop": 0.0,
15
+ "decoder_layers": 6,
16
+ "decoder_start_token_id": 59177,
17
+ "decoder_vocab_size": 59178,
18
+ "dropout": 0.1,
19
+ "encoder_attention_heads": 16,
20
+ "encoder_ffn_dim": 4096,
21
+ "encoder_layerdrop": 0.0,
22
+ "encoder_layers": 6,
23
+ "eos_token_id": 1422,
24
+ "forced_eos_token_id": null,
25
+ "init_std": 0.02,
26
+ "is_encoder_decoder": true,
27
+ "max_length": null,
28
+ "max_position_embeddings": 1024,
29
+ "model_type": "marian",
30
+ "normalize_embedding": false,
31
+ "num_beams": null,
32
+ "num_hidden_layers": 6,
33
+ "pad_token_id": 59177,
34
+ "scale_embedding": true,
35
+ "share_encoder_decoder_embeddings": true,
36
+ "static_position_embeddings": true,
37
+ "torch_dtype": "float32",
38
+ "transformers_version": "4.45.1",
39
+ "use_cache": true,
40
+ "vocab_size": 59178
41
+ }
generation_config.json ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bad_words_ids": [
4
+ [
5
+ 59177
6
+ ]
7
+ ],
8
+ "bos_token_id": 0,
9
+ "decoder_start_token_id": 59177,
10
+ "eos_token_id": 1422,
11
+ "forced_eos_token_id": 1422,
12
+ "max_length": 512,
13
+ "num_beams": 4,
14
+ "pad_token_id": 59177,
15
+ "transformers_version": "4.45.1"
16
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a59977d3c2fd15af9da2286652c2adf25d84d04fc8086149a427ea021762b821
3
+ size 948088920
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4377f8f748a2b734216340dfffacafc08e67c83dc15cdac45c038febe4e8afa4
3
+ size 948140165
source.spm ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0cdb21fe4ef6966659c0f3f45fe9e393c0a94fca256bb08aadae7953cd35ff2f
3
+ size 728726
special_tokens_map.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"eos_token": "</s>", "unk_token": "<unk>", "pad_token": "<pad>"}
target.spm ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9018c32e04d3fa38189ba81a6168066aca7a514b0084d55dc435c81191c8eda9
3
+ size 784035
tokenizer_config.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"source_lang": "zhx", "target_lang": "deu+eng+fra+por+spa", "unk_token": "<unk>", "eos_token": "</s>", "pad_token": "<pad>", "model_max_length": 512, "sp_model_kwargs": {}, "separate_vocabs": false, "special_tokens_map_file": null, "name_or_path": "marian-models/opusTCv20230926max50+bt+jhubc_transformer-big_2024-05-30/zhx-deu+eng+fra+por+spa", "tokenizer_class": "MarianTokenizer"}
vocab.json ADDED
The diff for this file is too large to render. See raw diff