tiedeman commited on
Commit
71201d4
1 Parent(s): f18f1e5

Initial commit

Browse files
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ *.spm filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,198 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ language:
4
+ - chm
5
+ - es
6
+ - et
7
+ - fi
8
+ - fkv
9
+ - fr
10
+ - hu
11
+ - it
12
+ - krl
13
+ - liv
14
+ - mrj
15
+ - myv
16
+ - pt
17
+ - se
18
+ - udm
19
+ - vep
20
+
21
+ tags:
22
+ - translation
23
+ - opus-mt-tc-bible
24
+
25
+ license: apache-2.0
26
+ model-index:
27
+ - name: opus-mt-tc-bible-big-fiu-fra_ita_por_spa
28
+ results:
29
+ - task:
30
+ name: Translation multi-multi
31
+ type: translation
32
+ args: multi-multi
33
+ dataset:
34
+ name: tatoeba-test-v2020-07-28-v2023-09-26
35
+ type: tatoeba_mt
36
+ args: multi-multi
37
+ metrics:
38
+ - name: BLEU
39
+ type: bleu
40
+ value: 48.5
41
+ - name: chr-F
42
+ type: chrf
43
+ value: 0.67829
44
+ ---
45
+ # opus-mt-tc-bible-big-fiu-fra_ita_por_spa
46
+
47
+ ## Table of Contents
48
+ - [Model Details](#model-details)
49
+ - [Uses](#uses)
50
+ - [Risks, Limitations and Biases](#risks-limitations-and-biases)
51
+ - [How to Get Started With the Model](#how-to-get-started-with-the-model)
52
+ - [Training](#training)
53
+ - [Evaluation](#evaluation)
54
+ - [Citation Information](#citation-information)
55
+ - [Acknowledgements](#acknowledgements)
56
+
57
+ ## Model Details
58
+
59
+ Neural machine translation model for translating from Finno-Ugrian languages (fiu) to unknown (fra+ita+por+spa).
60
+
61
+ This model is part of the [OPUS-MT project](https://github.com/Helsinki-NLP/Opus-MT), an effort to make neural machine translation models widely available and accessible for many languages in the world. All models are originally trained using the amazing framework of [Marian NMT](https://marian-nmt.github.io/), an efficient NMT implementation written in pure C++. The models have been converted to pyTorch using the transformers library by huggingface. Training data is taken from [OPUS](https://opus.nlpl.eu/) and training pipelines use the procedures of [OPUS-MT-train](https://github.com/Helsinki-NLP/Opus-MT-train).
62
+ **Model Description:**
63
+ - **Developed by:** Language Technology Research Group at the University of Helsinki
64
+ - **Model Type:** Translation (transformer-big)
65
+ - **Release**: 2024-08-17
66
+ - **License:** Apache-2.0
67
+ - **Language(s):**
68
+ - Source Language(s): chm est fin fkv hun krl liv mrj myv sme udm vep vro
69
+ - Target Language(s): fra ita por spa
70
+ - Valid Target Language Labels: >>fra<< >>ita<< >>por<< >>spa<< >>xxx<<
71
+ - **Original Model**: [opusTCv20230926max50+bt+jhubc_transformer-big_2024-08-17.zip](https://object.pouta.csc.fi/Tatoeba-MT-models/fiu-fra+ita+por+spa/opusTCv20230926max50+bt+jhubc_transformer-big_2024-08-17.zip)
72
+ - **Resources for more information:**
73
+ - [OPUS-MT dashboard](https://opus.nlpl.eu/dashboard/index.php?pkg=opusmt&test=all&scoreslang=all&chart=standard&model=Tatoeba-MT-models/fiu-fra%2Bita%2Bpor%2Bspa/opusTCv20230926max50%2Bbt%2Bjhubc_transformer-big_2024-08-17)
74
+ - [OPUS-MT-train GitHub Repo](https://github.com/Helsinki-NLP/OPUS-MT-train)
75
+ - [More information about MarianNMT models in the transformers library](https://huggingface.co/docs/transformers/model_doc/marian)
76
+ - [Tatoeba Translation Challenge](https://github.com/Helsinki-NLP/Tatoeba-Challenge/)
77
+ - [HPLT bilingual data v1 (as part of the Tatoeba Translation Challenge dataset)](https://hplt-project.org/datasets/v1)
78
+ - [A massively parallel Bible corpus](https://aclanthology.org/L14-1215/)
79
+
80
+ This is a multilingual translation model with multiple target languages. A sentence initial language token is required in the form of `>>id<<` (id = valid target language ID), e.g. `>>fra<<`
81
+
82
+ ## Uses
83
+
84
+ This model can be used for translation and text-to-text generation.
85
+
86
+ ## Risks, Limitations and Biases
87
+
88
+ **CONTENT WARNING: Readers should be aware that the model is trained on various public data sets that may contain content that is disturbing, offensive, and can propagate historical and current stereotypes.**
89
+
90
+ Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)).
91
+
92
+ ## How to Get Started With the Model
93
+
94
+ A short example code:
95
+
96
+ ```python
97
+ from transformers import MarianMTModel, MarianTokenizer
98
+
99
+ src_text = [
100
+ ">>ita<< Hány fényképet csináltatok?",
101
+ ">>ita<< Gazdag vagyok."
102
+ ]
103
+
104
+ model_name = "pytorch-models/opus-mt-tc-bible-big-fiu-fra_ita_por_spa"
105
+ tokenizer = MarianTokenizer.from_pretrained(model_name)
106
+ model = MarianMTModel.from_pretrained(model_name)
107
+ translated = model.generate(**tokenizer(src_text, return_tensors="pt", padding=True))
108
+
109
+ for t in translated:
110
+ print( tokenizer.decode(t, skip_special_tokens=True) )
111
+
112
+ # expected output:
113
+ # Quante foto avete scattato?
114
+ # Sono ricco.
115
+ ```
116
+
117
+ You can also use OPUS-MT models with the transformers pipelines, for example:
118
+
119
+ ```python
120
+ from transformers import pipeline
121
+ pipe = pipeline("translation", model="Helsinki-NLP/opus-mt-tc-bible-big-fiu-fra_ita_por_spa")
122
+ print(pipe(">>ita<< Hány fényképet csináltatok?"))
123
+
124
+ # expected output: Quante foto avete scattato?
125
+ ```
126
+
127
+ ## Training
128
+
129
+ - **Data**: opusTCv20230926max50+bt+jhubc ([source](https://github.com/Helsinki-NLP/Tatoeba-Challenge))
130
+ - **Pre-processing**: SentencePiece (spm32k,spm32k)
131
+ - **Model Type:** transformer-big
132
+ - **Original MarianNMT Model**: [opusTCv20230926max50+bt+jhubc_transformer-big_2024-08-17.zip](https://object.pouta.csc.fi/Tatoeba-MT-models/fiu-fra+ita+por+spa/opusTCv20230926max50+bt+jhubc_transformer-big_2024-08-17.zip)
133
+ - **Training Scripts**: [GitHub Repo](https://github.com/Helsinki-NLP/OPUS-MT-train)
134
+
135
+ ## Evaluation
136
+
137
+ * [Model scores at the OPUS-MT dashboard](https://opus.nlpl.eu/dashboard/index.php?pkg=opusmt&test=all&scoreslang=all&chart=standard&model=Tatoeba-MT-models/fiu-fra%2Bita%2Bpor%2Bspa/opusTCv20230926max50%2Bbt%2Bjhubc_transformer-big_2024-08-17)
138
+ * test set translations: [opusTCv20230926max50+bt+jhubc_transformer-big_2024-08-17.test.txt](https://object.pouta.csc.fi/Tatoeba-MT-models/fiu-fra+ita+por+spa/opusTCv20230926max50+bt+jhubc_transformer-big_2024-08-17.test.txt)
139
+ * test set scores: [opusTCv20230926max50+bt+jhubc_transformer-big_2024-08-17.eval.txt](https://object.pouta.csc.fi/Tatoeba-MT-models/fiu-fra+ita+por+spa/opusTCv20230926max50+bt+jhubc_transformer-big_2024-08-17.eval.txt)
140
+ * benchmark results: [benchmark_results.txt](benchmark_results.txt)
141
+ * benchmark output: [benchmark_translations.zip](benchmark_translations.zip)
142
+
143
+ | langpair | testset | chr-F | BLEU | #sent | #words |
144
+ |----------|---------|-------|-------|-------|--------|
145
+ | multi-multi | tatoeba-test-v2020-07-28-v2023-09-26 | 0.67829 | 48.5 | 10000 | 65198 |
146
+
147
+ ## Citation Information
148
+
149
+ * Publications: [Democratizing neural machine translation with OPUS-MT](https://doi.org/10.1007/s10579-023-09704-w) and [OPUS-MT – Building open translation services for the World](https://aclanthology.org/2020.eamt-1.61/) and [The Tatoeba Translation Challenge – Realistic Data Sets for Low Resource and Multilingual MT](https://aclanthology.org/2020.wmt-1.139/) (Please, cite if you use this model.)
150
+
151
+ ```bibtex
152
+ @article{tiedemann2023democratizing,
153
+ title={Democratizing neural machine translation with {OPUS-MT}},
154
+ author={Tiedemann, J{\"o}rg and Aulamo, Mikko and Bakshandaeva, Daria and Boggia, Michele and Gr{\"o}nroos, Stig-Arne and Nieminen, Tommi and Raganato, Alessandro and Scherrer, Yves and Vazquez, Raul and Virpioja, Sami},
155
+ journal={Language Resources and Evaluation},
156
+ number={58},
157
+ pages={713--755},
158
+ year={2023},
159
+ publisher={Springer Nature},
160
+ issn={1574-0218},
161
+ doi={10.1007/s10579-023-09704-w}
162
+ }
163
+
164
+ @inproceedings{tiedemann-thottingal-2020-opus,
165
+ title = "{OPUS}-{MT} {--} Building open translation services for the World",
166
+ author = {Tiedemann, J{\"o}rg and Thottingal, Santhosh},
167
+ booktitle = "Proceedings of the 22nd Annual Conference of the European Association for Machine Translation",
168
+ month = nov,
169
+ year = "2020",
170
+ address = "Lisboa, Portugal",
171
+ publisher = "European Association for Machine Translation",
172
+ url = "https://aclanthology.org/2020.eamt-1.61",
173
+ pages = "479--480",
174
+ }
175
+
176
+ @inproceedings{tiedemann-2020-tatoeba,
177
+ title = "The Tatoeba Translation Challenge {--} Realistic Data Sets for Low Resource and Multilingual {MT}",
178
+ author = {Tiedemann, J{\"o}rg},
179
+ booktitle = "Proceedings of the Fifth Conference on Machine Translation",
180
+ month = nov,
181
+ year = "2020",
182
+ address = "Online",
183
+ publisher = "Association for Computational Linguistics",
184
+ url = "https://aclanthology.org/2020.wmt-1.139",
185
+ pages = "1174--1182",
186
+ }
187
+ ```
188
+
189
+ ## Acknowledgements
190
+
191
+ The work is supported by the [HPLT project](https://hplt-project.org/), funded by the European Union’s Horizon Europe research and innovation programme under grant agreement No 101070350. We are also grateful for the generous computational resources and IT infrastructure provided by [CSC -- IT Center for Science](https://www.csc.fi/), Finland, and the [EuroHPC supercomputer LUMI](https://www.lumi-supercomputer.eu/).
192
+
193
+ ## Model conversion info
194
+
195
+ * transformers version: 4.45.1
196
+ * OPUS-MT git hash: 0882077
197
+ * port time: Tue Oct 8 11:00:58 EEST 2024
198
+ * port machine: LM0-400-22516.local
benchmark_results.txt ADDED
@@ -0,0 +1 @@
 
 
1
+ multi-multi tatoeba-test-v2020-07-28-v2023-09-26 0.67829 48.5 10000 65198
benchmark_translations.zip ADDED
File without changes
config.json ADDED
@@ -0,0 +1,41 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "pytorch-models/opus-mt-tc-bible-big-fiu-fra_ita_por_spa",
3
+ "activation_dropout": 0.0,
4
+ "activation_function": "relu",
5
+ "architectures": [
6
+ "MarianMTModel"
7
+ ],
8
+ "attention_dropout": 0.0,
9
+ "bos_token_id": 0,
10
+ "classifier_dropout": 0.0,
11
+ "d_model": 1024,
12
+ "decoder_attention_heads": 16,
13
+ "decoder_ffn_dim": 4096,
14
+ "decoder_layerdrop": 0.0,
15
+ "decoder_layers": 6,
16
+ "decoder_start_token_id": 59779,
17
+ "decoder_vocab_size": 59780,
18
+ "dropout": 0.1,
19
+ "encoder_attention_heads": 16,
20
+ "encoder_ffn_dim": 4096,
21
+ "encoder_layerdrop": 0.0,
22
+ "encoder_layers": 6,
23
+ "eos_token_id": 582,
24
+ "forced_eos_token_id": null,
25
+ "init_std": 0.02,
26
+ "is_encoder_decoder": true,
27
+ "max_length": null,
28
+ "max_position_embeddings": 1024,
29
+ "model_type": "marian",
30
+ "normalize_embedding": false,
31
+ "num_beams": null,
32
+ "num_hidden_layers": 6,
33
+ "pad_token_id": 59779,
34
+ "scale_embedding": true,
35
+ "share_encoder_decoder_embeddings": true,
36
+ "static_position_embeddings": true,
37
+ "torch_dtype": "float32",
38
+ "transformers_version": "4.45.1",
39
+ "use_cache": true,
40
+ "vocab_size": 59780
41
+ }
generation_config.json ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bad_words_ids": [
4
+ [
5
+ 59779
6
+ ]
7
+ ],
8
+ "bos_token_id": 0,
9
+ "decoder_start_token_id": 59779,
10
+ "eos_token_id": 582,
11
+ "forced_eos_token_id": 582,
12
+ "max_length": 512,
13
+ "num_beams": 4,
14
+ "pad_token_id": 59779,
15
+ "transformers_version": "4.45.1"
16
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:41abd5e4bb88c8e15a2cd52274abf5d36c88a74336f0523de00e2d8c8a8cdaa3
3
+ size 950557120
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1118a000c994e8571feb81a68d1dc9d76f7ad029518e0726a55aa03389fbdcce
3
+ size 950608389
source.spm ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:82528c3ef63b9b0abe90ff8d04bbc6f3936d6eafc7b33dbb121daf1e3fb6e215
3
+ size 822999
special_tokens_map.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"eos_token": "</s>", "unk_token": "<unk>", "pad_token": "<pad>"}
target.spm ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b4acacc1cb3c1f7cbfed7aad8bf1b8dfeb03c0245c4e6ae3a3b64d86816b2243
3
+ size 819902
tokenizer_config.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"source_lang": "fiu", "target_lang": "fra+ita+por+spa", "unk_token": "<unk>", "eos_token": "</s>", "pad_token": "<pad>", "model_max_length": 512, "sp_model_kwargs": {}, "separate_vocabs": false, "special_tokens_map_file": null, "name_or_path": "marian-models/opusTCv20230926max50+bt+jhubc_transformer-big_2024-08-17/fiu-fra+ita+por+spa", "tokenizer_class": "MarianTokenizer"}
vocab.json ADDED
The diff for this file is too large to render. See raw diff