pinzhenchen commited on
Commit
497f58b
1 Parent(s): b651546

add model in HF format

Browse files
Files changed (1) hide show
  1. README.md +34 -9
README.md CHANGED
@@ -9,7 +9,7 @@ license: cc-by-4.0
9
 
10
  ## HPLT MT release v1.0
11
 
12
- This repository contains the translation model for en-ar trained with OPUS and HPLT data. For usage instructions, evaluation scripts, and inference scripts, please refer to the [HPLT-MT-Models v1.0](https://github.com/hplt-project/HPLT-MT-Models/tree/main/v1.0) GitHub repository.
13
 
14
  ### Model Info
15
 
@@ -18,26 +18,51 @@ This repository contains the translation model for en-ar trained with OPUS and H
18
  * Dataset: OPUS and HPLT data
19
  * Model architecture: Transformer-base
20
  * Tokenizer: SentencePiece (Unigram)
21
- * Cleaning: We used OpusCleaner with a set of basic rules. Details can be found in the filter files in [Github](https://github.com/hplt-project/HPLT-MT-Models/tree/main/v1.0/data/ar-en/raw/v2)
22
 
23
- You can also read our deliverable report [here](https://hplt-project.org/HPLT_D5_1___Translation_models_for_select_language_pairs.pdf) for more details.
24
 
25
  ### Usage
26
 
27
 
28
- The model has been trained with Marian. To run inference, refer to the [Inference/Decoding/Translation](https://github.com/hplt-project/HPLT-MT-Models/tree/main/v1.0#inferencedecodingtranslation) section of our GitHub repository.
29
 
30
- The model can be used with the Hugging Face framework if the weights are converted to the Hugging Face format. We might provide this in the future; contributions are also welcome.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
31
 
32
  ### Benchmarks
33
 
34
- | testset | BLEU | chrF++ | COMET22 |
 
 
35
  | -------------------------------------- | ---- | ----- | ----- |
36
- | flores200 | 29.2 | 56.9 | 0.8542 |
37
- | ntrex | 25.8 | 52.2 | 0.8245 |
38
 
39
  ### Acknowledgements
40
 
41
  This project has received funding from the European Union's Horizon Europe research and innovation programme under grant agreement No 101070350 and from UK Research and Innovation (UKRI) under the UK government's Horizon Europe funding guarantee [grant number 10052546]
42
 
43
- Brought to you by researchers from the University of Edinburgh, Charles University in Prague, and the whole HPLT consortium.
 
9
 
10
  ## HPLT MT release v1.0
11
 
12
+ This repository contains the translation model for English-Arabic trained with OPUS and HPLT data. The model is available in both Marian and Hugging Face formats.
13
 
14
  ### Model Info
15
 
 
18
  * Dataset: OPUS and HPLT data
19
  * Model architecture: Transformer-base
20
  * Tokenizer: SentencePiece (Unigram)
21
+ * Cleaning: We used [OpusCleaner](https://github.com/hplt-project/OpusCleaner) with a set of basic rules. Details can be found in the filter files [here](https://github.com/hplt-project/HPLT-MT-Models/tree/main/v1.0/data/ar-en/raw/v2).
22
 
23
+ You can check out our [deliverable report](https://hplt-project.org/HPLT_D5_1___Translation_models_for_select_language_pairs.pdf), [GitHub repository](https://github.com/hplt-project/HPLT-MT-Models/tree/main/v1.0), and [website](https://hplt-project.org) for more details.
24
 
25
  ### Usage
26
 
27
 
28
+ The model has been trained with [MarianNMT](https://github.com/marian-nmt/marian) and the weights are in the Marian format. We have also converted the model into the Hugging Face format so it is compatible with `transformers`.
29
 
30
+ #### Using Marian
31
+ To run inference with MarianNMT, refer to the [Inference/Decoding/Translation](https://github.com/hplt-project/HPLT-MT-Models/tree/main/v1.0#inferencedecodingtranslation) section of our GitHub repository. You will need the model file `model.npz.best-chrf.npz` and the vocabulary file `model.en-ar.spm` from this repository.
32
+
33
+ #### Using transformers
34
+ We have also converted this model to the Hugging Face format and you can get started with the script below. **Note** that due a [known issue](https://github.com/huggingface/transformers/issues/26216) in weight conversion, the checkpoint cannot work with transformer versions <4.26 or >4.30. We tested and suggest `pip install transformers==4.28`.
35
+
36
+ ```
37
+ from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
38
+
39
+ tokenizer = AutoTokenizer.from_pretrained("HPLT/translate-en-ar-v1.0-hplt_opus")
40
+ model = AutoModelForSeq2SeqLM.from_pretrained("HPLT/translate-en-ar-v1.0-hplt_opus")
41
+
42
+ inputs = ["Input goes here.", "Make sure the language is right."]
43
+ batch_tokenized = tokenizer(inputs, return_tensors="pt", padding=True)
44
+ model_output = model.generate(
45
+ **batch_tokenized, num_beams=6, max_new_tokens=512
46
+ )
47
+ batch_detokenized = tokenizer.batch_decode(
48
+ model_output,
49
+ skip_special_tokens=True,
50
+ )
51
+
52
+ print(batch_detokenized)
53
+ ```
54
 
55
  ### Benchmarks
56
 
57
+ When decoded using Marian, the model has the following test scores.
58
+
59
+ | Test set | BLEU | chrF++ | COMET22 |
60
  | -------------------------------------- | ---- | ----- | ----- |
61
+ | FLORES200 | 29.2 | 56.9 | 0.8542 |
62
+ | NTREX | 25.8 | 52.2 | 0.8245 |
63
 
64
  ### Acknowledgements
65
 
66
  This project has received funding from the European Union's Horizon Europe research and innovation programme under grant agreement No 101070350 and from UK Research and Innovation (UKRI) under the UK government's Horizon Europe funding guarantee [grant number 10052546]
67
 
68
+ Brought to you by researchers from the University of Edinburgh and Charles University in Prague with support from the whole HPLT consortium.