Triangle104 commited on
Commit
b6304a4
1 Parent(s): 9128ad2

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +0 -71
README.md CHANGED
@@ -47,77 +47,6 @@ tags:
47
  This model was converted to GGUF format from [`utter-project/EuroLLM-9B-Instruct`](https://huggingface.co/utter-project/EuroLLM-9B-Instruct) using llama.cpp via the ggml.ai's [GGUF-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo) space.
48
  Refer to the [original model card](https://huggingface.co/utter-project/EuroLLM-9B-Instruct) for more details on the model.
49
 
50
- ---
51
- Model details:
52
- -
53
- This is the model card for EuroLLM-9B-Instruct. You can also check the pre-trained version: EuroLLM-9B.
54
-
55
-
56
- Developed by: Unbabel, Instituto Superior Técnico,
57
- Instituto de Telecomunicações, University of Edinburgh, Aveni,
58
- University of Paris-Saclay, University of Amsterdam, Naver Labs,
59
- Sorbonne Université.
60
- Funded by: European Union.
61
- Model type: A 9B parameter multilingual transfomer LLM.
62
- Language(s) (NLP): Bulgarian, Croatian, Czech,
63
- Danish, Dutch, English, Estonian, Finnish, French, German, Greek,
64
- Hungarian, Irish, Italian, Latvian, Lithuanian, Maltese, Polish,
65
- Portuguese, Romanian, Slovak, Slovenian, Spanish, Swedish, Arabic,
66
- Catalan, Chinese, Galician, Hindi, Japanese, Korean, Norwegian, Russian,
67
- Turkish, and Ukrainian.
68
- License: Apache License 2.0.
69
-
70
-
71
-
72
-
73
-
74
-
75
-
76
- Model Details
77
-
78
-
79
-
80
-
81
- The EuroLLM project has the goal of creating a suite of LLMs capable
82
- of understanding and generating text in all European Union languages as
83
- well as some additional relevant languages.
84
- EuroLLM-9B is a 9B parameter model trained on 4 trillion tokens divided
85
- across the considered languages and several data sources: Web data,
86
- parallel data (en-xx and xx-en), and high-quality datasets.
87
- EuroLLM-9B-Instruct was further instruction tuned on EuroBlocks, an
88
- instruction tuning dataset with focus on general instruction-following
89
- and machine translation.
90
-
91
-
92
-
93
-
94
-
95
-
96
-
97
- Model Description
98
-
99
-
100
-
101
-
102
- EuroLLM uses a standard, dense Transformer architecture:
103
-
104
-
105
- We use grouped query attention (GQA) with 8 key-value heads, since
106
- it has been shown to increase speed at inference time while maintaining
107
- downstream performance.
108
- We perform pre-layer normalization, since it improves the training stability, and use the RMSNorm, which is faster.
109
- We use the SwiGLU activation function, since it has been shown to lead to good results on downstream tasks.
110
- We use rotary positional embeddings (RoPE) in every layer, since
111
- these have been shown to lead to good performances while allowing the
112
- extension of the context length.
113
-
114
-
115
- For pre-training, we use 400 Nvidia H100 GPUs of the Marenostrum 5
116
- supercomputer, training the model with a constant batch size of 2,800
117
- sequences, which corresponds to approximately 12 million tokens, using
118
- the Adam optimizer, and BF16 precision.
119
-
120
- ---
121
  ## Use with llama.cpp
122
  Install llama.cpp through brew (works on Mac and Linux)
123
 
 
47
  This model was converted to GGUF format from [`utter-project/EuroLLM-9B-Instruct`](https://huggingface.co/utter-project/EuroLLM-9B-Instruct) using llama.cpp via the ggml.ai's [GGUF-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo) space.
48
  Refer to the [original model card](https://huggingface.co/utter-project/EuroLLM-9B-Instruct) for more details on the model.
49
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
50
  ## Use with llama.cpp
51
  Install llama.cpp through brew (works on Mac and Linux)
52