Hugging Face's logo

HiTZ
/
latxa-7b-v1.1

Text Generation
Transformers
PyTorch
Basque
English
llama
Eval Results
text-generation-inference
Inference Endpoints
Model card Files Community
latxa-7b-v1.1 / README.md
OSainz's picture
OSainz
Update README.md
864824d verified
preview code
|
raw
history blame contribute delete
15.8 kB
metadata
license: llama2
datasets:
  - HiTZ/latxa-corpus-v1.1
language:
  - eu
  - en
metrics:
  - accuracy
  - f1
  - perplexity
pipeline_tag: text-generation
model-index:
  - name: Latxa-7b-v1.1
    results:
      - task:
          type: multiple-choice
        dataset:
          name: xstory_cloze
          type: XStory
        metrics:
          - name: Accuracy (0-shot)
            type: Accuracy (0-shot)
            value: 65.72
        source:
          name: Paper
          url: https://arxiv.org/abs/2403.20266
      - task:
          type: multiple-choice
        dataset:
          name: belebele
          type: Belebele
        metrics:
          - name: Accuracy (5-shot)
            type: Accuracy (5-shot)
            value: 36.89
        source:
          name: Paper
          url: https://arxiv.org/abs/2403.20266
      - task:
          type: mix
        dataset:
          name: basque_glue
          type: BasqueGLUE
        metrics:
          - name: Average scores (5-shot)
            type: Average scores (5-shot)
            value: 51.78
        source:
          name: Paper
          url: https://arxiv.org/abs/2403.20266
      - task:
          type: multiple_choice
        dataset:
          name: eus_proficiency
          type: EusProficiency
        metrics:
          - name: Accuracy (5-shot)
            type: Accuracy (5-shot)
            value: 32.44
        source:
          name: Paper
          url: https://arxiv.org/abs/2403.20266
      - task:
          type: multiple_choice
        dataset:
          name: eus_reading
          type: EusReading
        metrics:
          - name: Accuracy (5-shot)
            type: Accuracy (5-shot)
            value: 30.4
        source:
          name: Paper
          url: https://arxiv.org/abs/2403.20266
      - task:
          type: multiple_choice
        dataset:
          name: eus_trivia
          type: EusTrivia
        metrics:
          - name: Accuracy (5-shot)
            type: Accuracy (5-shot)
            value: 44.37
        source:
          name: Paper
          url: https://arxiv.org/abs/2403.20266
      - task:
          type: multiple_choice
        dataset:
          name: eus_exams
          type: EusExams
        metrics:
          - name: Accuracy (5-shot)
            type: Accuracy (5-shot)
            value: 34.2
        source:
          name: Paper
          url: https://arxiv.org/abs/2403.20266

Model Card for Latxa 7b

IMPORTANT: This model is outdated and made available publicly for reproducibility purposes only. Please utilize the most recent version found in our HuggingFace collection.

We introduce Latxa, a family of large language models for Basque ranging from 7 to 70 billion parameters. Latxa is based on Llama 2, which we continue pretraining on a new Basque corpus comprising 4.3M documents and 4.2B tokens. In our extensive evaluation, Latxa outperforms all previous open models we compare to by a large margin. In addition, it is competitive with GPT-4 Turbo in language proficiency and understanding, despite lagging behind in reading comprehension and knowledgeintensive tasks. Both the Latxa family of models, as well as our new pretraining corpora and evaluation datasets, are publicly available under open licenses. Our suite enables reproducible research on methods to build LLMs for low-resource languages

Model Details

Model Description

Latxa is a family of Large Language Models (LLM) based on Meta’s LLaMA models. Current LLMs exhibit incredible performance for high-resource languages such as English, but, in the case of Basque and other low-resource languages, their performance is close to a random guesser. These limitations widen the gap between high- and low-resource languages when it comes to digital development. We present Latxa to overcome these limitations and promote the development of LLM-based technology and research for the Basque language. Latxa models follow the same architecture as their original counterparts and were further trained in Latxa Corpus v1.1, a high-quality Basque corpora.

The models are released in three sizes: 7B, 13B and 70B.

  • Developed by: HiTZ Research Center & IXA Research group (University of the Basque Country UPV/EHU)
  • Model type: Language model
  • Language(s) (NLP): en, eu
  • License: llama2
  • Parent Model: meta-llama/Llama-2-7b
  • Contact: hitz@ehu.eus

Getting started

Use the code below to get started with the model.


from transformers import pipeline

pipe = pipeline("text-generation", model="HiTZ/latxa-7b-v1.1")

text = "Euskara adimen artifizialera iritsi da!"

pipe(text, max_new_tokens=50, num_beams=5)

>> [
 {
  'generated_text': 'Euskara adimen artifizialera iritsi da!\nEuskararen eta adimen artifizialaren arteko harremana aspaldikoa da,'
  ' baina azken urteotan aurrerapauso handiak eman dira arlo horretan'
 }
]

Uses

Latxa models are intended to be used with Basque data; for any other language the performance is not guaranteed. Same as the original, Latxa inherits the LLaMA-2 License which allows for commercial and research use.

Direct Use

Latxa family models are pre-trained LLMs without any task-specific or instruction fine-tuning. That is, the model can either be prompted to perform a specific task or further fine-tuned for specific use cases.

Out-of-Scope Use

The model was not fine-tuned to follow instructions or to work as a chat assistant, therefore, this kind of usage is not tested nor recommended.

Bias, Risks, and Limitations

In an effort to alleviate the potentially disturbing or harmful content, Latxa has been trained on carefully selected and processed data which comes mainly from local media, national/regional newspapers, encyclopedias and blogs (see Latxa-Corpus below). Still, the model is based on LLaMA models and can potentially carry the same bias, risk and limitations.

Please see the LLaMA’s Ethical Considerations and Limitations for further information.

Training Details

Training Data

Our training corpus combines various existing datasets, as well as some new ones that we release with this work. We have prioritized quality over quantity when constructing our corpus, prioritizing high-quality data sources and applying a thorough deduplication and filtering process. In total, a 4.17B tokens corpus is used to train the model.

See more details in the Latxa Corpus dataset card.

Additionally, 500K documents of English data randomly selected from the Pile dataset were also included to avoid catastrophic forgetting.

Training Procedure

The training of Latxa was conducted using the GPT-Neox library. As infrastructure, we leveraged the CINECA HPC Leonardo computing cluster located in Italy, which is powered by 3456 nodes each containing 4x custom A100 64Gb GPUs. The models were trained for 10k steps with a sequence length of 4096 tokens and an effective batch size of 2M tokens, resulting in a total of 20B tokens (around 4 epochs). We used a cosine learning rate schedule, with a warm-up of 500 steps and decaying down to 3% of the peak learning rate. We set up the peak learning rate to be 1e-4. All other hyperparameters follow (Touvron et al., 2023).

Evaluation

We evaluated the models on zero-shot and few-shot settings on generative, multiple-choice and classification tasks. We used the basque partitions of each dataset.

Testing Data, Factors & Metrics

Testing Data

Metrics

For most of the task we used Accuracy, as they are framed as Multiple Choice questions. For the rest, particularly task from BasqueGLUE benchmark, we have used the following:

  • Micro F1: BEC2016-eu and BHTCv2
  • Macro F1: VaxxStance (favor & against)

Results

The model was evaluated using the LM Evaluation harness library from Eleuther AI. In order to reproduce our results please follow the instructions in Latxa's Github repository.

Model Size XStory Belebele BasGLUE EusProf EusRead EusTrivia EusExams Avg
Random 50.00 25.00 37.50 25.00 25.83 26.55 25.00 30.70
GPT 3.5 Turbo n/a -- 57.33 48.62 31.24 36.65 46.71 42.42 --
GPT 4 Turbo n/a -- 90.67 62.90 56.70 75.85 73.12 70.22 --
XGLM 7B 57.71 23.88 41.47 22.96 24.43 26.53 24.59 32.51
BLOOM 7B 57.18 27.00 40.17 25.34 28.41 27.17 25.07 33.86
Mistral 7B 51.09 38.89 39.22 25.01 29.26 34.58 32.15 35.94
Llama 2 7B 50.43 26.22 38.20 24.09 27.27 29.50 28.84 32.51
Latxa v1 7B 63.13 35.67 50.26 28.19 27.27 40.17 34.18 39.84
Latxa v1.1 7B 65.72 36.89 51.78 32.44 30.40 44.37 34.20 42.26
mGPT 13B 55.39 25.00 37.56 25.00 24.15 27.17 25.73 32.14
Llama 2 13B 50.63 32.00 38.98 25.90 28.98 33.53 29.66 34.36
Latxa v1 13B 65.85 53.56 54.49 41.19 40.06 51.14 42.92 49.95
Latxa v1.1 13B 67.24 51.56 54.04 45.02 29.83 56.44 43.18 49.62
Mixtral 8x7B 52.55 50.44 45.00 26.43 37.50 42.51 39.87 41.97
Yi 34B 52.22 54.56 43.90 27.30 34.66 42.57 39.68 42.05
Llama 2 70B 51.62 33.56 42.55 24.16 27.84 38.43 33.08 35.47
Latxa v1 70B 67.57 71.78 59.37 48.19 49.72 57.84 51.68 58.02
Latxa v1.1 70B 69.76 64.89 61.66 60.61 53.69 61.52 54.48 60.94

Environmental Impact

Carbon emissions are estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

Model Size Time (GPU Hours) Carbon Emitted (kg CO2 eq)
Latxa v1.1 7B 952.5h 124.47kg
Latxa v1.1 13B 2,518.0h 329.06kg
Latxa v1.1 70B 30,266.0h 3,955.17kg
Total - 33,636.5h 4,408,7kg
  • Hardware Type: HPC Cluster, 4x A100 64Gb nodes
  • Hours used: 33,636.5h
  • Compute cluster: CINECA HPC
  • Compute Region: Italy
  • Carbon Emitted: 4,408,7kg CO2 eq

Acknowledgements

This work has been partially supported by the Basque Government (IKER-GAITU project). It has also been partially supported by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project with reference 2022/TL22/00215335. The models were trained on the Leonardo supercomputer at CINECA under the EuroHPC Joint Undertaking, project EHPC-EXT-2023E01-013.

Citation

To cite our work, please use:

@misc{etxaniz2024latxa,
      title={{L}atxa: An Open Language Model and Evaluation Suite for {B}asque}, 
      author={Julen Etxaniz and Oscar Sainz and Naiara Perez and Itziar Aldabe and German Rigau and Eneko Agirre and Aitor Ormazabal and Mikel Artetxe and Aitor Soroa},
      year={2024},
      eprint={2403.20266},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}