---
license: llama3
base_model: meta-llama/Meta-Llama-3-8B
tags:
- llama
- llama-3
- Catalan
model-index:
- name: CataLlama-v0.1-Base
  results: []
datasets:
- catallama/Catalan-Raw-Text
language:
- ca
- en
pipeline_tag: text-generation
---

## Model Details

**CataLlama-v0.1-Base** is a fine-tune of [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) on the [catallama/Catalan-Raw-Text](https://huggingface.co/datasets/catallama/Catalan-Raw-Text) raw Catalan corpus dataset

The model shows improved proficiency with the Catalan language.

**This is a base model and it is not fine-tuned for downstream tasks.**

The model achieves a loss rate of 1.4608 on the validation dataset after one epoch.


**Model developers** [Laurentiu Petrea](https://www.linkedin.com/in/laurentiupetrea/) based on Llama-3 from Meta.

**Model Architecture** CataLlama is an auto-regressive language model that uses an optimized transformer architecture. The tuned versions use supervised fine-tuning (SFT) and direct preference optimisation (DPO) to align with human preferences for helpfulness and safety.

**License** The model uses the llama-3 license available at: [https://llama.meta.com/llama3/license](https://llama.meta.com/llama3/license)


### Use with transformers

See the snippet below for usage with Transformers:

```python
import transformers
import torch

model_id = "catallama/CataLlama-v0.1-Base"

pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)

outputs = pipeline("Ei com estàs avui?")

print(outputs[0]["generated_text"][len(prompt):])
```


## Training procedure

The model was trained **without a prompt template**, only with raw text separated by BOS and EOS tokens.

The model was trained for two epochs on **6x A100 80GB GPUs using DeepSpeed ZeRO** State-3 without CPU offloading.

Example:

```text
<|begin_of_text|>Després d'anys sense veure's, un grup d'amics es reuneix en una casa a la muntanya durant el cap de setmana. Sembla que res hagi canviat entre ells, però entre rialles i anècdotes s'amaga un tèrbol episodi del passat que els segueix turmentant. Tot d'una, un incident estrany altera els seus plans i queden completament aïllats i sense possibilitat de comunicació amb l'exterior. Els joves decideixen buscar ajuda, però en el camí el grup es va desintegrant, mentre el que sembla un nou ordre natural s'imposa davant dels seus ulls atònits. Obligat a fugir del seu Bangla Desh natal, el jove Fahim i el seu pare [..]<|end_of_text|>
<|begin_of_text|>Els contes que t'explicaven abans d'anar a dormir. Apropeu-vos a descobrir uns contes magnífics i viatgeu a un univers fascinant en què els somnis s'entrellacen amb la màgia i el misteri! En aquests moments s'ha exhaurit l'estoc d'aquest producte. Si vols ser el primer a conèixer quan en tornem a tenir, si us plau, escriu el teu correu al camp de baix.<|end_of_text|>
```

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-05
- distributed_type: multi-GPU
- num_devices: 6
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 5
- num_epochs: 1

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 1.5037        | 0.07  | 100  | 1.5150          |
| 1.5181        | 0.15  | 200  | 1.5097          |
| 1.4901        | 0.22  | 300  | 1.5041          |
| 1.5042        | 0.3   | 400  | 1.4980          |
| 1.494         | 0.37  | 500  | 1.4928          |
| 1.4462        | 0.45  | 600  | 1.4875          |
| 1.475         | 0.52  | 700  | 1.4822          |
| 1.4811        | 0.6   | 800  | 1.4774          |
| 1.4511        | 0.67  | 900  | 1.4730          |
| 1.4712        | 0.75  | 1000 | 1.4691          |
| 1.4531        | 0.82  | 1100 | 1.4656          |
| 1.4761        | 0.89  | 1200 | 1.4626          |
| 1.4636        | 0.97  | 1300 | 1.4608          |

## Intended Use

**Note:** This model is not intended to beat benchmarks, but to demonstrate techniques for augmenting LLMs on new languages and preserve rare languages as part of our world heritage.

**Intended Use Cases** Llama 3 is intended for commercial and research use in English. Instruction tuned models are intended for assistant-like chat, whereas pretrained models can be adapted for a variety of natural language generation tasks.

**Out-of-scope** Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in any other way that is prohibited by the Acceptable Use Policy and Llama 3 Community License. Use in languages other than English**.

**Note: Developers may fine-tune Llama 3 models for languages beyond English provided they comply with the Llama 3 Community License and the Acceptable Use Policy.