ALIA Spanish Cultural and Heritage 7B Base Model

This repository contains a domain-adapted version of the Salamandra 7B model, optimized for the Spanish cultural heritage domain.

This model is the result of a continual pre-training process on the Salamandra 7B model, followed by instruction tuning using curated cultural heritage corpora.

DISCLAIMER: This model is a domain-specific proof-of-concept for cultural heritage use. It has NOT been validated for production use in institutional, archival or heritage-critical workflows. It may produce incorrect, incomplete, or misleading information about cultural assets, historical sources or heritage documentation. Users should verify outputs against authoritative sources.

Model Details

Description

This model is a Transformer-based decoder-only language model that builds on the Salamandra 7B architecture through domain adaptation targeted to Spanish cultural heritage.

Continual Pre-training (CPT): The base model was further pre-trained on the SINAI/ALIA-es-cultural-heritage corpus to adapt its weights to the specific vocabulary and structures of cultural and heritage Spanish texts.

Architecture


Base Model	Salamandra 7B
Total Parameters	7,768,117,248
Embedding Parameters	1,048,576,000
Layers	32
Hidden size	4,096
Attention heads	32
Context length	8,192
Vocabulary size	256,000
Precision	bfloat16
Embedding type	RoPE
Activation Function	SwiGLU
Layer normalization	RMS Norm
Flash attention	✅
Grouped Query Attention	✅
Num. query groups	8

Hyperparameters

Parameter	Value
Sequence length	8,192
Num. epochs	2
Learning rate	1.63e-5
Optimizer	adamw_torch_fused
Adam beta1	0.9
Adam beta2	0.9435750969
Adam epsilon	1e-8
LR scheduler	cosine
Cosine min. LR ratio	0.2098344114
Warmup ratio	0.001567993392
Weight decay	1.99373126
NEFTune noise alpha	5
Max grad norm	0.07158097239
Micro batch size	8
Gradient accumulation steps	2
Eval batch size	1
Gradient checkpointing	true
Distributed backend	nccl
Val set size	1,000
Eval table size	5
Eval max new tokens	100
Do causal LM eval	true
BF16	true
FP16	false
Seed	42
Strict	false

Intended Use

Direct Use

The model is intended for research and development applications within Spanish cultural and heritage natural language processing. Representative use cases include:

Summarization of heritage documents, museum records and historical publications for research purposes.
Question answering over cultural and archival texts to support information retrieval.
Plain-language explanations of heritage content for dissemination and educational use.

Out-of-scope Use

This model is not approved for operational decision-making in heritage institutions. It must not be used as the sole source for cataloguing, legal attribution, conservation, provenance assessment or other actions that directly affect cultural assets. Any deployment that impacts heritage management requires expert validation and review.

How to use

Python Example

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "SINAI/ALIA-es-cultural-heritage-7B-Base"

prompt = "Resume la importancia histórica y patrimonial del Castillo de Jaén."

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16
)

inputs = tokenizer.encode(prompt, return_tensors="pt")
inputs = inputs.to(model.device)
outputs = model.generate(input_ids=inputs, max_new_tokens=200)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Data

Domain Adaptation Data

To adapt the model to Spanish cultural and heritage texts we used the following resources created and curated by the SINAI Research Group (Universidad de Jaén):

Continual Pre-training
- Dataset: SINAI/ALIA-es-cultural-heritage
- Description: A large collection of cultural and heritage Spanish texts, including heritage inventories, journals, archival records, institutional publications and reference resources used to adapt the base model's language distribution to the cultural and heritage domain.

Original Pre-training Data (Base Model)

The underlying base model (Salamandra 7B) was pre-trained on 12.875 trillion tokens of highly curated data, covering 35 European languages and code. For a full detailed list of the original pre-training sources, please refer to the Original Salamandra Model Card.

Additional Information

License

Apache License, Version 2.0

Citation

@misc{ALIA-es-cultural-heritage-7B-Base,
    title={ALIA Spanish Cultural and Heritage 7B Base Model},
    author={SINAI Research Group},
    year={2026},
    publisher={HuggingFace},
    howpublished={\url{https://huggingface.co/datasets/SINAI/ALIA-es-cultural-heritage-7B-Base}}
}

Please also cite the base models:

@misc{gonzalezagirre2025salamandratechnicalreport,
      title={Salamandra Technical Report}, 
      author={Aitor Gonzalez-Agirre and Marc Pàmies and Joan Llop and Irene Baucells and Severino Da Dalt and Daniel Tamayo and José Javier Saiz and Ferran Espuña and Jaume Prats and Javier Aula-Blasco and Mario Mina and Adrián Rubio and Alexander Shvets and Anna Sallés and Iñaki Lacunza and Iñigo Pikabea and Jorge Palomar and Júlia Falcão and Lucía Tormo and Luis Vasquez-Reina and Montserrat Marimon and Valle Ruíz-Fernández and Marta Villegas},
      year={2025},
      eprint={2502.08489},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.08489}, 
}

Funding

This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project ALIA.

Acknowledgments

Training of this model was conducted thanks to BSC (Barcelona Supercomputing Center) on MareNostrum 5, a pre-exascale EuroHPC supercomputer hosted and operated by them.

Contact: ALIA Project - SINAI Research Group - Universidad de Jaén

More Information: SINAI Research Group | ALIA-UJA Project