Marco-LLM-ES-7B

Introduction

Marco-LLM-ES is a series of enhanced language models specifically fine-tuned for common languages used in Spain, including Catalan, Basque, Galician, and Spanish. This repository contains the 7B Marco-LLM-ES base language model.

Compared with the state-of-the-art open-source language models, Marco-LLM-ES has undergone extensive continued pretraining on a dataset containing approximately 50 billion tokens, enhancing its capabilities in the targeted languages while maintaining competitiveness in general benchmarks.

For more details, please refer to our Hugging Face page.

Model Details

Marco-LLM-ES series includes models of varying sizes, from 7B to 72B parameters, including both base and instruction-tuned (Instruct) models. The models are based on the Transformer architecture with SwiGLU activation, attention QKV bias, and group query attention. Additionally, the models employ an improved tokenizer adaptive to multiple languages.

Usage

It is not advised to use the base language models for direct text generation tasks. Instead, it is recommended to apply post-training methods such as Supervised Fine-tuning (SFT), Reinforcement Learning with Human Feedback (RLHF), or continued pretraining to adapt the models for specific use cases.

Performance

The evaluation of Marco-LLM-ES models focuses on performance in natural language understanding, general question answering, coding, mathematics, scientific knowledge, reasoning, multilingual capability, and their enhanced performance in the targeted languages: Catalan, Basque, Galician, and Spanish.

The datasets used for evaluation include:

Spanish-specific Tasks: Evaluations in Catalan, Basque, Galician, and Spanish at LaLeaderboard (5-shot)

Marco-LLM-ES-7B performance

Datasets	Marco-LLM-ES-7B
Spanish	44.49
Catalan	39.45
Basque	28.66
Galician	24.04
Average	34.16

Citation

If you find our work helpful, please give us a citation.

@article{unique_identifier,

title={Marco-LLM: Bridging Languages via Massive Multilingual Training for Cross-Lingual Enhancement},

journal={arXiv},

volume={},

number={2412.04003},

year={2024},

url={https://arxiv.org/abs/2412.04003}

}