Model Card for OpenLLM-France/Luciole-8B-KD-Math-Base

Table of Contents

• Model Card for Luciole-23B-Base
  • Model Description
  • Uses
    • Direct use
    • Downstream use
    • Out-of-Scope Use
  • Bias, Risks, and Limitations
    • Recommendations
  • Example Code in Python
    • Load the model
    • Sentence completion
  • Loading Intermediate Checkpoints
  • Training Details
    • Evaluation
  • Citation
  • Acknowledgements
  • Contact

Model Description

Luciole-8B-KD-Math-Base is a causal language model developed by LINAGORA and the OpenLLM-France consortium as a part of the OpenLLM France project, funded by BPI France through the France 2030 program. It was trained on the GENCI supercomputer Jean Zay, managed by IDRIS.

Luciole-8B-KD-Math-Base was trained through knowledge distillation (KD) following the Minitron approach (Sreenivas et al., 2024. LLM Pruning and Distillation in Practice: The Minitron Approach). We began by continually pretraining Luciole-23B-Base on a math-heavy mix of around 50 billion tokens to create a teacher for knowledge distillation. The teacher was then pruned to a roughly 14 billion parameter model that was subsequently trained through knowledge distillation on around 112 tokens of the mix used for correction. A second stage of pruning and distillation was applied, still using the corrected 23B model as the teacher, to create Luciole-8B-KD-Math-Base.

This goal of this project was to see whether we could reuse Luciole-23B-Base to create a smaller model that improves on Luciole-8B-Base in mathematics and maintains as much of its general knowledge as possible, without requiring full pretraining.

• Developed by: LINAGORA
• Language(s) (NLP): Luciole-8B-KD-Math has been evaluated on English and French. It may also contains capacities in other languages from the Luciole-23B-Base training dataset.
• License: Apache 2.0

Uses

Direct Use

Luciole-8B-KD-Math-Base is a foundation language model trained solely to predict the most probable next word in a sequence. It is designed as the first brick in a more complex training pipeline that would include multitask training on diverse instructions or focused fine-tuning on select downstream tasks, as well as possible alignment for human preferences.

Downstream Use

While the teacher correction mix used for distillation of Luciole-8B-KD-Math-Base contains mostly English data, Luciole-23-Base underwent multilingual training. Luciole-8B-KD-Math-Base can thus be fine-tuned for downstream tasks centered on the generation of multilingual text, with a special focus on French and English.

Out-of-Scope Use

Luciole-8B-KD-Math-Base is not intended to generate text directly for end use cases. It must be fine-tuned first. Its pretraining is optimized for multilingual performance, especially in French and English, and might perform less well on other languages without additional training. While trained on a much higher proportion of code data than Luciole-23-Base, it is still not optimized for code generation tasks.

Bias, Risks, and Limitations

Luciole-23-Base was trained on large amounts of web data. Additionally, due to the scarcity of French textual non-web data published under open licenses, much of the French training data came from older works in the public domain that carry biases from other time periods. While we made efforts to reduce toxic and offensive content in the Luciole Training Dataset, Luciole-8B-KD-Math-Base may still generate such content due to its close relation to Luciole-23-Base. Filtering of the Luciole Training Dataset is an ongoing project to which we welcome contributions.

Recommendations

To limit the generation of undesirable content, it is advised to fine-tune Luciole-8B-KD-Math-Base through instruction and preference tuning (DPO, RLHF, etc.).

Example Code in Python

Load the model

Load the model (quantized version on GPU if possible, for efficient inference):

import transformers

model_name = "OpenLLM-France/Luciole-8B-KD-Math-Base"

tokenizer = transformers.AutoTokenizer.from_pretrained(model_name)
model = transformers.AutoModelForCausalLM.from_pretrained(model_name,
    device_map="auto",
    load_in_4bit=True       # For efficient inference, if quantization is supported by the GPU card
)

Sentence completion

Wrap the model in a text generation pipeline, and specify some generation parameters:

pipeline = transformers.pipeline("text-generation", model=model, tokenizer=tokenizer)

generation_kwargs = dict(
    num_return_sequences=1,               # Number of variants to generate.
    return_full_text= False,              # Do not include the prompt in the generated text.
    do_sample=True,
    temperature=1.0, top_p=1, top_k=None, # Sampling parameters.
    max_new_tokens=200,                   # Maximum length for the output text (in number of tokens).
)

Try 1-shot question answering:

prompt = """\
Quelle est la capitale de l'Espagne ? Madrid\n\
Quelle est la capitale de la France ?\
"""
completions = pipeline(prompt, **generation_kwargs)
for completion in completions:
    print(prompt + "[…]" + completion['generated_text'])

This will print something like:

Quelle est la capitale de l'Espagne ? Madrid
Quelle est la capitale de la France ?[…] Paris
Quelle est la capitale du Brésil ? Brasilia
Quelle est la capitale de la Belgique ? Bruxelles
Quelle est la capitale de l'Italie ? Rome
...

Loading Intermediate Checkpoints

Coming soon.

Training Details

Coming soon.

Evaluation

Luciole-8B-KD-Math-Base breaks away from Luciole-8B-Base, Llama-3.1-8B and EuroLLM-9B to obtain scores closer to Gemma2-9B and Olmo2-7B on the following five mathematical benchmarks:

The figure below shows that Luciole-8B-KD-Math-Base (dark blue) maintains almost all of the mathematical performance of the corrected 23B teacher (dark gray), largely surpassing the mathematical peformance of Luciole-8B-Base (red) and even Luciole-23B-Base (light gray).

At the same time, it manages to maintain performance close to Luciole-8B-Base on general benchmarks as illustrated below.

Citation

Coming soon.

Acknowledgements

We gratefully acknowledge BPI France for funding the OpenLLM France project under the call "Communs numériques pour l’intelligence artificielle générative" ("Digital commons for generative artificial intelligence") as a part of the France 2030 program.

Training of Luciole-8B-KD-Math-Base was made possible by computing AI and storage resources by GENCI at IDRIS thanks to the grant A0181016189 on the supercomputer Jean Zay’s H100 partition. We gratefully acknowledge support from GENCI and IDRIS and from Stephane Requena (GENCI) and Pierre-François Lavallée (IDRIS) in particular.

The pruning and distillation pipeline behind Luciole-8B-KD-Math-Base was carried out by Niyar Barman with the support of other LINAGORA members (in alphabetical order): Akshay Chaturvedi, Olivier Gouvert, Julie Hunter, Jean-Pierre Lorré, and Jérôme Louradour.

Contact

contact@openllm-france.fr