README.md

---
license: apache-2.0
datasets:
- PleIAs/common_corpus
language:
- en
- fr
- de
- es
- it
- nl
- la
- pt
---

<div style="text-align: center;">
  <img src="https://raw.githubusercontent.com/Pleias/logos/d6152d7943905da32a1e04fdfd7708ed9c7eed5e/PleIAs%201_0%20Full%20Logo%20(Black).png" style="width: 80%; margin: 0 auto; display: inline-block;"/>
</div>


**Pleias-Pico** is a 353 million parameters specialized language model designed by PleIAs for Retrieval-Augmented Generation.

Similarly to its base model, Pleias-350m, Pleias-Pico aims to be a fully open model (weights, code, data), only trained on content with a permissible license and fully compliant with the European AI Act.

## Description
Pleias-Pico is continuous pretrain of Pleias-350m on a new dataset of 45,088,768,000 tokens modeling common retrieval tasks. All the content of the dataset is ultimately coming from [Common Corpus](https://huggingface.co/datasets/PleIAs/common_corpus).

Pleias-Pico includes the main features of the original base model:
* Only trained on open data under a permissible license and in compliance with the European AI Act. By design, all Pleias model are unable to output copyrighted content.
* Extensive multilingual support for main European languages: English, French, German, Spanish, Italian, Dutch, Latin, Portuguese and Polish.
* Extremely low level of toxicity and problematic content.

Pleias-Pico supports retrieval-augmented generation with enhanced verifiability, source analysis and grounding on submitted sources. This includes:
* Standardized structure and special tokens to include queries, sources, references.
* Anticipation of various query forms in multiple languages, from actual drafted questions to unstructured list of keyword search.
* Source analysis/criticism which also acts as an integrated reranker step.
* Generation of ground answers with references and excerpts linked to the original sources.

Initial tests have shown that the RAG design has significantly improved the factuality and verifiability of the model. Even when the grounding does not work perfectly, the information remains much closer to the original sources.

As a result, Pleias-Pico has been already tested and integrated into multiple applied RAG projects, including Pleias's flagship application Scholasticai.

## Training
Pleias-Pico was trained at Jean-Zay with 16 h100s with Nanotron, the pretraining library from HuggingFace. We provide the complete settings as a yaml file as part of our release.

Pleias-Pico derives from the last checkpoint of Pleias-350m (518,000). The training schedule reused the last learning rate value (6e-5) without decay for 90,000 steps.

Training covers the entire RAG dataset we have been designing out of Common Corpus for 1 epoch.

Further experiments were made with different learning rate values: none of theses tests have provided a better convergence than the one obtained with the final learning rate from the base model.

## Inference
Pleias-Pico relies on special tokens to encode the core RAG functionalities:

A typical example, with excerpts drawn from a Wikipedia article on Wikipedia
```bash
<|query_start|>Is Wikipedia reliable?<|query_end|>
<|source_start|><|source_id_start|>ebea70a3502acfbd<|source_id_end|>Articles for traditional encyclopedias such as Encyclopædia Britannica are written by experts, lending such encyclopedias a reputation for accuracy.[144] However, a peer review in 2005 of forty-two scientific entries on both Wikipedia and Encyclopædia Britannica by the science journal Nature found few differences in accuracy, and concluded that "the average science entry in Wikipedia contained around four inaccuracies; Britannica, about three."[145] Joseph Reagle suggested that while the study reflects "a topical strength of Wikipedia contributors" in science articles, "Wikipedia may not have fared so well using a random sampling of articles or on humanities subjects."<|source_end|>
<|source_start|><|source_id_start|>5f862e733d38288e<|source_id_end|>As a consequence of the open structure, Wikipedia "makes no guarantee of validity" of its content, since no one is ultimately responsible for any claims appearing in it.[W 54] Concerns have been raised by PC World in 2009 regarding the lack of accountability that results from users' anonymity, the insertion of false information,[152] vandalism, and similar problems. Legal Research in a Nutshell (2011), cites Wikipedia as a "general source" that "can be a real boon" in "coming up to speed in the law governing a situation" and, "while not authoritative, can provide basic facts as well as leads to more in-depth resources".<|source_end|>
<|source_start|><|source_id_start|>354fa4908152b336<|source_id_end|>Wikipedia's open structure inherently makes it an easy target for Internet trolls, spammers, and various forms of paid advocacy seen as counterproductive to the maintenance of a neutral and verifiable online encyclopedia.[70][W 55] In response to paid advocacy editing and undisclosed editing issues, Wikipedia was reported in an article in The Wall Street Journal to have strengthened its rules and laws against undisclosed editing.[162] The article stated that: "Beginning Monday [from the date of the article, June 16, 2014], changes in Wikipedia's terms of use will require anyone paid to edit articles to disclose that arrangement. Katherine Maher, the nonprofit Wikimedia Foundation's chief communications officer, said the changes address a sentiment among volunteer editors that 'we're not an advertising service; we're an encyclopedia.'"<|source_end|>
<|source_analysis_start|>
```

As a specialized language model, Pleias-Pico will be unable to work properly with prompts that detracts from that design.

## Acceptable use
Pleias-Pico includes a much wider range of support for verifiability and grounding than most generalist models. 

The model is not a substitute for an integrated RAG application. Retrieval errors as well as challenging texts and questions can still create a range of issues. We especially encourage end users to take advantage of the citations and the references to provide better indicators of accuracy.

For best results we recommend the following setting: 
* Deterministic generation (temp = 0) and no repetition penalty (which is unsurprisingly detrimental to the accuracy of citations).
* Standardized hashes of 16 characters. While the model has been trained on many other patterns (including full bibliographic entries), this has proven the most convenient for systematic citation parsing.

### RAG Evaluation

We evaluate Pico and Nano models on a RAG task. As existing benchmarks are largely limited to English, we develop a custom multilingual RAG benchmark. We synthetically generate queries and small sets of documents. To evaluate, we prompted models with the query and documents. We then ran a head-to-head ELO-based tournament with GPT-4o as judge. We [release the prompts and generations for all models we compared](https://huggingface.co/datasets/PleIAs/Pleias-1.0-eval/tree/main/RAGarena). Our nano (1.2B) model outperforms Llama 3.2 1.1B and EuroLLM 1.7B. Our pico (350M) model outperforms other models in its weight class, such as SmolLM 360M and Qwen2.5 500M, in addition to much larger models, such as Llama 3.2 1.1B and EuroLLM 1.7B.
  
| **Rank** | **Model**                | **ELO**    |
|----------|--------------------------|------------|
| 1        | Qwen2.5-Instruct-7B      | 1294.6     |
| 2        | Llama-3.2-Instruct-8B    | 1269.8     |
| 3        | **Pleias-nano-1.2B-RAG**   | **1137.5** |
| 4        | Llama-3.2-Instruct-3B    | 1118.1     |
| 5        | Qwen2.5-Instruct-3B      | 1078.1     |
| 6        | **Pleias-pico-350M-RAG** | **1051.2** |
| 7        | Llama-3.2-1B-Instruct    | 872.3      |
| 8        | EuroLLM-1.7B-Instruct    | 860.0      |
| 9        | SmolLM-360M-Instruct     | 728.6      |
| 10       | Qwen2.5-0.5B-Instruct    | 722.2      |
| 11       | SmolLM-1.7B-Instruct     | 706.3      |

## Ethical Considerations

pleias-pico model, like all large language models, carries inherent ethical risks that require careful consideration. Our approach to mitigating these risks begins at the data level, where we exclusively use vetted sources, deliberately excluding CommonCrawl. The primary challenge comes from our public domain dataset component, which contains historical texts that may reflect outdated social norms and potentially harmful language, particularly regarding minoritized groups.

To address this, we implemented a systematic ethical filtering process using toxicity classifiers to identify extremely harmful content. We also employed synthetic rewriting techniques to transform mildly problematic passages while preserving the underlying informational value. This process significantly reduced potential societal harm without compromising the dataset's size or textual quality, resulting in notably low toxicity scores in benchmarks compared to other models.

Despite these preventive measures, users should be aware that the model has not undergone additional safety alignment procedures and may still produce problematic outputs. The model's capabilities in generative AI tasks must be balanced against the risks of bias, misinformation propagation, and autonomous decision-making challenges. We explicitly prohibit any malicious utilization and emphasize the responsibility of users to implement appropriate safeguards.

At Pleias, we continue to research and develop improved methods for creating safer and more equitable models and datasets. This includes ongoing work in toxicity reduction, bias mitigation, and the development of more sophisticated ethical filtering techniques.

## Acknowledgements

The training was conducted as part of the Grand Challenge of GENCI, aligned with the European strategy for establishing AI factories through the EuroHPC Joint Undertaking, aimed at supporting European startups and providing open-source models to the community.

We express our gratitude to GENCI's Jean Zay supercomputer, France's AI flagship facility, which was instrumental in our model's training. The project benefited from the new NVIDIA H100 partition specifically dedicated to the French AI community. We appreciate the generous allocation of compute hours over five months and the invaluable technical expertise provided by IDRIS, EVIDEN, and NVIDIA (as well as its Inception program).

We are deeply grateful to the Mozilla Foundation Local AI Program for their generous support. 

Finally, we acknowledge the significant contributions from the open science LLM community, particularly HuggingFace, Eleuther AI and Allen AI whose insights and cooperation have been invaluable to our work.

## Future updates
Pleias-Pico will be continuously improved through iterative retraining/adaptation.

The current roadmap includes the following features:
* Longer training on the same dataset for more than one epochs.
* Context length expansion.
* Better handling of multilingual sources. In its current form, Pleias-Pico will generally switch language if a query is made to sources in a different language.
* New sampling methods inspired by Entropix for a better combined support of text creativity and accuracy.
* Interactive/conversational RAG.

End users are encouraged to update to the latest version whenever possible.