README.md

---
license: apache-2.0
datasets:
- PleIAs/common_corpus
language:
- en
- fr
- de
- es
- it
- nl
- la
- pt
---
**Pleias-360m-RAG 0.1** is a specialized language model designed by PleIAs for Retrieval-Augmented Generation.

Similarly to its base model, Pleias-360m, Pleias-360m-RAG 0.1 aims to be a fully open model (weights, code, data), only trained on content with a permissible license and fully compliant with the European AI Act.

## Description
PleIAs-360m-RAG is continuous pretraining of Pleias-360m on a new dataset of 45,088,768,000 tokens modeling common retrieval tasks. All the content of the dataset is ultimately coming from Common Corpus.

Pleias-360m-RAG includes the main features of the original base model:
* Only trained on open data under a permissible license and in compliance with the European AI Act. By design, all Pleias model are unable to output copyrighted content.
* Extensive multilingual support for main European languages: English, French, German, Spanish, Italian, Dutch, Latin, Portuguese and Polish.
* Extremely low level of toxicity and problematic content.

Pleias-360m-RAG supports retrieval-augmented generation with enhanced verifiability, source analysis and grounding on submitted sources. This includes:
* Standardized structure and special tokens to include queries, sources, references.
* Anticipation of various query forms in multiple languages, from actual drafted questions to unstructured list of keyword search.
* Source analysis/criticism which also acts as an integrated reranker step.
* Generation of ground answers with references and excerpts linked to the original sources.

Given its small size, Pleias-360m-RAG 0.1 was originally conceived as an experimental model. 

Initial tests have shown that the RAG design has significantly improved the factuality and verifiability of the model. Even when the grounding does not work perfectly, the information remains much closer to the original sources.

As a result, Pleias-360m-RAG 0.1 has been already tested and integrated into multiple applied RAG projects, including Pleias flagship application Scholasticai.

## Training
PleIAs-360m-RAG was trained at Jean-Zay with 16 h100s with Nanotron, the pretraining library from HuggingFace. We provide the complete settings as a yaml file as part of our release.

PleIAs-360m-RAG derives from the last checkpoint of PleIAs-360m (518,000). The training schedule reused the last learning rate value (6e-5) without decay for 90,000 steps.

Training covers the entire RAG dataset we have been designing out of Common Corpus for 1 epoch.

Further experiments were made with different learning rate values: none of theses tests have provided a better convergence than the one obtained with the final learning rate from the base model.

## Inference
PleIAs-360m-RAG relies on special tokens to encode the core RAG functionalities:

A typical example, with excerpts drawn from a Wikipedia article on Wikipedia
```bash
<|query_start|>Is Wikipedia reliable?<|query_end|>
<|source_start|><|source_id_start|>ebea70a3502acfbd<|source_id_end|>Articles for traditional encyclopedias such as Encyclopædia Britannica are written by experts, lending such encyclopedias a reputation for accuracy.[144] However, a peer review in 2005 of forty-two scientific entries on both Wikipedia and Encyclopædia Britannica by the science journal Nature found few differences in accuracy, and concluded that "the average science entry in Wikipedia contained around four inaccuracies; Britannica, about three."[145] Joseph Reagle suggested that while the study reflects "a topical strength of Wikipedia contributors" in science articles, "Wikipedia may not have fared so well using a random sampling of articles or on humanities subjects."<|source_end|>
<|source_start|><|source_id_start|>5f862e733d38288e<|source_id_end|>As a consequence of the open structure, Wikipedia "makes no guarantee of validity" of its content, since no one is ultimately responsible for any claims appearing in it.[W 54] Concerns have been raised by PC World in 2009 regarding the lack of accountability that results from users' anonymity, the insertion of false information,[152] vandalism, and similar problems. Legal Research in a Nutshell (2011), cites Wikipedia as a "general source" that "can be a real boon" in "coming up to speed in the law governing a situation" and, "while not authoritative, can provide basic facts as well as leads to more in-depth resources".<|source_end|>
<|source_start|><|source_id_start|>354fa4908152b336<|source_id_end|>Wikipedia's open structure inherently makes it an easy target for Internet trolls, spammers, and various forms of paid advocacy seen as counterproductive to the maintenance of a neutral and verifiable online encyclopedia.[70][W 55] In response to paid advocacy editing and undisclosed editing issues, Wikipedia was reported in an article in The Wall Street Journal to have strengthened its rules and laws against undisclosed editing.[162] The article stated that: "Beginning Monday [from the date of the article, June 16, 2014], changes in Wikipedia's terms of use will require anyone paid to edit articles to disclose that arrangement. Katherine Maher, the nonprofit Wikimedia Foundation's chief communications officer, said the changes address a sentiment among volunteer editors that 'we're not an advertising service; we're an encyclopedia.'"<|source_end|>
<|source_analysis_start|>
```

As a specialized language model, PleIAs-360m-RAG will be unable to work properly with prompts that detracts from that design.

## Acceptable use
Pleias-360m-RAG includes a much wider range of support for verifiability and grounding than most generalist models. 

The model is not a substitute for an integrated RAG application. Retrieval errors as well as challenging texts and questions can still create a range of issues. We especially encourage end users to take advantage of the citations and the references to provide better indicators of accuracy.

For best results we recommend the following setting: 
* Deterministic generation (temp = 0) and no repetition penalty (which is unsurprisingly detrimental to the accuracy of citations).
* Standardized hashes of 16 characters. While the model has been trained on many other patterns (including full bibliographic entries), this has proven the most convenient for systematic citation parsing.

## Future updates
PleIAs-360m-RAG will be continuously improved through iterative retraining/adaptation.

The current roadmap includes the following features:
* Longer training on the same dataset for more than one epochs.
* Context length expansion.
* Better handling of multilingual sources. In its current form, PleIAs-360m-RAG will generally switch language if a query is made to sources in a different language.
* New sampling methods inspired by Entropix for a better combined support of text creativity and accuracy.
* Interactive/conversational RAG.

End users are encouraged to update to the latest version whenever possible.