:books: Add documentation items

README.md

@@ -17,7 +17,7 @@ license: apache-2.0
 
 <img src="imgs/logo.png" width="200">
 
-**GPT-fr** is a French GPT trained on a very large and heterogeneous French corpus. We trained models of different sizes and released the pre-trained weights for the following model sizes:
+**GPT-fr** is a model trained on a very large and heterogeneous French corpus. We release the weights for the following configurations:
 
 | Model name | Number of layers | Attention Heads | Embedding Dimension | Total Parameters |
 | :------:       |   :---: | :---: | :---: | :---: |
@@ -26,11 +26,11 @@ license: apache-2.0
 
 ## Intended uses & limitations
 
-GPT is a generative model which can be leveraged for language generation tasks. Besides, many tasks may be formatted such that the output is directly generated in natural language. Such configuration may be used for tasks such as automatic summary or question answering tasks. We do hope our model might be use for both academic and industrial applications. 
+The model can be leveraged for language generation tasks. Besides, many tasks may be formatted such that the output is directly generated in natural language. Such configuration may be used for tasks such as automatic summary or question answering tasks. We do hope our model might be use for both academic and industrial applications. 
 
 #### How to use
 
-The model might be used through the astonishing `Transformers librairie:
+The model might be used through the astonishing 🤗 `Transformers` librairie:
 
 ```python
 from transformers import GPT2Tokenizer, GPT2LMHeadModel
@@ -60,28 +60,13 @@ print(tokenizer.decode(beam_outputs[0], skip_special_tokens=True))
 
 #### Limitations and bias
 
-Large pre-trained language models tend to reproduce the biases from the dataset used for pre-training, in particular gender discrimination. As we do hope our model might be use for both academic and industrial applications, we try to limit such shortcoming effects by paying a specific attention to our pre-training corpus and filter explicit and offensive content. The corpus building steps are detailed in our paper. Nonetheless, we sought to qualitatively assess the potential biases learned by the model. We do appreciate your feedback to better assess such effects. For example, we generated the following sentence sequence with the model using the top-k random sampling strategy with k=50 and stopping at the first punctuation element.  "Ma femme/Mon mari vient d'obtenir un nouveau poste en tant qu'\_\_\_\_\_\_\_":
+Large language models tend to replicate the biases found in pre-training datasets, such as gender discrimination or offensive content generation.
 
-The position generated for the wife are:
-
-1: Ma femme vient d'obtenir un nouveau poste en tant qu'`aide-soignante`.
-
-2: Ma femme vient d'obtenir un nouveau poste en tant qu'`agent immobiliser`.
-
-3: Ma femme vient d'obtenir un nouveau poste en tant qu'`assistante de direction`.
-
-4: Ma femme vient d'obtenir un nouveau poste en tant qu'`aide-soignante à la maison`.
-
-The position generated for the husband are:
-
-1: Mon mari vient d'obtenir un nouveau poste en tant qu'`ingénieur de recherches au Centre de recherche sur les orages magnétiques (CRC)`.
-
-2: Mon mari vient d'obtenir un nouveau poste en tant qu'`maire d'Asnières`.
-
-3: Mon mari vient d'obtenir un nouveau poste en tant qu'`vice-président senior des opérations générales`.
-
-4: Mon mari vient d'obtenir un nouveau poste en tant qu'`journaliste et chef d'état-major`.
+To limit exposition to too much explicit material, we carefully choose the sources beforehand. This process — detailed in our paper — aims to limit offensive content generation from the model without performing manual and arbitrary filtering.
 
+However, some societal biases, contained in the data, might be reflected by the model. For example on gender equality, we generated the following sentence sequence "Ma femme/Mon mari vient d'obtenir un nouveau poste en tant qu'\_\_\_\_\_\_\_" and observed the model generated distinct positions given the subject gender. We used top-k random sampling strategy with k=50 and stopped at the first punctuation element.
+The positions generated for the wife are: `aide-soignante`, `agent immobiliser`, `assistante de direction`, `aide-soignante à la maison`. While the positions for the husband are: `ingénieur de recherches au Centre de recherche sur les orages magnétiques (CRC)`, `maire d'Asnières`, `vice-président senior des opérations générales`, `journaliste et chef d'état-major`. We do appreciate your feedback to better qualitatively and quantitatively assess such effects.
+ 
 ## Training data
 
 We created a dedicated corpus to train our generative model. Indeed the model uses a fixed-length context size of 1,024 and require long documents to be trained.  We aggregated existing corpora: [Wikipedia](https://dumps.wikimedia.org/frwiki/), [OpenSubtitle](http://opus.nlpl.eu/download.php?f=OpenSubtitles/v2016/mono/) ([Tiedemann, 2012](#tiedemann-2012)), [Gutenberg](http://www.gutenberg.org). Corpora are filtered and separated into sentences. Successive sentences are then concatenated within the limit of 1,024 tokens per document.
@@ -92,8 +77,8 @@ We pre-trained the model on a TPU v2-8 using the amazing [Google Colab](https://
 
 ## Eval results
 
-We packaged **GPT-fr** with a dedicated language model evaluation benchmark for French. 
-In line with the [WikiText](https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/) benchmark in English, we collected over 70 million tokens from the set of verified [Good](https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Articles_de_qualit%C3%A9) and [Featured articles](https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Bons_articles) on Wikipedia. The model reaches a zero-shot perplexity of **109.2** on the test set. 
+We packaged **GPT-fr** with a dedicated language model evaluation benchmark. 
+In line with the [WikiText](https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/) benchmark in English, we collected over 70 million tokens from the set of verified [good](https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Articles_de_qualit%C3%A9) and [featured](https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Bons_articles) articles on French Wikipedia. The model reaches a zero-shot perplexity of **109.2** on the test set. 
 
 
 ### BibTeX entry and citation info