# LLaMandement-13B: A Llama-2-based French Models for Summarization of French Legislative Proposals. LLaMandement-13B is a French chat LLM, based on [LLaMA-2-13B](https://ai.meta.com/llama), optimized to summarize of French Legislative Proposals. ## Model Details - **Developed by:** [DGFIP](https://www.impots.gouv.fr/presentation-de-la-dgfip-overview-dgfip) : - yannis.tannier@dgfip.finances.gouv.fr - joseph.gesnouin@dgfip.finances.gouv.fr - **Model type:** An auto-regressive language model based on the transformer architecture - **License:** Llama 2 Community License Agreement - **Finetuned from model:** [Llama 2](https://arxiv.org/abs/2307.09288) - **Repository:** https://gitlab.adullact.net/dgfip/projets-ia/llamandement - **Paper:** working ## Prompt Template The prompt for LLaMandement-13B is based on alpaca template : ``` Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: {instruction} ### Response: ``` ## How to Get Started with the Model - Command line interface: https://github.com/lm-sys/FastChat - APIs (OpenAI API, Huggingface API): https://github.com/lm-sys/FastChat/tree/main#api ## Training Details Llamandement-13B is fine-tuned from Llama 2 using Low-Rank Adaptation (LORA). This method is efficient and adds minimal computational load. It introduces additional low-rank parameters, enabling the model to better handle complex legislative language without major changes to the original structure. **LORA Settings Adjustments:** - **Learning Rate (LR):** Set to a low value of 2e-5 to ensure stable and gradual improvements. - **Adaptation Depth (lora_r):** Set at 64, influencing the dimension of the low-rank matrix in LORA. This affected about 0.40% of the model's weights. - **Decay Rate:** Employed at 0.01 to prevent overfitting to specific legislative text structures. - **LORA Alpha (α):** Set at 16, it fine-tunes the model's response to legislative text. - **LORA Dropout:** A rate of 0.1 applied to LORA layers to prevent overfitting and enhance generalization. - **Optimizer and Scheduler:** Utilized a cosine learning rate scheduler with a warmup ratio of 0.03 for optimal training. For more information, visit [dgfip.finance.com](http://dgfip.finance.com). Additional details about the training dataset composition can be found [here](http://dgfip.finance.com/training-dataset-info). ## Citation Please cite the repo if you use the data, method or code in this repo. [...]