LLaMandement-13B: A Llama-2-based French Models for Summarization of French Legislative Proposals.
LLaMandement-13B is a French chat LLM, based on LLaMA-2-13B, optimized to summarize of French Legislative Proposals.
Model Details
- Developed by: DGFIP :
- Model type: An auto-regressive language model based on the transformer architecture
- License: Llama 2 Community License Agreement
- Finetuned from model: Llama 2
- Repository: https://gitlab.adullact.net/dgfip/projets-ia/llamandement
- Paper: working
Prompt Template
The prompt for LLaMandement-13B is based on alpaca template :
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
{instruction}
### Response:
How to Get Started with the Model
- Command line interface: https://github.com/lm-sys/FastChat
- APIs (OpenAI API, Huggingface API): https://github.com/lm-sys/FastChat/tree/main#api
Training Details
Llamandement-13B is fine-tuned from Llama 2 using Low-Rank Adaptation (LORA). This method is efficient and adds minimal computational load. It introduces additional low-rank parameters, enabling the model to better handle complex legislative language without major changes to the original structure.
LORA Settings Adjustments:
- Learning Rate (LR): Set to a low value of 2e-5 to ensure stable and gradual improvements.
- Adaptation Depth (lora_r): Set at 64, influencing the dimension of the low-rank matrix in LORA. This affected about 0.40% of the model's weights.
- Decay Rate: Employed at 0.01 to prevent overfitting to specific legislative text structures.
- LORA Alpha (α): Set at 16, it fine-tunes the model's response to legislative text.
- LORA Dropout: A rate of 0.1 applied to LORA layers to prevent overfitting and enhance generalization.
- Optimizer and Scheduler: Utilized a cosine learning rate scheduler with a warmup ratio of 0.03 for optimal training.
For more information, visit dgfip.finance.com. Additional details about the training dataset composition can be found here.
Citation
Please cite the repo if you use the data, method or code in this repo.
[...]