Model Card for gemma-2b-it-peru-legal-es-V2 ⚖️

Model Illustration

Gemma-2B-IT-Peru-Legal-ES

The Gemma-2B-IT-Peru-Legal-ES model is a state-of-the-art language model fine-tuned specifically for legal text comprehension and generation tasks in Spanish, focusing on the legal context of the Peruvian Constitution. Leveraging advanced techniques such as Low-Rank Adaptation (LoRA) and Bits and Bytes Quantization (BNB), this model provides accurate and contextually relevant responses to legal queries, making it a valuable tool for legal professionals, researchers, and AI enthusiasts interested in the legal domain.

Gemma-2B-IT-Peru-Legal-ES ⚖️
- Model Description 📘

Model Description

Gemma-2b-it-peru-legal-es-V2 is a contextual legal language model designed to provide personalized legal advice in Spanish based on Peruvian legal texts. Leveraging advanced techniques like LoRA, the model offers accurate and contextually relevant responses to legal queries, covering various aspects of Peruvian law and regulation. Whether it's understanding rights, navigating legal procedures, or interpreting statutes, gemma-2b-it-peru-legal-es-V2 empowers users with comprehensive and reliable legal guidance tailored to the Peruvian legal landscape.

Developed by: Alonso Quispe
Model type: Causal Language Model, specially fine-tuned with LoRA for the distinct domain of Peruvian law and regulation.
Language(s) (NLP): Spanish, tailored for the legal and regulatory context of Peru.
License: Apache License. This open-source license ensures that the model can be freely used, modified, and distributed. Please check the model's page on Hugging Face for specific licensing details.
Base model: google/gemma-2b-it`

Recommendations

Users should verify model outputs against current regulations and consult with professionals for critical applications. Awareness of the model's scope and limitations is crucial for effective use.

How to Get Started with the Model

from transformers import AutoModelForCausalLM, AutoTokenizer

# Cargar el modelo y el tokenizador
model_name = "somosnlp/gemma-2b-it-peru-legal-es-V2"
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Ejemplo de uso
input_text = "Una persona ha sido despedida de su trabajo injustamente y necesita entender cuáles son sus derechos laborales según la Constitución Política del Perú."
input_ids = tokenizer.encode(input_text, return_tensors="pt")
output = model.generate(input_ids, max_length=100, num_return_sequences=1, temperature=0.7)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Training Dataset

The Gemma-2B-IT-Peru-Legal-ES-V2 model was fine-tuned exclusively on the "Constitución Política del Perú dataset available through Hugging Face Datasets. This dataset serves as a rich source of questions and answers pertaining to the legal framework outlined in the Peruvian Constitution. The dataset encompasses a wide range of topics and provisions within the Peruvian Constitution, providing comprehensive coverage of constitutional principles, rights, and legal interpretations.

Finetuning Progress

Loss Function Graph

Environment and Libraries

The training process was executed within a Python environment, utilizing essential libraries to facilitate various tasks:

transformers: Used for loading and fine-tuning the model, providing a seamless interface for working with pre-trained language models.
datasets: Employed for efficient handling and preprocessing of the training dataset, ensuring streamlined data pipelines.
torch: Utilized as the primary deep learning framework to support model training and optimization.
peft: Integrated for applying Low-Rank Adaptation (LoRA) techniques to the model, enabling efficient adaptation to specialized domains without compromising performance.
qlora: Additionally utilized for further model adaptation, enhancing its ability to comprehend and generate responses specific to the legal context of the "Constitucióm Política del Perú" dataset.

QLoRA Configuration

QLoRA (Quantization LoRA) was employed to optimize the model's computational efficiency and memory footprint while preserving its accuracy. Two configurations were utilized:

BitsAndBytesConfig: This configuration enabled the model to load in 4-bit quantization, leveraging the nf4 quantization type with a torch.bfloat16 compute data type for enhanced efficiency during inference.
```
    bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=False,
)
```
LoRAConfig: This configuration applied Low-Rank Adaptation (LoRA) to the model, optimizing its parameters for the specific task of language modeling in the legal domain. Key parameters include:
- r: Reduced to 8 from the default 32, controlling the rank of adaptation layers.
- lora_alpha: Adjusted to 16 from the default 64, regulating the sparsity of adapted weights.
- target_modules: Specified modules for adaptation, focusing on query, key, value, and output projection layers.
- bias: Set to "none" to exclude bias terms from adaptation, simplifying the model architecture.
- lora_dropout: Reduced to 0.025 from the default 0.05, controlling the dropout rate during adaptation.
- task_type: Configured as "CAUSAL_LM" to indicate the task type of the language model.
```
    config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=['q_proj', 'k_proj', 'v_proj', 'o_proj'],
    bias="none",
    lora_dropout=0.025,
    task_type="CAUSAL_LM",
)
```

These configurations were crucial for optimizing the model's performance and resource utilization during training and inference, ensuring efficient deployment.

Model Merging and Saving 💾

After fine-tuning, the LoRA-adjusted weights were merged back with the base Gemma model to create the final gemma-2b-it-peru-legal-es-V2. The model was then saved and made available through Hugging Face for easy access and further development.

Logging with Wandb 📊

During the training process, Wandb (Weights & Biases) was used for comprehensive logging and visualization of key metrics. Wandb's powerful tracking capabilities enabled real-time monitoring of training progress, evaluation metrics, and model performance. Through interactive dashboards and visualizations, Wandb facilitated deep insights into the training dynamics, allowing for efficient model optimization and debugging. This logging integration with Wandb enhances transparency, reproducibility, and collaboration among researchers and practitioners.

Impacto Ambiental

The training of gemma-2b-it-peru-legal-es-V2 was conducted optimizing the computational expenditure required.

Hardware Type: A10G-24GB
Hours Utilized: Approximately --- hours
Energy Consumption: Approximately --- kWh
Estimated CO2 Emissions: Approximately --- kg

To-Do List

Dataset Generation, Human Feedback Review with Argilla, and Finetuning with the Following Legal Texts:

Constitution of Peru
Penal Code
Congress Regulation
New Constitutional Procedural Code
Civil Code
(TUO) Civil Procedural Code
Penal Procedural Code (D.L 638)
New Penal Procedural Code (D.L 957)
Penal Execution Code
Military Police Penal Code
Military Police Justice Code
Children and Adolescents Code
Adolescent Penal Responsibility Code
Commercial Code
Consumer Protection and Defense Code
Tax Code (TUO)
Criminal Procedure Code

License

This project is distributed under the Apache 2.0 license.

daqc
/

gemma-2b-it-peru-legal-es-V2