Upload README.md

README.md

@@ -2,199 +2,264 @@
 library_name: transformers
 tags:
 - unsloth
+- LLMs-Aviation
+- AI-Regulatory-Compliance
+- RAC-AI-Colombia
+license: apache-2.0
+datasets:
+- somosnlp/ColombiaRAC_FullyCurated
+language:
+- es
+widget:
+- text: >
+    <bos><start_of_turn>system\n\nYou are a helpful AI assistant.\n\nResponde en formato json.\n\nEres un agente experto en la normativa aeronautica Colombiana.<end_of_turn>\n\n<start_of_turn>user\n\n¿Qué sucede con las empresas de servicios aéreos comerciales que no hayan actualizado su permiso de operación después del 31 de marzo de 2024?<end_of_turn>\n\n<start_of_turn>model
 ---
-
-# Model Card for Model ID
-
-<!-- Provide a quick summary of what the model is/does. -->
-
+# Model Card for GemmaColRAC-AeroExpert Language Model: Gemma 2B for Colombian Aviation Regulations 🛫
 
 
 ## Model Details
 
 ### Model Description
 
-<!-- Provide a longer summary of what this model is. -->
+Este documento ofrece una visión detallada de `GemmaColRAC-AeroExpert`, la quinta iteración de nuestro modelo especializado en regulaciones aeronáuticas colombianas. Presenta un salto cualitativo con respecto a las versiones previas, exhibiendo mejoras en precisión y un uso de recursos de GPU más eficiente, reflejando nuestro compromiso con el desarrollo sostenible y de calidad de tecnologías de IA para la aviación.
 
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+<p align="center">
+  <img src="https://cdn-uploads.huggingface.co/production/uploads/6419c2f6b4adb0e101b17b6c/0undo4kZc7OtfGI5nnAa8.png" alt="Imagen del Reglamento Aeronáutico Colombiano" style="width: 40%; max-height: 550px;">
+</p>
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+- **Developed by:** [Edison Bejarano](https://huggingface.co/ejbejaranos), [Nicolai Potes](https://huggingface.co/NickyNicky) and [Santiago Pineda](https://huggingface.co/Sapinedamo) ✨
+- **Funded by:** Fundación Universitaria Los Libertadores, SomosNLP, HuggingFace
+- **Model type:** Specialized Language Model for Colombian Aeronautical Regulations
+- **Language(s):** Spanish (`es-CO`)
+- **License:** apache-2.0 <!-- Elegid una licencia lo más permisiva posible teniendo en cuenta la licencia del model pre-entrenado y los datasets utilizados -->
+- **Fine-tuned from model:** [More Information Needed] <!-- Enlace al modelo pre-entrenado que habéis utilizado como base -->
+- **Dataset used:** [RAC Corpus: Base de Datos del Reglamento Aeronáutico Colombiano 🛫📚🇨🇴](https://huggingface.co/datasets/somosnlp/Reglamento_Aeronautico_Colombiano_2024/blob/01bf7eebef40aaba374ffd30697582ab10ec3503/README.md)
 
-### Model Sources [optional]
 
-<!-- Provide the basic links for the model. -->
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
 
-## Uses
+### Model Sources
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+- **Demo:** [Model Demo on HuggingFace Spaces](https://huggingface.co/spaces/somosnlp/ColombiaRAC-V1)
+- **Video presentation:** [Aviación Inteligente: LLMs para Navegar el RAC | Hackathon] (https://youtu.be/IGKU1qUur2c?si=Na4d3XIU3vbdaaJj)
+
+## Uses
 
 ### Direct Use
+Is designed to assist professionals and students in the aviation industry by providing enhanced access to the Colombian Aeronautical Regulations through advanced language processing capabilities.
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+### Out-of-Scope Use
 
-[More Information Needed]
+This model is not intended for making legally binding decisions without human oversight.
 
-### Downstream Use [optional]
+## Bias, Risks, and Limitations
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+The model may inherit biases from the data used for training, which primarily includes official legal texts. Users should exercise caution and not rely solely on the model for critical decision-making.
 
-[More Information Needed]
+## How to Get Started with the Model
 
-### Out-of-Scope Use
+Use the code below to get started with the model.
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+```python
+from transformers import AutoModel, AutoTokenizer
 
-[More Information Needed]
+tokenizer = AutoTokenizer.from_pretrained("somosnlp/GemmaColRAC-AeroExpert")
+model = AutoModel.from_pretrained("somosnlp/GemmaColRAC-AeroExpert")
 
-## Bias, Risks, and Limitations
+# Example of how to use the model
+encoded_input = tokenizer("Example query about aviation regulations", return_tensors='pt')
+output = model(**encoded_input)
+```
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+## Training Details
 
-[More Information Needed]
+### Training Data
 
-### Recommendations
+The model was trained on a curated dataset consisting of detailed question-answer pairs related to the Colombian Aeronautical Regulations.
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
 
-## How to Get Started with the Model
 
-Use the code below to get started with the model.
+### Training Procedure
 
-[More Information Needed]
+The model was fine-tuned from a base language model using the following specifications:
 
-## Training Details
+- **Tipo de GPU:** NVIDIA GeForce RTX 3090
+- **Tiempo Total de Entrenamiento:** 12607 segundos
+- **Optimizador:** AdamW con Bitfitting y Neutrino Noise
+- **Pasos Máximos:** 4904
+- **Tamaño de Secuencia:** 2048
+- **Tamaño de Lote por Dispositivo:** 2
+- **Versión de Transformers:** 4.39.2
+- **Framework de Optimización:** Unsloth 2024.4
+- **Métodos de Cuantificación:** bf16 con gradient_accumulation_steps de 2
+- **Función de Activación:** gelu_pytorch_tanh
 
-### Training Data
+- [Notebook to train the model](https://colab.research.google.com/drive/1VmcSVvkaXVe-ya5ATDxKilPY9kN-x2_I?usp=sharing)
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
 
-[More Information Needed]
+### Comparison with Previous Version 🔄
 
-### Training Procedure
+The previous iteration, `GemmaColRAC-AeroExpertV4`, utilized an NVIDIA A100-SXM4-40GB GPU and was trained for approximately 50 minutes (3007 seconds). It operated with a learning rate of 0.00005 and used an 8-bit Paged AdamW optimizer. Furthermore, it was trained with a batch size per device of 1 and utilized version 4.39.0 of the Transformers library.
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+**Key differences with the current version include:**
 
-#### Preprocessing [optional]
+- **GPU Upgrade:** 🆙 Switched from NVIDIA A100-SXM4-40GB to NVIDIA GeForce RTX 3090, offering better performance during training.
+- **Training Time:** ⏳ Increased to allow more extensive fine-tuning of the model, resulting in improved accuracy.
+- **Batch Size:** 🔢 Increased the batch size per device from 1 to 2, allowing for more efficient optimization.
+- **Optimizer Upgrade:** 🛠️ Introduction of advanced techniques such as Bitfitting and Neutrino Noise to enhance model convergence.
+- **Maximum Steps:** 🚶‍♂️ Significantly increased the maximum steps from 1638 to 4904, suggesting a broader coverage of data and deeper learning.
 
-[More Information Needed]
+These changes have resulted in a more robust and efficient version of our model, enhancing its capacity to assist and provide guidance in Colombian aeronautical regulation.
 
 
 #### Training Hyperparameters
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
+- **Training regime:** bf16 mixed precision
+- **Optimizer:** Paged AdamW 8-bit
+- **Learning Rate:** 5e-5
+- **Batch Size per Device:** 3
+- **Gradient Accumulation Steps:** 4
+- **Warmup Steps:** Computed as 3% of total steps
+- **Max Steps:** 14,688
+- **Total Training Time:** Approx. 5 hours 21 minutes (based on epochs and iteration speed)
+- **Max Sequence Length:** 2048
+- **Weight Decay:** 0.001
+- **Learning Rate Scheduler:** Cosine
+- **Adam Betas:** Beta1 = 0.99, Beta2 = 0.995
+- **Max Gradient Norm:** 0.4
+- 
+#### Speeds, Sizes, Times
+
+- **Training Duration:** Approx. 3 hours 30 minutes for full training
+- **Training Throughput:** 0.76 iterations per second (it/s)
+- **Total Steps:** 14,688 steps over 8 epochs
+- **Checkpoint Size:** Final model size was not specified; typical sizes for models of this type are several gigabytes.
+- **Total Number of Trainable Parameters:** 78,446,592
 [More Information Needed]
 
-## Evaluation
 
-<!-- This section describes the evaluation protocols and provides the results. -->
+### Metrics
 
-### Testing Data, Factors & Metrics
+Here is a detailed summary of the training metrics for `GemmaColRAC-AeroExpert`:
 
-#### Testing Data
+- **Total Floating Point Operations (FLOPs):** 204,241,541,673,615,360
+- **Train Loss:** 0.393565042567292 (final reported loss)
+- **Training Runtime:** 10,763.56 seconds (approximately 2.99 hours)
+- **Samples per Second:** 4.556
+- **Steps per Second:** 0.456
+- **Total Training Epochs:** 2
+- **Total Training Steps:** 4,904
+- **Gradient Norm:** 3.515625
+- **Final Learning Rate:** 0 (end of training)
+- **Average Loss over Training:** 0.1934
 
-<!-- This should link to a Dataset Card if possible. -->
+### Results
 
-[More Information Needed]
 
-#### Factors
+<p align="center">
+  <img src="https://cdn-uploads.huggingface.co/production/uploads/6419c2f6b4adb0e101b17b6c/zuEetm8ifT5e3QtHfBBVD.png" alt="Trainning Loss" style="width: 80%; max-height: 350px;">
+</p>
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
 
-[More Information Needed]
+## Model Examination [optional]
+This model was evaluated the performance in simplifying RAC's content based on feedback from aeronautical experts, thereby enhancing regulatory compliance and understanding.
 
-#### Metrics
+<p align="center">
+  <img src="https://cdn-uploads.huggingface.co/production/uploads/6419c2f6b4adb0e101b17b6c/5iPvAhaTMnqRDBn2g7XIK.png" alt="Evaluation for model by Aeronautical experts" style="width: 40%; max-height: 550px;">
+</p>
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
 
-[More Information Needed]
+Previous table shows the model's strong performance with average scores of 7 from 276 tests. However, RAC 3's low scores (mean 3.464, median 1) indicate areas needing improvement, while high ratings in RACs 1 and 5 suggest strengths. These results confirm the model's potential for accuracy and generalization, though RAC 3 requires adjustments.
 
-### Results
 
-[More Information Needed]
+## Environmental Impact 🌱
 
-#### Summary
+The development of `GemmaColRAC-AeroExpert` has been carried out with a strong focus on sustainability 🌿. Efforts have been made to optimize efficiency and minimize environmental impact, including reducing energy consumption and lowering the carbon footprint during the model's training process. These measures not only enhance operational efficiency but also align with our commitment to environmental responsibility 🌎.
 
+### Energy Consumption and Carbon Emissions 📉
 
+- **Power Consumption:** 0.25 kW (250 watts)
+- **Runtime Hours:** 3.6 hours
+- **Carbon Intensity:** 475 gCO2eq per kWh (Global average)
 
-## Model Examination [optional]
+Given the use of an NVIDIA V100 GPU for approximately 3.6 hours, the carbon emissions have been meticulously estimated. Here are the details:
 
-<!-- Relevant interpretability work for the model goes here -->
+- **Hardware Type:** NVIDIA GeForce RTX 3090 GPU
+- **Total Hours Used:** ~3.6 hours
+- **Total Carbon Emitted:** Approximately 356.25 grams of CO₂ equivalents
 
-[More Information Needed]
+These carbon emissions were calculated using the [Machine Learning Impact Calculator](https://mlco2.github.io/impact#compute) introduced in Lacoste et al. (2019), which considers hardware type, runtime, and other relevant factors to provide a comprehensive view of the environmental impact of training large AI models 📊.
 
-## Environmental Impact
+This proactive approach to understanding and mitigating our ecological footprint underlines our commitment to pioneering environmentally friendly AI development practices, setting a benchmark for sustainability within the AI industry 🌟.
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+#### Hardware
 
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+- **Hardware Used:** NVIDIA GeForce RTX 3090
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
+#### Software 🛠️
 
-## Technical Specifications [optional]
+The `GemmaColRAC-AeroExpert` model was developed and trained using a comprehensive stack of modern software libraries designed for high-performance machine learning tasks, particularly in Natural Language Processing (NLP). Here are the key libraries and tools used:
 
-### Model Architecture and Objective
+- **Python Libraries:**
+  - `json`: For parsing JSON files and handling serialization 📄.
+  - `pandas`: A powerful data manipulation and analysis library providing data structures and operations for manipulating numerical tables and time series 📊.
+  - `torch`: PyTorch is an open-source machine learning library used for applications such as computer vision and natural language processing, developed by Facebook's AI Research lab (FAIR) 🔥.
+  - `datasets`: A lightweight and extensible library to easily share and access datasets and evaluation metrics for machine learning tasks 📚.
+  - `huggingface_hub`: Used for managing model repositories on Hugging Face and interacting with Hugging Face Hub APIs 🌐.
 
-[More Information Needed]
+- **Hugging Face Ecosystem:**
+  - `transformers`: Provides thousands of pre-trained models to perform tasks on texts such as classification, information extraction, question answering, summarization, translation, and text generation in over 100 languages. It's designed to be both user-friendly for machine learning researchers and efficient to use in production 🤖.
+  - `BitsAndBytesConfig`, `TrainingArguments`: Advanced configurations from the Transformers library for fine-tuning the performance and efficiency of training neural networks ⚙️.
+  - `pipeline`: A utility for creating easy-to-use pipelines for various NLP tasks 🧪.
+  - `AutoModelForCausalLM`, `AutoTokenizer`: Utilities for loading and initializing pre-trained language models and their tokenizers 📝.
+  - `logging`: For configuring the logging level and output formats to track model training and inference processes effectively 📌.
 
-### Compute Infrastructure
+- **PEFT and LoRA Extensions:**
+  - `LoraConfig`, `PeftModel`: Extensions from the PEFT (Parameter Efficient Fine-Tuning) library, which include LoRA (Low-Rank Adaptation of large models), allowing efficient fine-tuning and adaptation of large pre-trained models with minimal computational overhead 🚀.
+  
+- **Transformers Reinforcement Learning (TRL):**
+  - `SFTTrainer`: A component from the TRL library for applying reinforcement learning techniques to transformer models, specifically for sequence-to-sequence tasks 🎮.
 
-[More Information Needed]
+These tools collectively support the robust training environment necessary to develop state-of-the-art NLP models like `GemmaColRAC-AeroExpert`, ensuring that the model is both highly effective and efficient in processing and understanding complex regulatory texts.
 
-#### Hardware
+## License 📜
 
-[More Information Needed]
+`GemmaColRAC-AeroExpert` is released under the Apache 2.0 license 🏷️. This license is one of the most permissive and widely used licenses in the open-source community, allowing for both academic and commercial use without significant restrictions.
 
-#### Software
+- **Why Apache 2.0?** 🤔
+  - **Openness:** The Apache 2.0 license allows users to use, modify, and distribute the software freely, which encourages innovation and widespread use.
+  - **Protection:** It provides an explicit grant of patent rights from contributors to users, protecting them from patent litigation.
+  - **Commercial friendly:** Apache 2.0 is business-friendly, allowing the commercial use of the software which is crucial for wider adoption in industry settings.
 
-[More Information Needed]
+By choosing Apache 2.0, we ensure that `GemmaColRAC-AeroExpert` can be freely used and integrated into a wide array of projects and products, from academic research to commercial applications, thus supporting the growth and accessibility of AI technologies across different sectors 🌐.
 
-## Citation [optional]
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
 
-**BibTeX:**
 
-[More Information Needed]
+## Glossary [optional]
 
-**APA:**
+- **RAC**: Reglamento Aeronáutico Colombiano 
 
-[More Information Needed]
+## More Information
 
-## Glossary [optional]
+<!-- Indicar aquí que el marco en el que se desarrolló el proyecto, en esta sección podéis incluir agradecimientos y más información sobre los miembros del equipo. Podéis adaptar el ejemplo a vuestro gusto. -->
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+This project was developed during the [Hackathon #Somos600M](https://somosnlp.org/hackathon) organized by SomosNLP. The model was trained using GPUs sponsored by their own team.
 
-[More Information Needed]
+## Team 👥
 
-## More Information [optional]
+The development of the `GemmaColRAC-AeroExpert` model was supported by a dedicated team of experts specializing in machine learning, natural language processing, and aeronautics. Below are the key team members who contributed significantly to this project:
 
-[More Information Needed]
+- [Edison Bejarano](https://huggingface.co/ejbejaranos) - Lead AI Scientist, expert in NLP and machine learning, with a strong background in aeronautics.
+- [Nicolai Potes](https://huggingface.co/NickyNicky) - Data Scientist, specializes in AI-driven regulatory compliance solutions.
+- [Santiago Pineda](https://huggingface.co/Sapinedamo) - Project Manager and Senior ML Engineer, with extensive experience in deploying scalable AI solutions.
+- [Alec Mauricio](https://huggingface.co/alecrosales1) - AI Researcher, focused on developing innovative models for text analysis and interpretation.
+- [Danny Stevens](https://huggingface.co/dannystevens) - Software Engineer, provides expertise in software development and integration for machine learning applications.
 
-## Model Card Authors [optional]
+These individuals bring a wealth of knowledge and expertise, ensuring the highest quality and performance of the `GemmaColRAC-AeroExpert` model. Their collaborative efforts have been pivotal in pushing the boundaries of what's possible with AI in the aviation sector.
 
-[More Information Needed]
 
-## Model Card Contact
+## Contact [optional]
 
-[More Information Needed]
+Ejbejaranos@gmail.com