bugdaryan
/

Code-Llama-2-13B-instruct-text2sql

@@ -1,196 +1,125 @@
 ---
 license: llama2
 ---
-# Model Card for Code-Llama-2-13B-instruct-text2sql
-<!-- Provide a quick summary of what the model is/does. -->
-This modelcard aims to be a base template for new models. It has been generated using [this raw template](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/templates/modelcard_template.md?plain=1).
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-- **Developed by:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Data Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]

 ---
 license: llama2
+datasets:
+- bugdaryan/sql-create-context-instruction
+language:
+- en
+pipeline_tag: text-generation
 ---
+# **Code-Llama-2-13B-instruct-text2sql Model Card**
+**Model Name**: Code-Llama-2-13B-instruct-text2sql
+**Description**: This model is a fine-tuned version of the Code Llama 2 with 13 billion parameters, specifically tailored for text-to-SQL tasks. It has been trained to generate SQL queries given a database schema and a natural language question.
+## Model Information
+- **Base Model**: [codellama/CodeLlama-13b-hf](https://huggingface.co/codellama/CodeLlama-13b-hf)
+- **Finetuning Dataset**: [bugdaryan/sql-create-context-instruction](Dataset URL)
+- **Training Time**: Approximately 4 hours on 2 V100 32GB GPUs
+## LoRA Parameters
+- **lora_r**: 64
+- **lora_alpha**: 16
+- **lora_dropout**: 0.1
+## bitsandbytes Parameters
+- **use_4bit**: True
+- **bnb_4bit_compute_dtype**: float16
+- **bnb_4bit_quant_type**: nf4
+- **use_nested_quant**: False
+## Training Parameters
+- **Output Directory**: ./results
+- **Number of Training Epochs**: 1
+- **Mixed-Precision Training (fp16/bf16)**: False
+- **Batch Size per GPU for Training**: 4
+- **Batch Size per GPU for Evaluation**: 4
+- **Gradient Accumulation Steps**: 1
+- **Gradient Checkpointing**: True
+- **Maximum Gradient Norm (Gradient Clipping)**: 0.3
+- **Initial Learning Rate**: 2e-4
+- **Weight Decay**: 0.001
+- **Optimizer**: paged_adamw_32bit
+- **Learning Rate Scheduler Type**: cosine
+- **Max Steps**: -1
+- **Warmup Ratio**: 0.03
+- **Group Sequences by Length**: True
+- **Save Checkpoint Every X Update Steps**: 0
+- **Log Every X Update Steps**: 25
+## License
+This model is governed by a custom commercial license from Code Llama. For details, please visit: [Custom Commercial License](https://ai.meta.com/resources/models-and-libraries/llama-downloads/)
+## Intended Use
+**Intended Use Cases**: This model is intended for commercial and research use in English. It is designed for text-to-SQL tasks, enabling users to generate SQL queries from natural language questions.
+**Out-of-Scope Uses**: Any use that violates applicable laws or regulations, use in languages other than English, or any other use prohibited by the Acceptable Use Policy and Licensing Agreement for Code Llama and its variants.
+## Model Capabilities
+- Code completion.
+- Infilling.
+- Instructions / chat.
+## Model Architecture
+Code-Llama-2-13B-instruct-text2sql is an auto-regressive language model that uses an optimized transformer architecture.
+## Model Dates
+This model was trained between January 2023 and July 2023.
+## Ethical Considerations and Limitations
+Code-Llama-2-13B-instruct-text2sql is a powerful language model, but it may produce inaccurate or objectionable responses in some instances. Safety testing and tuning are recommended before deploying this model in specific applications.
+## Hardware and Software
+- **Training Libraries**: Custom training libraries
+- **Training Hardware**: 2 V100 32GB GPUs
+- **Carbon Footprint**: Training all Code Llama models required 400K GPU hours on A100-80GB hardware with emissions offset by Meta's sustainability program.
+## Training Data
+This model was trained and fine-tuned on the same data as Llama 2 with different weights.
+## Evaluation Results
+For evaluation results, please refer to Section 3 and safety evaluations in Section 4 of the research paper.
+## Example Code
+You can use the Code-Llama-2-13B-instruct-text2sql model to generate SQL queries from natural language questions, as demonstrated in the following code snippet:
+```python
+from transformers import (
+    AutoModelForCausalLM,
+    AutoTokenizer,
+    pipeline
+)
+import torch
+model_name = 'bugdaryan/Code-Llama-2-13B-instruct-text2sql'
+model = AutoModelForCausalLM.from_pretrained(model_name, device_map='auto')
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+pipe = pipeline('text-generation', model=model, tokenizer=tokenizer)
+table = "CREATE TABLE sales ( sale_id number PRIMARY KEY, product_id number, customer_id number, salesperson_id number, sale_date DATE, quantity number, FOREIGN KEY (product_id) REFERENCES products(product_id), FOREIGN KEY (customer_id) REFERENCES customers(customer_id), FOREIGN KEY (salesperson_id) REFERENCES salespeople(salesperson_id)); CREATE TABLE product_suppliers ( supplier_id number PRIMARY KEY, product_id number, supply_price number, FOREIGN KEY (product_id) REFERENCES products(product_id)); CREATE TABLE customers ( customer_id number PRIMARY KEY, name text, address text ); CREATE TABLE salespeople ( salesperson_id number PRIMARY KEY, name text, region text ); CREATE TABLE product_suppliers ( supplier_id number PRIMARY KEY, product_id number, supply_price number );"
+question = 'Find the salesperson who made the most sales.'
+prompt = f"[INST] Write SQLite query to answer the following question given the database schema. Please wrap your code answer using ```: Schema: {table} Question: {question} [/INST] Here is the SQLite query to answer to the question: {question}: ``` "
+ans = pipe(prompt, max_new_tokens=100)
+print(ans[0]['generated_text'].split('```')[2])
+```
+This code demonstrates how to utilize the model for generating SQL queries based on a provided database schema and a natural language question. It showcases the model's capability to assist in SQL query generation for text-to-SQL tasks.