AyuK007/gemma3-270m-spider-text2sql-lora-merged

Gemma-3-270M Text-to-SQL (Custom LoRA Merged)

This repository hosts a specialized, fine-tuned version of Google's Gemma 3 270M foundation model optimized for Text-to-SQL conversion tasks.

Instead of relying on high-level parameter-efficient fine-tuning (PEFT) frameworks, this model was optimized by injecting custom low-rank parameter adapters ($r=8, \alpha=16$) written entirely from scratch in raw PyTorch. Following training, the adapter pathways were mathematically merged back into the original weights by reference ($W_{\text{final}} = W_0 + \Delta W$) to yield a standalone model artifact with zero external code dependencies.

Model Details

• Developed by: [Your Name/Profile]
• Model Type: Causal Language Model (Transformer Decoder)
• Base Architecture: google/gemma-3-270m
• Language(s): English (Primary)
• Fine-tuning Task: Text-to-SQL (Semantic Parsing)
• Primary Optimization Dataset: SuperMax991/spider-text2sql (Subset of 1,000 samples)

---

Intended Uses & Limitations

Direct Intent

This model is directly intended for lightweight, edge-compatible deployment environments to translate standard English questions into clean, execution-ready SQLite queries based on an explicit structural database schema context.

Out-of-Scope Uses

• Direct execution of unvetted, un-sanitized generated code strings against real-world production databases without human-in-the-loop review.
• General knowledge retrieval, open-domain chat conversations, or standard Python/JavaScript code generation.

Systemic Limitations

As an ultra-lightweight 270M parameter model, its operational horizon is constrained. While it handles standard queries, aggregations (COUNT, SUM), and simple inner joins flawlessly, execution accuracy may drop when it faces multi-level nested subqueries, uncommon mathematical operators, or vast schema maps exceeding 10 tables simultaneously.

---

Technical Training Profile

Hyperparameters & Architecture

• Target Modules: Query, Key, Value, and Output linear projections (q_proj, k_proj, v_proj, o_proj).
• Adapter Configuration: Rank ($r$) = 8, Alpha ($\alpha$) = 16, Scaling Factor ($\frac{\alpha}{r}$) = 2.0.
• Global Batch Size: 1 (with 16 Gradient Accumulation Steps simulating an effective batch size of 16).
• Learning Rate: 2e-4 (Using AdamW optimizer acting exclusively on active adapter parameters).
• Loss Target Tracking: Cross-Entropy loss computed exclusively on output SQL token indices by masking the context sequence with an ignore index boundary value of -100.

Convergence Curve

• Epoch 1 Average Loss: 0.74627
• Epoch 2 Average Loss: 0.42663 (Reflecting direct stabilization and accurate keyword alignment)

---

How to Use & Prompt Format

Because this is a merged model, it retains the exact native architecture profile of Gemma 3. It can be initialized natively out-of-the-box using the standard Hugging Face transformers environment.

Prompt Template

To ensure correct prediction formats, you must supply database contexts using the schema prefix template implemented during the optimization run:

### Context Schema:
[Insert Table Definitions and Data Types Here]

### Question:
[Insert Natural Language Question Here]

### SQL:
---

Python Inference Script Example

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

# Target repository identifier string
model_id = "your-username/gemma3-270m-spider-text2sql-standalone"

# Load unified model parameters and tokenizer configurations
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float16,
    device_map="auto"
)

# Structure a sample execution problem
schema = "Table: departments (dept_id INT, name VARCHAR); Table: employees (emp_id INT, name VARCHAR, dept_id INT, salary INT)"
question = "List the names of all employees working in the Sales department."

# Construct the schema-aware prompt format
prompt = f"### Context Schema:\n{schema}\n\n### Question:\n{question}\n\n### SQL:\n"

# Tokenize prompt inputs
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# Generate predictions deterministically using greedy decoding
model.eval()
with torch.no_grad():
    output_tokens = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=False,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id
    )

# Decode output token sequences back to string text format
full_output_text = tokenizer.decode(output_tokens[0], skip_special_tokens=True)

# Extract only the predicted SQL statement string extension
predicted_sql = full_output_text[len(prompt):].strip()
print(f"Generated Query: {predicted_sql}")

---

Licensing & Citation

Use of this model is subject to the standard Google Gemma Terms of Use. If you use this model in research or downstream text-to-SQL workflows, please reference the foundational elements below:

@misc{gemma_3_2024,
  title={Gemma 3: Open Models from Google},
  author={Google DeepMind},
  year={2026}
}

@inproceedings{yu2018spider,
  title={Spider: A Large-Scale Hierarchical Semantic Parsing and Text-to-SQL Dataset on Cross-Domain Databases},
  author={Yu, Tao and others},
  booktitle={EMNLP},
  year={2018}
}