Instructions to use AaronTekle/PySparkQwen with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use AaronTekle/PySparkQwen with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-1.5B-Instruct") model = PeftModel.from_pretrained(base_model, "AaronTekle/PySparkQwen") - Notebooks
- Google Colab
- Kaggle
PySparkQwen: Qwen/Qwen2.5-Coder-1.5B-Instruct Fine-Tuning with QLoRA
QLoRA fine-tuning of Qwen/Qwen2.5-Coder-1.5B-Instruct for PySpark code generation.
PySparkQwen is a PySpark coding model, that is used for data engineering tasks, including DataFrame transformations, joins, Spark SQL, structured streaming, Delta Lake, performance optimization, MLlib, and distributed processing.
Given:
- PySpark task (prompt)
- DataFrame or table schemas, column names (and/or relevant runtime constraints)
Training note: The base model was loaded with 4-bit NF4 quantization and adapted locally using parameter-efficient LoRA weights. This repository uses a PEFT adapter, not a standalone copy of the base model.
Example
DataFrame context:
DataFrame: orders
Columns: customer_id (string), order_id (string), amount (double)
Prompt:
Calculate total revenue per customer and return the five customers with the highest revenue.
Expected Model Output:
from pyspark.sql import functions as F
result = (
orders.groupBy("customer_id")
.agg(F.sum("amount").alias("total_revenue"))
.orderBy(F.col("total_revenue").desc())
.limit(5)
)
Dataset: Synthetic PySpark Code Tasks
| Dataset statistic | Count |
|---|---|
| Total examples | 900 |
| Training examples | 600 |
| Validation examples | 150 |
| Held-out test examples | 150 |
| Task categories | 15 |
| Algorithm families | 180 |
Dataset is balanced across 15 task categories, with 60 examples per category (40 training, 10 validation, and 10 test). It covers:
- DataFrame creation and transformations
- Data ingestion and file formats
- Data cleaning and preprocessing
- Filtering, sorting, and aggregation
- Joins and window functions
- Spark SQL
- UDFs and Pandas UDFs
- Structured Streaming
- Performance optimization and caching
- Partitioning and repartitioning
- Machine learning with MLlib
- Nested and complex data types
- Error handling and debugging
- Delta Lake operations
- Distributed computing and cluster execution
Note: All 900 reference snippets passed Python ast.parse syntax checks.
Model Evaluation
The original Qwen2.5-Coder model and the fine-tuned adapter were evaluated on the same 150 held-out synthetic tasks, using the same prompt construction and greedy decoding (do_sample=False) with an output limit of 384 new tokens.
Greedy decoding selects the highest-probability next token at each step, making this comparison deterministic under the same runtime conditions. It does not search for the globally best full program.
Held-out Test Set (150 Examples)
| Metric | Base Qwen2.5-Coder-1.5B-Instruct | Fine-tuned PySparkQwen-Coder |
|---|---|---|
| Evaluated examples | 150 | 150 |
| Python AST-valid outputs | 90.00% (135/150) | 98.67% (148/150) |
| Mean character-sequence similarity to reference | 0.1168 | 0.6621 |
Model Limitations
- Missing context: In rare cases (without context): The model may invent columns, tables, file paths, or dependencies if the task or schema is incomplete.
- Review before use: Generated code should be reviewed and tested in a controlled Spark environment before running against production data.
Example Import (PEFT)
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
# Model repositories
base_model = "Qwen/Qwen2.5-Coder-1.5B-Instruct"
adapter_model = "AaronTekle/PySparkQwen"
# Load the original Qwen2.5-Coder tokenizer
tokenizer = AutoTokenizer.from_pretrained(base_model)
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
# Configure 4-bit quantization for lower GPU memory usage
compute_dtype = (
torch.bfloat16
if torch.cuda.is_bf16_supported()
else torch.float16
)
quantization = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=compute_dtype,
)
# Load the base Qwen2.5-Coder model
base_model = AutoModelForCausalLM.from_pretrained(
base_model,
quantization_config=quantization,
device_map="auto",
)
# Load the fine-tuned PySparkQwen LoRA adapter
model = PeftModel.from_pretrained(
base_model,
adapter_model,
)
model.eval()
# Example PySpark coding task
context = (
"DataFrame orders: customer_id string, "
"order_id string, amount double"
)
question = (
"Calculate total revenue per customer "
"and show the top five customers."
)
# Define coding assistant behavior
system_prompt = (
"You are a PySpark data engineering coding assistant. "
"Return only executable Python/PySpark code with necessary imports. "
"Prefer Spark DataFrame APIs over pandas and RDDs. "
"Use the provided schemas and names without inventing tables or columns. "
)
messages = [
{"role": "system", "content": system_prompt},
{
"role": "user",
"content": f"{question}\n\nContext / schema:\n{context}",
},
]
# Apply Qwen2.5-Coder's native instruct chat template
prompt = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
)
# Tokenize the prompt and move tensors to the model's device
inputs = tokenizer(
prompt,
return_tensors="pt",
).to(model.get_input_embeddings().weight.device)
# Generate PySpark code using greedy decoding
with torch.inference_mode():
output_ids = model.generate(
**inputs,
max_new_tokens=384,
do_sample=False,
pad_token_id=tokenizer.pad_token_id,
eos_token_id=tokenizer.convert_tokens_to_ids("<|im_end|>"),
)
# Decode only newly generated tokens
new_tokens = output_ids[0][inputs["input_ids"].shape[-1]:]
generated_code = tokenizer.decode(
new_tokens,
skip_special_tokens=True,
).strip()
print("Generated PySpark Code:")
print(generated_code)
References
- Qwen Team. Qwen2.5-Coder-1.5B-Instruct Model Card
- Hui et al. Qwen2.5-Coder Technical Report (2024)
- Hu et al. LoRA: Low-Rank Adaptation of Large Language Models (2021)
- Dettmers et al. QLoRA: Efficient Finetuning of Quantized LLMs (2023)
- Hugging Face. TRL SFTTrainer
- Hugging Face. PEFT LoRA
- Hugging Face. PEFT Adapters in Transformers
- Downloads last month
- 6
Model tree for AaronTekle/PySparkQwen
Base model
Qwen/Qwen2.5-1.5B