Sreemaan05/ai-data-analyst-ml-engine

AI Data Analyst ML Engine 🧠📊

A production-ready, hybrid ML + deterministic pipeline for dataset-agnostic business data analysis.
Built for the AI Data Engineer platform's Data Analysis tab.

🏗️ Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                         User Request                                     │
│        Question: "Show me sales performance by country"                  │
│        Dataset:  uploaded CSV file                                        │
└────────────────────────────┬────────────────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                      FastAPI Router (api.py)                              │
│                POST /analysis/ml-plan                                     │
└────────────────────────────┬────────────────────────────────────────────┘
                             │
           ┌─────────────────┼─────────────────┐
           │                 │                 │
           ▼                 ▼                 ▼
    ┌──────────────┐  ┌──────────────┐  ┌──────────────┐
    │ ML Layer     │  │ Deterministic│  │ ML Layer     │
    │              │  │ Layer        │  │              │
    │ 1. Intent    │  │              │  │ 4. Insight   │
    │    Classify  │  │ 3. Stats     │  │    Generate  │
    │ (DeBERTa)   │  │    Engine    │  │ (FLAN-T5)   │
    │              │  │ (pandas)     │  │              │
    │ 2. Column    │  │              │  │ 5. Follow-up │
    │    Match     │  │ KPIs, aggs,  │  │    Questions │
    │ (MiniLM)    │  │ trends,      │  │ (FLAN-T5 +  │
    │              │  │ correlations │  │  templates)  │
    └──────┬───────┘  └──────┬───────┘  └──────┬───────┘
           │                 │                 │
           └─────────────────┼─────────────────┘
                             │
                             ▼
                  ┌──────────────────┐
                  │ Chart Recommender │
                  │ (Rules + Types)   │  ← No ML needed
                  └────────┬─────────┘
                           │
                           ▼
              ┌────────────────────────┐
              │    JSON Response       │
              │  • intent              │
              │  • relevant_columns    │
              │  • charts[]            │
              │  • insights[]          │
              │  • follow_up_questions │
              │  • statistics          │
              │  • data_quality        │
              └────────────────────────┘

Design Principles

Principle	Implementation
Hybrid approach	ML for reasoning/planning, pandas for all numbers
No hallucinated numbers	ML models never compute stats — they narrate pre-computed ones
No hallucinated columns	Column detection validates against actual df.columns
Dataset-agnostic	Zero-shot models + heuristic type detection work on any CSV
Fast on CPU	All models run on CPU with < 5s per component
Fallback logic	Low-confidence ML predictions fall back to deterministic defaults

---

🤖 Model Choices & Reasoning

1. Intent Classification — MoritzLaurer/deberta-v3-base-zeroshot-v2.0

Property	Details
Why this model	Best zero-shot classifier on HF Hub. F1=0.619 across 28 tasks, beating BART-large-mnli (0.497) at 1/4 the size. Based on Laurer et al. 2023.
Size	352 MB (safetensors)
CPU latency	~1-2 seconds
How it works	NLI-based: "This question is about [comparing groups]" → entailment probability = intent score
Fallback	Keyword boosting for low-confidence cases; defaults to summary_analysis
Alternative	MoritzLaurer/deberta-v3-xsmall-zeroshot-v1.1-all-33 (83MB ONNX, ~0.5s, slightly lower accuracy)

2. Column Matching — sentence-transformers/all-MiniLM-L6-v2

Property	Details
Why this model	Most downloaded embedding model on HF (200M+). 384-dim embeddings, 22M params. Trained on 1B sentence pairs including Q&A data.
Size	87 MB (22 MB ONNX quantized)
CPU latency	< 50ms for batch encoding
How it works	Encodes question + enriched column descriptions → cosine similarity ranking
Key trick	Column descriptions are enriched with dtype, sample values, and statistics for much better matching
Alternative	all-MiniLM-L12-v2 (129MB, +3-5% accuracy) or all-mpnet-base-v2 (420MB, best accuracy)

3. Insight Generation — google/flan-t5-base

Property	Details
Why this model	Instruction-tuned on 1000+ tasks, excellent at structured generation. Seq2seq (no KV cache pressure on CPU). Predictable, fast output.
Size	944 MB
CPU latency	~1-3 seconds
How it works	Receives pre-computed statistics as prompt context → generates business narrative
Guard rails	Only sees computed numbers, never raw data. Template insights supplement ML output.
Alternative	Qwen/Qwen2.5-0.5B-Instruct via GGUF (350MB, richer narrative, 5-10s)

4. Chart Recommendation — Pure Rules

Property	Details
Why no ML	Visualization grammar is a solved problem (Vega-Lite, Draco, CompassQL). Rules are faster, more deterministic, and never hallucinate.
How it works	Intent → candidate charts → column type compatibility filter → axis assignment
Coverage	18 chart types with automatic axis assignment

Total Model Download: ~1.4 GB (first run only, cached after)

---

📁 Project Structure

ai-data-analyst-ml-engine/
├── analysis_ml_engine.py   # Core ML + deterministic engine (5 main functions)
├── api.py                  # FastAPI application with all endpoints
├── test_pipeline.py        # Comprehensive test suite (5 dataset types)
├── requirements.txt        # Python dependencies
└── README.md               # This file

---

⚡ Quick Start (Windows)

Prerequisites

• Python 3.10+
• pip
• ~2 GB disk space for models (downloaded automatically on first run)
• ~4 GB RAM recommended

Step 1: Clone or Download

# Option A: Clone from HF Hub
git clone https://huggingface.co/Sreemaan05/ai-data-analyst-ml-engine
cd ai-data-analyst-ml-engine

# Option B: Download individual files from
# https://huggingface.co/Sreemaan05/ai-data-analyst-ml-engine/tree/main

Step 2: Create Virtual Environment

# Windows PowerShell
python -m venv venv
.\venv\Scripts\Activate.ps1

# Or CMD
python -m venv venv
venv\Scripts\activate.bat

Step 3: Install Dependencies

pip install -r requirements.txt

Step 4: Quick Test (No ML models needed)

# Test deterministic logic only — instant, no downloads
python test_pipeline.py --quick

Step 5: Full Test (Downloads ML models on first run)

# Full test with ML models (~1.4 GB download on first run)
python test_pipeline.py

# Generate example JSON output
python test_pipeline.py --example

Step 6: Start the API Server

# Start FastAPI server
uvicorn api:app --host 0.0.0.0 --port 8000

# Or with auto-reload for development
uvicorn api:app --host 0.0.0.0 --port 8000 --reload

# Pre-load all models at startup (slower start, faster first request)
set PRELOAD_MODELS=true
uvicorn api:app --host 0.0.0.0 --port 8000

Step 7: Test the API

# Upload a CSV
curl -X POST http://localhost:8000/upload -F "file=@your_data.csv"
# Returns: {"dataset_id": "dataset_abc123", ...}

# Run analysis
curl -X POST http://localhost:8000/analysis/ml-plan \
  -H "Content-Type: application/json" \
  -d '{"question": "Show me sales by country", "dataset_id": "dataset_abc123"}'

Or open http://localhost:8000/docs for the interactive Swagger UI.

---

🔌 API Reference

POST /upload

Upload a CSV file. Returns dataset_id for subsequent requests.

POST /analysis/ml-plan ⭐ Main Endpoint

Full analysis pipeline. Takes question + dataset_id, returns everything.

Request:

{
  "question": "Show me sales by country",
  "dataset_id": "dataset_abc123"
}

Response:

{
  "question": "Show me sales by country",
  "intent": {
    "label": "comparison",
    "confidence": 0.8234,
    "method": "ml"
  },
  "relevant_columns": {
    "target_metric": "Amount",
    "dimensions": ["Country", "Product"],
    "time_column": "Date",
    "kpi_columns": ["Amount", "Boxes Shipped"],
    "segment_columns": ["Country", "Sales Person", "Product"]
  },
  "charts": [
    {
      "chart_type": "bar",
      "title": "Amount by Country",
      "x": "Country",
      "y": "Amount",
      "reason": "Best for comparing values across categories"
    },
    {
      "chart_type": "grouped_bar",
      "title": "Amount by Country and Product",
      "x": "Country",
      "y": "Amount",
      "series": "Product",
      "reason": "Shows side-by-side comparison across two dimensions"
    },
    {
      "chart_type": "heatmap",
      "title": "Amount Heatmap: Country vs Product",
      "x": "Country",
      "y": "Product",
      "value": "Amount",
      "reason": "Shows intensity patterns across two categorical dimensions"
    },
    {
      "chart_type": "pie",
      "title": "Amount Distribution by Country",
      "x": "Country",
      "y": "Amount",
      "reason": "Displays proportional share of each category"
    }
  ],
  "insights": [
    "India leads in Country with a total of 31,100.00 (34.5% of total).",
    "The top 3 Country values (India, USA, Germany) account for 81.0% of total.",
    "The total Amount is 90,200.00 across 20 records, with an average of 4,510.00."
  ],
  "follow_up_questions": [
    "How has Amount changed over time?",
    "What is the breakdown of Amount by Product?",
    "Which Country are underperforming compared to the average Amount?",
    "Is there a correlation between Amount and Boxes Shipped?"
  ],
  "data_quality": {
    "total_rows": 20,
    "total_columns": 6,
    "missing_pct": 0.0,
    "duplicate_rows": 0
  },
  "processing_time_ms": 3200.5
}

POST /analysis/intent — Intent only

POST /analysis/columns — Column detection only

POST /analysis/charts — Chart recommendations only

GET /health — Health check

GET /models/status — Model loading status

GET /datasets — List uploaded datasets

---

🔗 Integration with Your Existing Project

Option A: Import as Python Module

# In your existing FastAPI app
from analysis_ml_engine import analyze

@app.post("/your-existing-route")
def your_handler(question: str, df: pd.DataFrame):
    result = analyze(question, df)
    # result contains everything: intent, columns, charts, insights, follow-ups
    return result

Option B: Use Individual Functions

from analysis_ml_engine import (
    classify_question_intent,
    detect_relevant_columns,
    recommend_charts,
    compute_statistics,
    generate_insights,
    generate_followup_questions,
)

# Step 1: Classify intent
intent = classify_question_intent("Show me sales by country")

# Step 2: Find relevant columns
columns = detect_relevant_columns("Show me sales by country", df)

# Step 3: Compute stats (deterministic)
stats = compute_statistics(df, intent["intent"], columns)

# Step 4: Get chart recommendations (rule-based)
charts = recommend_charts("Show me sales by country", df, intent["intent"], columns)

# Step 5: Generate insights (ML + templates)
insights = generate_insights("Show me sales by country", df, stats)

# Step 6: Generate follow-up questions
followups = generate_followup_questions("Show me sales by country", df, insights, columns)

Option C: Mount as Sub-Application

# In your main FastAPI app
from api import app as ml_app

main_app = FastAPI()
main_app.mount("/ml", ml_app)
# Now accessible at /ml/analysis/ml-plan, /ml/upload, etc.

Apache ECharts Integration

The chart specs returned by the API map directly to ECharts options:

// Frontend: Convert chart spec to ECharts option
function chartSpecToECharts(spec) {
  const option = {
    title: { text: spec.title },
    tooltip: { trigger: 'axis' },
  };

  switch (spec.chart_type) {
    case 'bar':
    case 'horizontal_bar':
      option.xAxis = { type: spec.chart_type === 'horizontal_bar' ? 'value' : 'category', data: [] };
      option.yAxis = { type: spec.chart_type === 'horizontal_bar' ? 'category' : 'value' };
      option.series = [{ type: 'bar', data: [] }];
      break;
    case 'line':
    case 'area':
      option.xAxis = { type: 'category', data: [] };
      option.yAxis = { type: 'value' };
      option.series = [{ type: 'line', areaStyle: spec.chart_type === 'area' ? {} : undefined, data: [] }];
      break;
    case 'pie':
      option.series = [{ type: 'pie', data: [] }];
      break;
    case 'scatter':
      option.xAxis = { type: 'value' };
      option.yAxis = { type: 'value' };
      option.series = [{ type: 'scatter', data: [] }];
      break;
    case 'heatmap':
      option.visualMap = { min: 0, max: 100 };
      option.xAxis = { type: 'category', data: [] };
      option.yAxis = { type: 'category', data: [] };
      option.series = [{ type: 'heatmap', data: [] }];
      break;
  }

  return option;
}

---

🎯 Fine-Tuning Plan

When to Fine-Tune

Fine-tuning is optional — the zero-shot pipeline works well out of the box.
Consider fine-tuning when:

1. Intent accuracy drops below 80% on your specific domain
2. You have consistent misclassifications for specific question patterns
3. You want faster inference (smaller fine-tuned model > larger zero-shot model)

Intent Classification Fine-Tuning

Dataset Format:

{"text": "Show me the top products by revenue", "label": "ranking"}
{"text": "How did sales change quarter over quarter?", "label": "trend_analysis"}
{"text": "Compare East vs West region performance", "label": "comparison"}

Label Schema: 10 intents (see INTENT_LABELS in analysis_ml_engine.py)

Minimum Examples: 50 per class = 500 total (recommended: 200+ per class)

Generating Synthetic Training Data:

# Use FLAN-T5 or GPT to generate diverse question phrasings
from analysis_ml_engine import ModelManager

models = ModelManager()

INTENTS_WITH_SEEDS = {
    "ranking": [
        "Which products sell the most?",
        "Top 10 customers by spending",
        "What are the highest revenue items?",
    ],
    "trend_analysis": [
        "How have sales changed over time?",
        "Show monthly revenue trends",
        "Is there seasonal variation in orders?",
    ],
    # ... add seeds for all 10 intents
}

synthetic_data = []
for intent, seeds in INTENTS_WITH_SEEDS.items():
    for seed in seeds:
        prompt = f"Generate 10 different business questions similar to: '{seed}'. Return as numbered list."
        # Use FLAN-T5 or GPT API to generate variations
        variations = generate_variations(prompt)
        for v in variations:
            synthetic_data.append({"text": v, "label": intent})

Fine-Tuning Script:

from transformers import AutoModelForSequenceClassification, AutoTokenizer, TrainingArguments, Trainer
from datasets import Dataset

model_name = "microsoft/deberta-v3-base"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=10)

dataset = Dataset.from_list(synthetic_data)
dataset = dataset.map(lambda x: tokenizer(x["text"], truncation=True, padding="max_length", max_length=128))

trainer = Trainer(
    model=model,
    args=TrainingArguments(
        output_dir="./intent_model",
        num_train_epochs=5,
        per_device_train_batch_size=16,
        learning_rate=2e-5,
        warmup_ratio=0.1,
        evaluation_strategy="epoch",
        save_strategy="epoch",
        load_best_model_at_end=True,
        metric_for_best_model="accuracy",
    ),
    train_dataset=dataset,
    eval_dataset=dataset,  # Use a proper split
)
trainer.train()

Evaluation Metrics:

• Accuracy (macro)
• F1-score per intent class
• Confusion matrix to identify systematic misclassifications

Column Matching Fine-Tuning

Generally not recommended — the zero-shot embedding approach works well.
If needed, create (question, column_name, relevance_score) triplets and fine-tune with sentence-transformers contrastive loss.

---

📏 Evaluation Plan

Metrics by Component

Component	Metric	Target	How to Measure
Intent Classification	Macro F1	≥ 80%	Compare predicted vs. labeled intent on test set
Column Detection	Precision@3	≥ 70%	Top 3 detected columns vs. human-labeled relevant columns
Column Detection	Recall	≥ 85%	Fraction of truly relevant columns that were detected
Chart Recommendation	Human rating (1-5)	≥ 3.5	Evaluate appropriateness of chart type for question/data
Insight Faithfulness	Factual accuracy	100%	Verify all numbers in insights match computed statistics
Insight Quality	Human rating (1-5)	≥ 3.5	Clarity, specificity, actionability of insights
Follow-up Questions	Relevance score	≥ 80%	Fraction of questions judged as useful by human evaluator

Test Datasets

The test suite (test_pipeline.py) covers 5 dataset types:

1. Sales dataset — Products, countries, salesperson, amounts, dates
2. E-commerce dataset — Categories, segments, regions, profit/loss, discounts
3. Hotel booking dataset — Room types, meal plans, cancellations, daily rates
4. Customer churn dataset — Demographics, services, contract types, churn labels
5. Logistics dataset — Origins, destinations, carriers, delivery times, costs

Running Evaluation

# Full evaluation with all ML models
python test_pipeline.py

# Quick check (deterministic only, no model download)
python test_pipeline.py --quick

# With example output
python test_pipeline.py --example

Generalization Test Protocol

To test on a new dataset:

1. Save your CSV file
2. Run:

import pandas as pd
from analysis_ml_engine import analyze

df = pd.read_csv("your_new_data.csv")
result = analyze("Your business question here", df)

# Check: are the detected columns real?
for col in result["relevant_columns"]["kpi_columns"]:
    assert col in df.columns, f"Hallucinated column: {col}"

# Check: do insights contain accurate numbers?
print(result["insights"])

---

⚡ Performance Optimization

For Faster CPU Inference

# Install ONNX Runtime for 2-3x speedup on encoder models
pip install optimum[onnxruntime]

Then modify ModelManager to use ONNX:

# In analysis_ml_engine.py, replace the classifier loader with:
from optimum.onnxruntime import ORTModelForSequenceClassification

model = ORTModelForSequenceClassification.from_pretrained(
    "MoritzLaurer/deberta-v3-xsmall-zeroshot-v1.1-all-33",
    subfolder="onnx",
    file_name="model_quantized.onnx"
)

For Better Insight Quality

Replace FLAN-T5 with Qwen2.5 via llama-cpp-python:

pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="Qwen/Qwen2.5-0.5B-Instruct-GGUF",
    filename="*q4_k_m.gguf",
    n_ctx=2048,
    n_threads=4,
    verbose=False,
)

response = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a data analyst. Generate business insights from statistics."},
        {"role": "user", "content": f"Statistics: {stats}\nQuestion: {question}\nInsight:"}
    ],
    max_tokens=200,
)

Caching Column Embeddings

For repeated queries on the same dataset:

import hashlib
import pickle

# Cache embeddings keyed by dataset hash
_embedding_cache = {}

def get_cached_embeddings(df, embedder):
    key = hashlib.md5(pd.util.hash_pandas_object(df).values.tobytes()).hexdigest()
    if key not in _embedding_cache:
        descriptions = _build_column_descriptions(df)
        _embedding_cache[key] = embedder.encode(
            list(descriptions.values()),
            convert_to_tensor=True,
            normalize_embeddings=True,
        )
    return _embedding_cache[key]

---

🛡️ Constraint Compliance

Constraint	How It's Met
No hallucinated columns	Column detection validates against df.columns; test asserts this
Only actual columns used	Chart specs reference columns from detect_relevant_columns output
Works on unseen CSVs	Zero-shot models + heuristic type detection; tested on 5 different domains
Returns structured JSON	Pydantic models enforce schema; all outputs are JSON-serializable
Fast for web app	Intent: ~1-2s, Columns: <0.5s, Charts: <1ms, Insights: ~2-3s, Total: ~5-8s
Handles missing values	dropna() in all aggregations; missing value report in data quality
Handles messy column names	Rich column descriptions include samples and stats for better matching
Fallback on low confidence	Intent defaults to summary_analysis; column matching uses type-based heuristics
Deterministic numbers	All KPIs, aggregations, trends computed by pandas
ML for reasoning only	ML used for intent, column matching, and narrative — never for math

---

📜 License

MIT — Use freely in your AI Data Engineer project.

---

🔗 Links

• Repository: Sreemaan05/ai-data-analyst-ml-engine
• Models Used:
  • MoritzLaurer/deberta-v3-base-zeroshot-v2.0 — Intent Classification
  • sentence-transformers/all-MiniLM-L6-v2 — Column Matching
  • google/flan-t5-base — Insight Generation