SreeBand/financial-assistant-rag

Tony Pizza Financial Assistant RAG

Introduction

This repository contains a prototype retrieval-augmented generation pipeline for a custom LLM-powered financial assistant for small local businesses. The prototype uses a synthetic pizza shop dataset called Tony Pizza to answer natural-language financial questions about sales, cash flow, payroll, ingredient costs, expenses, product performance, and sales channels. This problem is important because small-business owners often collect useful data through point-of-sale systems, payroll records, supplier invoices, employee schedules, and expense reports, but they may not have the time or accounting background to turn that data into clear decisions. A general LLM alone can struggle with this task because it may hallucinate numbers, use the wrong table, or perform arithmetic incorrectly. To address this, I built a RAG pipeline using Qwen2.5-0.5B-Instruct, retrieved table evidence from the Tony Pizza workbook, and deterministic calculation instructions. In the final evaluation, the RAG pipeline produced a moderate but meaningful improvement over the no-retrieval baseline: the full test split overall score increased from 0.252960 before retrieval to 0.319598 after retrieval, a relative improvement of about 26.3%.

Data

The data for this project is the Tony Pizza dummy cashflow workbook, a synthetic 12-month operating dataset from April 1, 2025 through March 31, 2026. The workbook represents the types of records a real restaurant or pizza shop might collect, including products, employees, daily sales, item-level sales, cash-in records, raw material purchases, payroll, fixed bills, cash-out records, monthly summaries, weekly summaries, and product summaries. For the RAG pipeline, I reformatted the Excel workbook so that each sheet became a searchable table and each row became a retrievable document containing the table name, row index, and text version of the row. I also created a 10-question internal test set from the dataset. These test cases evaluate whether the system can retrieve the correct business evidence, use deterministic calculation instructions, and produce a faithful financial explanation for questions about cash flow, cost comparison, product performance, sales performance, payroll, expenses, sales channels, and margins.

Methodology

The final method is a tool-augmented RAG pipeline rather than a fully fine-tuned model. I chose this approach because financial questions require exact evidence and calculations, and the LLM should explain results rather than invent or estimate numbers. The pipeline first converts each Tony Pizza workbook row into a searchable text document. I compared three retrieval setups: word-level TF-IDF with cosine similarity, word-level TF-IDF with dot product, and character-level TF-IDF with cosine similarity. The best retrieval configuration was character-level TF-IDF with cosine similarity, which achieved a BEIR-style Recall@5 of 0.70 and MRR of 0.625. After retrieval, the model receives the retrieved context, the user question, and deterministic calculation instructions, then responds in a three-part business explanation format. The generation model is Qwen2.5-0.5B-Instruct, run through the Hugging Face transformers text-generation pipeline with max_new_tokens=250 and do_sample=False.

Evaluation

I evaluated the system using three benchmark-style task groups that match the intended use case of the assistant. Benchmark 1: Cash Flow QA checks whether the model can answer cash-flow questions, such as identifying the week with the lowest net cash flow. Benchmark 2: Cost Comparison QA checks whether the model can compare financial changes across time, such as ingredient costs versus labor costs from June to July. Benchmark 3: Business Performance QA covers broader business questions about product performance, sales channels, payroll, expenses, sales, and margins. I also report performance on the full 10-question internal test split. The final RAG model is compared against the base Qwen2.5-0.5B-Instruct model without retrieval, plus two comparison models of similar size: SmolLM2-135M-Instruct and TinyLlama-1.1B-Chat-v1.0. These comparison models were selected based on earlier model-selection testing and then evaluated on the same 10-question no-retrieval benchmark split.

System	Role	Benchmark 1: Cash Flow QA	Benchmark 2: Cost Comparison QA	Benchmark 3: Business Performance QA	Full Test Split	Summary
Tony Pizza RAG + Qwen2.5-0.5B-Instruct	Final RAG pipeline	0.297739	0.172055	0.340773	0.319598	Best overall full-split score after adding retrieval
Qwen2.5-0.5B-Instruct without retrieval	Base no-retrieval model	0.254912	0.253575	0.252639	0.252960	Fluent responses, but lacked exact Tony Pizza evidence
SmolLM2-135M-Instruct without retrieval	Comparison no-retrieval model	0.191188	0.199466	0.221897	0.216583	Weaker baseline with lower full-split performance
TinyLlama-1.1B-Chat-v1.0 without retrieval	Comparison no-retrieval model	0.254963	0.256122	0.251772	0.252526	Similar to Qwen no-RAG, but below the final RAG system

The final RAG pipeline achieved the best full test split score, improving from 0.252960 for Qwen2.5-0.5B-Instruct without retrieval to 0.319598 after retrieval. This is an absolute improvement of 0.066638 and a relative improvement of about 26.3%. The RAG system performed especially well on broader business-performance questions, where it reached 0.340773, higher than every no-retrieval comparison model. However, Benchmark 2: Cost Comparison QA remained difficult. The RAG pipeline scored 0.172055 on that task, below the no-retrieval Qwen and TinyLlama baselines. This suggests that retrieval helped overall grounding, but exact multi-step financial comparisons still need stronger deterministic Python calculation support.

Usage and Intended Uses

This repository is intended as a class project prototype and portfolio artifact demonstrating how RAG can support small-business financial question answering. The intended use case is to answer structured questions over synthetic business data, retrieve relevant table evidence, and explain the result in a concise business-friendly format. This system is not intended to replace an accountant, financial advisor, or production accounting system.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline

model_id = "Qwen/Qwen2.5-0.5B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    dtype=torch.bfloat16
)

pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    dtype=torch.bfloat16,
    device_map="auto",
    max_new_tokens=250,
    do_sample=False,
)

prompt = """
You are a financial assistant for Tony Pizza.
Use the retrieved evidence and deterministic calculation instructions below.
Do not invent numbers.

User question:
Which product generated the highest gross profit?

Retrieved evidence:
Table: Product_Summary
Relevant row:
Product: Pepperoni Pizza
Gross Profit: 124593.54

Tool calculation:
Sort Product_Summary by Gross Profit in descending order and select the first row.

Answer in this format:
1. Direct answer
2. Evidence used
3. Business interpretation
"""

response = pipe(
    prompt,
    max_new_tokens=250,
    do_sample=False,
    return_full_text=False
)

print(response[0]["generated_text"])

Prompt Format

The prompt is formatted as a structured financial assistant instruction. It includes the user question, retrieved evidence, deterministic calculation instructions, and a required answer format.

You are a financial assistant for Tony Pizza.
Use the retrieved evidence and deterministic calculation instructions below.
Do not invent numbers.

User question:
<USER QUESTION>

Retrieved evidence:
Table: <TABLE NAME>
Relevant row or columns:
<RETRIEVED EVIDENCE>

Tool calculation:
<DETERMINISTIC PYTHON OR SQL CALCULATION DESCRIPTION>

Answer in this format:
1. Direct answer
2. Evidence used
3. Business interpretation

Expected Output Format

The expected output is a short, evidence-grounded business explanation. The model should answer directly, identify the evidence used, and explain why the result matters for Tony Pizza.

1. Direct answer:
The product with the highest gross profit was Pepperoni Pizza, with gross profit of $124,593.54.

2. Evidence used:
I used the Product_Summary table and sorted products by Gross Profit in descending order.

3. Business interpretation:
This product is a major profit driver, so Tony Pizza should monitor its ingredient costs, pricing, and promotion strategy closely.

Limitations

This project is a successful prototype because it demonstrates measurable improvement from RAG, exposes clear failure modes, and supports the next development step: moving more arithmetic into deterministic Python tools. However, it is not a production-ready financial assistant. First, the Tony Pizza dataset is synthetic, so the results do not prove performance on real restaurant financial data. Second, RAG improved the full test split score from 0.252960 to 0.319598, but it did not improve every benchmark category. The cost-comparison benchmark remained difficult because it required exact arithmetic across monthly values, and the final RAG system scored lower on that task than the no-retrieval comparison models. The table mention score also remained 0.000000, meaning the model often failed to explicitly name the source table in its final response. For this financial assistant, the best future version should combine RAG with stricter table filtering, stronger deterministic Python calculations, shorter retrieved context, and required table/value citations in every response.

Files in This Repository

README.md                              # Main model card / repository documentation
requirements.txt                       # Python dependencies
rag_pipeline.py                        # Minimal RAG pipeline implementation
prompt_template.txt                    # General prompt template
expected_output_template.txt           # Expected response template
tony_pizza_dummy_cashflow_dataset.xlsx # Synthetic Tony Pizza financial dataset

Citation and Source Links

• Base model: Qwen2.5-0.5B-Instruct
• Comparison model: SmolLM2-135M-Instruct
• Comparison model: TinyLlama-1.1B-Chat-v1.0
• Framework: Hugging Face Transformers
• Retrieval/evaluation inspiration: BEIR, KILT, and RAGAS