Apug98/boardgame-rag-qa

Board Game Rules Assistant

1. Introduction

Board game rulebooks are often lengthy and complex, with important rules distributed across multiple sections. As a result, learning a new game and explaining its setup and rules to new players can be a significant barrier to gameplay. While current Large Language Models (LLMs) have likely seen board game rulebooks while training, they tend to conflate rules from different games or hallucinate rules when asked specified questions about gameplay. To solve this, I developed the Board Game Rules Assistant, a system designed specifically to answer questions about board game set up, general gameplay, and specific rules, grounded only in the official game rulebooks. This project utilizes a hierarchical parent-child Retrieval-Augmented Generation (RAG) pipeline to ensure response accuracy by retrieving specific rules from official board game rulebook PDFs and using those rules to answer user questions. In addition to the RAG system itself, this repository includes scripts for parsing PFDs into Markdown files and chunking Markdown file based on Markdown headings (levels 2-4).

Overall, the results show that grounding responses in source text can improve response accuracy and reduce hallucinations compared with using a generation model alone.

2. Data

The knowledge base for this RAG pipeline consists of text extracted from official board game rulebook PDFs using the included parsing script. These rulebooks were obtained directly from the publishers’ websites. After parsing, the resulting Markdown files were manually validated to ensure the text accurately reflected the original documents. Other than minor formatting changes made to preserve the structure of the PDFs in Markdown, the text was not edited or rephrased. The current knowledge base contains rulebooks for the following games: Betrayal at House on the Hill (2nd Edition), Blokus, Catan, Clue, Codenames, Hues and Cues, Monopoly, Really Loud Librarian, Ticket to Ride, and Wingspan.

Copyright notice: The rulebooks are copyrighted by their respective publishers and are included solely as the retrieval corpus for this project. Ownership of the underlying game content remains with the respective copyright holders.

To evaluate the RAG pipeline, I created a custom evaluation dataset consisting of 50 question-answer pairs derived from the official rule books. The dataset contains five questions for each game in the knowledge base, covering topics ranging from specific gameplay rules to general setup and gameplay overviews. It also includes unanswerable questions, whose answers were intentionally absent from the knowledge base, to evaluate the model’s ability to refrain from answering when sufficient information is unavailable. I authored all evaluation questions and most of the reference answers. For the general gameplay overview questions, I provided the relevant rulebook chunks to OpenAI's ChatGPT (GPT-5.5) using the prompt: “Using the provided context only, answer the question: how do I play <board game name>?” All generated responses were manually reviewed and edited for factual accuracy, clarity, and conciseness before being included in the evaluation dataset.

Due to copyright and data licensing restrictions, the knowledge base and evaluation dataset are not included in this repository. Please see Section 9: Data Licensing for more information.

3. Methodology

This project implements a hierarchical parent-child RAG pipeline to leverage the natural hierarchical structure of board game rulebooks. Each rulebook is first divided into larger parent chunks based on Markdown headings (levels 2-4). Parent chunks that exceed a predefined size threshold are then recursively split into smaller child chunks while preserving metadata that links each child to its corresponding parent.

Only the child chunks are embedded in the vector database. During retrieval, the user query is compared against the child chunk using cosine similarity with k=5 to identify the most relevant passages. The corresponding parent chunk or chunks are then retrievethrough the metadata links and provided to the generation model as context.

This approach allows semantic search to operate over fine-grained passages while avoiding the computational cost of embedding large parent documents. At the same time, the generation model receives broader contextual information than would be available from the retrieved child chunks alone, increasing the likelihood of producing coherent and well-grounded responses.

I selected Qwen3-Embedding-4B as the embedding model because I found it achieved higher scores on context relevance, context recall, and answer relevance compared to the other embedding model I tested.

I selected Qwen3-4B-Instruct-2507 as the generation model because during preliminary benchmarking, it outperformed the larger Qwen2.5-7B-Instruct model while requiring fewer computational resources.

4. Evaluation

To evaluate the retrieval performance and response quality of the RAG pipeline, I utilized three established benchmarks: RAGBench, LaRA, and RAGTruth, in addition to my own custom dataset. I selected these datasets because they target common failure reasons of RAG systems: retrieving irrelevant context, failing to retrieve relevant information, and generating hallucinated or unsupported responses.

I. RAGBench: I used the test split of the DelucionQA dataset. This dataset was developed to evaluate hallucinations and consists of question-answer pairs based on the Jeep Gladiator 2023 Owner’s Manual. Unlike a traditional RAG benchmark, the dataset provides the relevant context directly in each example rather than as a document corpus and does not include ground truth answers. I modified the dataset only by removing columns that were not used during evaluation and renaming columns for consistency across benchmarks.

II. LaRA: I used the 32k paper corpus together with all four 32k_paper query files. To evaluate retrieval under the same conditions as my board game knowledge base, I processed the document corpus using the chunking strategy described in Section 2. The query files were loaded into Pandas DataFrames, and I added a source_file column to preserve the mapping between each query and its original Markdown document.

III. RAGTruth: I used the QA tasks from the source_info.json file. This dataset stores the question and supporting passages together and includes a prompt column containing dataset-specific instructions. To use RAGTruth within a consistent evaluation pipeline, I separated the question from the supporting passages and replaced the dataset prompt with the same prompt used for the other benchmarks. Like RAGBench, RAGTruth does not include a ground truth response.

All datasets were evaluated using gpt-4o-mini (OpenAI) as the judge for several metrics from the RAGAS framework.

• Context Relevance: Measures whether the retrieved context is relevant to the user’s query.
• Context Recall: Measures how much of the information required to answer the query is contained in the retrieved context.
• Faithfulness: Measures whether the generated answer is supported by the retrieved context.
• Answer Relevance: Measures how well the generated answer addresses the user's query.
• Answer Correctness: Measures how closely the generated answer matches the reference answer.

Because the three benchmarks differ in the data they provide, not every RAGAS metric was computed for every dataset. Specifically, Answer Correctness was only evaluated on datasets containing reference answers. Benchmarks such as DelucionQA, which do not provide ground-truth answers, were evaluated using only the metrics that could be computed from the available date. This ensured that each benchmark was evaluated using metrics appropriate to its design while maintaining a consistent evaluation pipeline.

Benchmark	LaRA			Board Game Rules			RAGBench			RAGTruth
Model	Qwen	Llama	Microsoft	Qwen	Llama	Microsoft	Qwen	Llama	Microsoft	Qwen	Llama	Microsoft
Faithfulness	80.08	75.05	84.68	89.67	81.79	78.26	93.72	89.39	92.67	91.73	84.62	90.09
Answer Relevance	52.06	47.98	54.06	66.99	56.55	62.09	78.12	77.45	75.79	65.51	55.83	59.40
Context Relevance	67.44	68.83	66.97	80.65	81.26	79.85
Context Recall	78.89	81.32	79.54	71.46	72.49	71.03
Answer Correctness	47.17	47.49	47.99	47.57	45.18	38.80

To further evaluate performance, I compared my primary model, Qwen3-4B-Instruct-2507, against two similarly sized models, Llama-3.2-3B-Instruct) and Phi-3.5-mini-instruct. The results in the table above show all three models performed comparably on retrieval metrics, likely because they used the same embedding model and similarity search strategy. However, Qwen3-4B-Instruct-2507 consistently performed better on the generation metrics, achieving a higher score for faithfulness, answer relevance, and answer completeness on most of the benchmarks.

5. Usage and Intended Uses

The intended use case for this model is to assist board game players in understanding specific rules or providing a general overview of the game which can be used to start the game playing session.

Although the model was designed to be utilized with a document corpus, it does support the ability for a user to enter relevant context as text to be used when generating the response. In this scenario, the input text is not chunked but supplied directly to the generation model along with the query. This mode is automatically enabled if a BoardGameRag instance is constructed without children and parent arguments

Below is an example of how the RAG pipeline can be used from parsing to response generation:

from pathlib import Path
from parse_rulebooks import parse
from chunk_rulebooks import read_markdown, split_section, split_subsections, split chunks
from boardgame_rag import BoardGameRag

# Parse a rulebook PDF into Markdown
parsed_path = parse(file_name = "catan.pdf", out_name = "catan", ocr = False)

# Chunk the Markdown into parent sections and child chunks
markdown_text = read_markdown("catan.md", parsed_dir = Path("parsed"))
sections = split_sections(markdown_text, game = "Catan", source_file = "catan.md")

parent_docs = []
child_docs = []

for section in sections:
  subections = split.subsections(section)
  parent_docs.extend(subsections):
  for subsection in subsections:
    child_docs.extend(split_chunks(subsection))

# Initialize the RAG pipeline
rag = BoardGameRag(
        model_name = "Qwen/Qwen3-4B-Instruct-2507"),
        tokenizer_name = None,       # uses the same model as model_name
        children = child_docs,
        parents = parent_docs,
        max_new_tokens = 300,
        do_sample = False,
        persist_directory = None
)

# Ask a question
result = rag.generate_answer(
  question = "How many resources do I need to build a settlement?",
  prompt = None        # uses the default rules-QA prompt
)

print(result["answer"])

Below is an example of how to generate a response when there is no document corpus

# Context is stored as a list of strings
context = ["To build a settlement, a player must pay one brick, one wood, "
            "one wheat, and one sheep. A settlement may only be built on an "
            "unoccupied intersection that is not adjacent to another settlement "
            "or city."
]

# Initialize the RAG pipeline with no children or parents
rag = BoardGameRag(
        model_name = "Qwen/Qwen3-4B-Instruct-2507"),
        tokenizer_name = None,       # uses the same model as model_name
        max_new_tokens = 300,
        do_sample = False,
        persist_directory = None

result = rag.generate_from_context(
  question = "How many resources do I need to build a settlement?"),
  prompt = None,
  context = context
)

print(result)

6. Prompt Format

This model relies on a formatted user prompt instead of a system prompt. The prompt injects the retrieved RAG context directly into the user message so when generating an answer, the user only has to provide the text of the prompt If no prompt is provided during initialization, the system uses the default board game prompt.

You are a helpful assistant answering questions about how to play board games. 
Answer the user's question thoroughly using only the provided context. 
If the answer is not directly supported by the context you must say that you cannot answer the question with the information provided. 
If the answer differs across game modes, variants, or optional rules described in the context, summarize the differences and specify which version each applies to. 
Do not provide gameplay tips or strategies unless they are explicitly mentioned in the context.

CONTEXT
{context}

QUESTION
{question}

ANSWER:

If a prompt is provided during initialization, the instruction text changes but the context, question, and answer remains the same.

You are a helpful assistant answering questions using only the context provided.
If the answer is not directly supported by the context you must say that you cannot answer the question with the information provided.

CONTEXT
{context}

QUESTION
{question}

ANSWER:

7. Expected Output Format

The model is expected to generate a plain-English response to the user's question using only information retrieved from the knowledge base. If the requested information is unavailable or falls outside the scope of the knowledge base (e.g., strategy questions), the model should refuse to answer. The exact wording and formatting of responses may vary because the generation model was not further fine-tuned.

8. Limitations

The primary limitation of this RAG pipeline is that it is limited to the board games and rulebook editions included in its knowledge base and cannot answer questions about games or rules outside that corpus, including unofficial rulings, FAQs, or gameplay strategies. Additionally, while the chunking strategy was designed to leverage the hierarchical structure of board game rulebooks, not all rulebooks follow the same document structure, so the chunking strategy may not generalize well to other document collections. Because the system relies on retrieval, retrieval failures may result in incomplete responses or appropriate refusals to answer. Finally, because the generation model was not fine-tuned for this task, the wording and formatting of responses may vary.

9. Data Licensing

This project uses official board game rulebooks obtained from publisher websites. These documents remain the intellectual property of their respective publishers and are not released under this repository's license. Users are responsible for ensuring they have the appropriate rights to use or redistribute the rulebooks.