DOI: 10.5281/zenodo.17837376
title: Leveraging LLMs for Enhanced Access and Retrieval of Governmental Information

description: Leveraging LLMs for Enhanced Access and Retrieval of Governmental Information

This is the third version of the dataset used in the study Leveraging LLMs for Enhanced Access and Retrieval of Governmental Information. It extends the previous Zenodo release by providing:



A single merged per‑question results file covering all retrieval experiments over the Spanish official gazette (Boletín Oficial del Estado, BOE).

A lightweight sample of the results for online preview.

The underlying question metadata for both validated questions and negative questions, including a pipeline that generates question–answer pairs and topics from raw BOE documents.

A rich set of aggregate metrics and diagnostics for safety, calibration, selective prediction and refusal quality.

Excludes figures and source code: All figures and source code (notebooks and scripts) reside in the BOEGPT GitHub repository and are not included in this Zenodo archive.


All CSV files are encoded in UTF‑8 and may contain both Spanish and English text (e.g., Spanish user questions and answers). This README is entirely in English.

The core experiments evaluate several retrieval‑augmented generation (RAG) configurations, covering embedding models, chunking strategies, re‑ranking and safety / refusal behaviours. For full pipeline details, see the BOEGPT GitHub repository.

Zenodo record (current): 17837376

 

Experimental configurations

All experiments follow a cumulative ablation-style design and share the same technical backbone:





A crawled corpus of the Spanish Official Gazette (BOE)




Passage segmentation into 5,000-character chunks




A dense FAISS index built over multilingual semantic embeddings(BGE-M3 family, based on M3-Embedding)https://github.com/facebookresearch/faisshttps://arxiv.org/abs/2402.03216




A parallel lexical BM25 index over the same passageshttps://www.staff.city.ac.uk/~sbrp622/papers/robertson09probabilistic.pdf




A Spanish-capable generator model with a fixed context window



Across all configurations, prompts, decoding parameters, and the maximum context budget are kept constant.As a result, observed differences can be attributed to retrieval and ranking choices, rather than to changes in models, chunking, or prompt length.

Configurations overview

On top of this shared backbone, we define six configurations (C0–C5) that progressively introduce retrieval components.C0 is a non-retrieval baseline, while C1–C5 are retrieval-augmented variants that differ only in how passages are selected and ranked before being concatenated into the context.

In the paper, we refer to these systems as C0–C5 for readability.In the BOEGPT repository and Zenodo artefacts, the same runs are labelled using the original experiment identifiers:





C0 ↔ Exp. 0




C1 ↔ Exp. 1




C2 ↔ Exp. 1.1




C3 ↔ Exp. 1.1.1




C4 ↔ Exp. 1.2




C5 ↔ Exp. 2



This mapping is one-to-one; the difference is purely nominal.

Summary of experimental configurations




Cfg.
Short name
Retrieval and ranking
Role / added modules




C0
No-RAG
No retrieval; the LLM answers using only the question and general BOE instructions
Lower bound; tests behaviour without external evidence


C1
Dense
Dense retrieval over FAISS; top-k passages by cosine similarity
Isolates dense retrieval (no query reformulation, no lexical reranking)


C2
Dense + QR
C1 applied to LLM-generated query reformulations; results merged and deduplicated
Query reformulation with frequency filtering under a fixed context budget


C3
Dense + QR + Ensemble
Combination of dense variants (direct similarity, MMR, thresholded)
Dense ensemble for coverage and diversity; no lexical reranking


C4
Hybrid
Dense retrieval followed by BM25 reranking with metadata boosting
Hybrid semantic + lexical ranking (replaces the QR stage)


C5
Hybrid + QR + Ensemble
Dense retrieval with QR and ensemble variants, followed by BM25 reranking
All modules combined under a fixed context budget (final system)




The design is cumulative: each configuration adds or replaces a small set of modules while keeping the corpus, segmentation, indexes, generator, and prompts fixed.

Execution flow

For all retrieval-augmented configurations (C1–C5), the query processor activates the modules corresponding to the current configuration and executes them in a fixed order:





Optional query reformulation




Dense retrieval (optionally using ensemble variants)




Optional BM25 lexical reranking with metadata boosting




Context construction under a fixed character budget



The selected passages are appended to the system prompt and user question and passed to the generator, which is always instructed to answer in Spanish and to refuse when the available context is insufficient.

Rationale

This setup ensures that comparisons across configurations reflect differences in retrieval and ranking, rather than changes in model capacity.We follow a cumulative ablation design instead of a full factorial exploration.

For this reason, we do not include an intermediate Hybrid + QR configuration between C4 and C5.The effect of query reformulation is already isolated in the transition from C1 to C2, and is evaluated again only in the fully combined system (C5).C4 is retained as a clean hybrid baseline to isolate the contribution of lexical signals and lightweight metadata cues over dense-only retrieval.

This design choice is consistent with prior findings for legal and regulatory collections, where exact identifiers, formal terminology, and sparse lexical cues often complement semantic similarity:





Borck et al., From dense to hybrid retrieval for legal IRhttps://arxiv.org/abs/2309.06140




Hindi et al., Enhancing retrieval for legal RAG systemshttps://arxiv.org/abs/2501.07872




Reuter et al., Reliable legal RAG through hybrid rankinghttps://arxiv.org/abs/2503.01801



Finally, configuration C0 quantifies how much of the final performance depends on explicit access to BOE content, serving as a non-retrieval lower bound.

 

Question topics and coverage

The validated BOE Q&A dataset (file boe_questions.csv, described below) contains 391 questions. In this version, 228 questions belong to the manually verified subset (human_checked=True and is_correct=True). The negative question set (boe_wrongqa_questions.csv) contains 59 questions used for hallucination and absence‑detection evaluation.




Topic (English)
# questions
What they cover




Labour / Social Security
76
Employment law, collective agreements, public employment offers, social security contributions and benefits.


Public Procurement
69
Calls for tenders, contract award notices, contract modifications and procurement procedures for public bodies.


Education / Universities
67
Regulations and calls related to schools and universities, scholarships, research projects and academic administration.


Environment, Water & Waste
50
Environmental regulations, water management, waste treatment, conservation and climate-related measures.


Public Administration
42
Organisation of public bodies, civil service rules, staffing and general administrative procedures.


Digitalization, Data & AI
22
Digital government, information systems, data protection, AI strategies and digital public services.


Justice
16
Judicial appointments, court procedures, legal aid and administration of justice.


Transport & Mobility
10
Transport infrastructure, public transport services and mobility-related regulation.


Financial & Banking
9
Financial supervision, banking regulations, financial instruments and credit institutions.


Energy
7
Energy markets, electricity and gas regulation, renewables and energy efficiency policies.


Culture & Sports
5
Cultural programmes, heritage protection, arts funding and sports regulation or subsidies.


Taxation
5
Tax rules, procedures, deadlines and fiscal incentives.


Interior & Security
4
Public safety, civil protection, policing and internal security.


Economy, Business & Consumer
4
Economic policy, business support schemes, competition and consumer protection.


Foreign Affairs & EU
2
International agreements, EU-related measures and foreign policy instruments.


Other / Miscellaneous
1
Cross-cutting or residual issues not fitting other categories.


Agriculture, Livestock & Fisheries
1
Agricultural policy, rural development, livestock and fisheries regulation.


Defense
1
Military regulations, defence organisation and personnel matters.




 

File list

1. Primary per‑question results

1.1 Results - LLMs for Access and Retrieval of Governmental Information - all.csv

The main merged results file (packaged under a stable Zenodo‑friendly filename). Each row corresponds to a single model answer to a single BOE question under a specific experiment configuration.

It contains a full suite of RAG evaluation metrics, question metadata and derived safety labels. Key columns include:



Experiment metadata: timestamp, experiment, experiment_label, is_rag

Question identifiers: question_id, question_dataset

RAG evaluation metrics: context_recall, llm_context_precision_without_reference, faithfulness, answer_correctness, factual_correctness(mode=f1), answer_relevancy, context_length, quality_score

Text fields: user_input, response, reference, retrieved_contexts

Negative-case behaviour: detection_type, negative_case_behavior, negative_case_is_safe, is_explicit_refusal


1.2 Sample - Results - LLMs for Access and Retrieval of Governmental Information - all.csv

A lightweight sample of the master results file (first 36 rows). The retrieved_contexts field (when present) is whitespace‑normalised and truncated (≈2000 characters) so that the file can be previewed comfortably on Zenodo.

 

2. Question metadata

2.1 boe_questions.csv

Validated BOE Q&A dataset used for the main evaluation. This file provides the ground‑truth question text, topic labels, document identifiers and manual validation flags (human_checked, is_correct) for the verified subset.

Typical fields include: question_id, question_es, topic_es / topic_en, doc_id, publication_date, and validation metadata.

2.2 boe_wrongqa_questions.csv

Negative questions designed to test refusal behaviour and robustness against hallucinations / absence‑of‑context. These follow the same topic taxonomy where applicable.

 

3. Aggregate tables – validated questions

3.1 valid_summary_by_experiment_and_validation_type.csv

Aggregated metrics (means and counts) by experiment and validation type for the validated questions.

3.2 valid_summary_safety_and_accuracy_rates.csv

High‑level safety and accuracy rates computed from the validated dataset.

3.3 retrieval_metrics_by_experiment.csv

Retrieval‑specific metrics (e.g., retrieval quality) summarised by experiment.

3.4 retrieval_metrics_by_experiment_and_regime.csv

Retrieval metrics broken out by experiment and retrieval regime.

3.5 retrieval_metrics_table.tex

LaTeX table summarising retrieval metrics across experiments and regimes.

3.6 high_correctness_with_ci_by_experiment_and_validation_type.csv

High‑correctness rates with confidence intervals, by experiment and validation type.

 

4. Aggregate tables – normal validated questions

4.1 valid_normal_summary_by_experiment_and_validation_type.csv

Same as section 3.1, restricted to the “normal” validated subset (excluding negative‑case questions).

4.2 valid_normal_summary_safety_and_accuracy_rates.csv

Safety and accuracy rates restricted to the normal validated subset.

 

5. Aggregate tables – negative‑case validated questions

5.1 negative_summary_by_experiment_and_validation_type.csv

Summary metrics restricted to the validated negative‑case subset.

5.2 negative_summary_safety_and_accuracy_rates.csv

Safety and accuracy rates restricted to the validated negative‑case subset.

5.3 negative_case_studies.csv

A curated set of representative negative examples (hallucination and absence regimes) with model outputs and analysis labels.

5.4 best_practice_negative_case_studies.csv

Curated negative cases for the best‑practice subset of experiments (if present in the repository).

5.5 best_practice_llm_negative_summary_by_experiment.csv

Summary metrics for negative‑case questions within the best-practice LLM experiments.

5.6 human_negative_validation_summary.txt

Narrative summary of human validations for negative-case questions.

5.7 human_negative_validation_table.tex

LaTeX-formatted table of human validation outcomes for negative cases.

5.8 human_vs_llm_negative_summary.csv

Comparison of human evaluation outcomes versus LLM predictions for negative-case questions.

 

6. Aggregate tables – Wrong‑QA questions

6.1 wrongqa_summary_by_experiment_and_validation_type.csv

Summary metrics for the Wrong‑QA dataset.

6.2 wrongqa_summary_safety_and_accuracy_rates.csv

Safety and accuracy rates for Wrong‑QA.

6.3 wrongqa_negative_summary_by_experiment_and_validation_type.csv

Summary metrics for negative cases within Wrong‑QA.

6.4 wrongqa_negative_summary_safety_and_accuracy_rates.csv

Safety and accuracy rates for negative cases within Wrong‑QA.

6.5 best_practice_llm_wrongqa_confusion_matrix.csv

Confusion matrix summarising best-practice LLM performance on Wrong-QA predictions.

 

7. Selective prediction diagnostics

7.1 selective_prediction_auc.csv

Selective prediction AUC values across experiments.

7.2 best_practice_selective_prediction_auc.csv

Selective prediction AUC values for the best‑practice subset.

7.3 metric_correlations_verified_unverified.csv

Correlations comparing metrics computed on verified vs unverified subsets.

7.4 answer_correctness_bootstrap_ci_verified.csv

Bootstrap summary of Answer Correctness on the manually verified subset (n=228), reported per experiment with mean_answer_correctness and 95% CI (ci_low, ci_high).

7.5 answer_correctness_pairwise_diffs_verified.csv

Pairwise bootstrap comparison of mean Answer Correctness differences on the verified subset (n=228), with mean_diff, 95% CI, and a significant_at_95 flag for each (baseline, improved) experiment pair.

 

8. Refusal quality metrics

8.1 refusal_quality_metrics.csv

Refusal‑quality diagnostics across experiments.

8.2 best_practice_refusal_quality.csv

Refusal‑quality diagnostics for the best‑practice subset.

 

9. Error categorisation

9.1 error_annotation_template_demo.csv

This CSV serves as a worked example for annotating BOE QA errors.

9.2 error_annotation_template.csv

This is the blank template for annotating errors. It has the same columns as the demo file but the error_type column is empty.

9.3 error_category_counts.csv

Provides aggregated statistics of error annotations.

9.4 error_category_examples.csv

Contains detailed examples of each error category for qualitative analysis.

9.5 error_category_table.tex

A LaTeX table summarising the error counts and percentages. It accompanies the CSV for reporting purposes.

 

10. Notebooks and reproducing the pipeline

The notebooks and scripts used to generate this dataset are not included in this Zenodo package. They remain available in the BOEGPT GitHub repository, ensuring that you can inspect and rerun the full pipeline without duplicating code here. For reference, the notebooks are:



boe_crawler/boe_crawler_to_jsonl.ipynb – Crawls BOE bulletins and converts them to JSONL for downstream processing.

boe_qa_pipeline/boe_qa_pipeline.ipynb – Generates Q&A pairs and metadata from BOE documents (dataset creation).

boe_vectordb/boe_faiss_builder.ipynb – Builds FAISS vector indices for BOE document retrieval.

boe_experiments/boe_rag_experiments.ipynb – Runs the retrieval‑augmented generation experiments and evaluations.

boe_analysis/boe_rag_analysis.ipynb – Aggregates experimental results and produces article figures/tables.

boe_analysis/error_taxonomy_study.ipynb – Analyses model errors and categorises error types.

boe_analysis/engineering_summary.ipynb – Examines latency and token usage trade‑offs across configurations.


 

11. Figures included in this archive

No figures are included in this Zenodo package. All figures referenced in the LaTeX manuscript are available in the BOEGPT GitHub repository and are intentionally omitted from this archive.

 

Usage

These files enable reproducible evaluation of retrieval‑augmented generation systems on a non‑English governmental corpus. Researchers can:



Inspect per‑question results in Results - LLMs for Access and Retrieval of Governmental Information - all.csv to analyse individual failures and successes across different experiments.

Compare validated vs negative subsets using boe_questions.csv and boe_wrongqa_questions.csv; human_checked and is_correct flags indicate manual review.

Use the summary tables to compare experiments, compute safety/accuracy rates and assess high‑correctness with statistical confidence.

Explore calibration and selective prediction diagnostics to understand abstention trade‑offs and uncertainty quality.


 

Column reference for the master results file

The file Results - LLMs for Access and Retrieval of Governmental Information - all.csv is the main merged per‑question results file. Columns are grouped below for clarity.

Experiment metadata




Column
Description




timestamp
Date and time when the experiment row was generated.


experiment
Canonical identifier of the experiment configuration (e.g., 0, 1, 1.1, 1.1.1, 1.2, 2).


experiment_label
Human-readable description of the experiment configuration.


is_rag
Boolean indicating whether retrieval‑augmented generation was used (False for the LLM-only baseline).




Question identifiers and metadata




Column
Description




question_id
Numeric identifier linking the row to the corresponding entry in boe_questions.csv or boe_wrongqa_questions.csv.


question_dataset
Source dataset label: valid (validated set) vs wrongqa (negative set).


question_es
Canonical Spanish wording of the question.


topic_es
High‑level topic label in Spanish.


doc_id
BOE document identifier related to the question (e.g., BOEA2024xxxxx).


publication_date
Publication date of the BOE document (YYYYMMDD).


human_checked
Whether the question/answer pair was manually reviewed.


is_correct
Manual correctness flag for reviewed items (when applicable).


expected_response_type
Ground‑truth expectation (e.g., verified_answer, unverified_answer, refusal, cannot_answer_from_context).


validation_category
High‑level regime label (correctness_validation / hallucination_detection / absence_detection).


validation_type
Human‑readable regime label used in summary tables.


validation_label
Compact label used in plots/tables.




RAG evaluation metrics




Column
RAGAS
Description




context_recall
Yes
Fraction of reference answer statements supported by retrieved contexts (recall‑style).


llm_context_precision_without_reference
Yes
Proportion of retrieved context judged relevant to the question (precision‑style; ignoring reference answer).


faithfulness
Yes
Degree to which the model answer is supported by the provided contexts.


answer_correctness
Yes
Correctness of the model answer against the reference answer.


factual_correctness(mode=f1)
Yes
Fact‑level F1 correctness score (LLM‑judged).


answer_relevancy
Yes
Relevancy of the answer to the question.


context_length
 
Length of the retrieved context (pipeline‑dependent: chars/tokens).


quality_score
 
Composite score combining selected metrics (implementation‑specific).




Text fields




Column
Description




user_input
Full prompt shown to the model (system instructions + Spanish user question).


response
Raw model answer returned by the LLM.


reference
Human‑written reference answer used for evaluation.


retrieved_contexts
Concatenated retrieved context snippets fed to the model (empty for LLM-only baseline).




Derived analysis labels and negative‑case behaviour




Column
Description




detection_type
Ground‑truth negative‑case type (hallucination_detection / absence_detection / blank for normal questions).


inferred_detection_type
Inferred detection regime used internally by the analysis pipeline.


is_explicit_refusal
True if the model explicitly refused to answer.


is_strict_i_dont_know
True for strict abstentions with no speculative content.


negative_case_behavior
Behaviour label for negative questions (e.g., appropriate_refusal, hallucination_or_invented).


negative_case_is_safe
Boolean summary: True for safe behaviour (refusal), False for unsafe behaviour (hallucination).