DOI: 10.5281/zenodo.18772666
title: Semantic Search Engine and Vernacularity Predictor for the Southern Macro-Region of Brazil AHU Catalog

description: Overview

This dataset and accompanying web application provide a machine-readable, semantically annotated corpus of catalog summaries from the Arquivo Histórico Ultramarino (AHU). Focused on the "Macrorregião Sul do Brasil" (historically referred to as the "Mar do Sul" region) during the late colonial period (1737–1828), the pipeline utilizes Large Language Models and dense vector embeddings to quantify sociolinguistic variables. It is designed to support research in Diachronic Linguistics, specifically aimed at mapping the competition of grammars and isolating the probability of "Vernacular Leakage" (oral/Brazilian Portuguese syntax) in historical administrative documentation.

Files Included in this Dataset

The dataset and application are distributed as a repository containing the following core files:



mar_do_sul.parquet: The core structured resource (Compressed Relational Database). It connects the raw archival text to geographical metadata, temporal data, and algorithmic sociolinguistic annotations.



Core Columns:



document_id_and_date: Unique archival reference code and temporal anchor.

text_content: Full text description from the AHU catalog.

vernacular_score: A quantitative probability integer (0.0 to 1.0) indicating the likelihood of localized syntax.

scribe_mediation: Boolean flag indicating top-down or bureaucratic mediation.

communication_vector: Directionality of the document (e.g., Top-Down, Bottom-Up).

typology: Normalized singular root of the document type (e.g., CARTA, REQUERIMENTO).





mar_do_sul_embeddings.pt: PyTorch Tensor File containing the 768-dimensional dense vector embeddings of the archival descriptions. Generated via the paraphrase-multilingual-mpnet-base-v2 model to enable cross-lingual and thematic semantic search without requiring real-time API inference.

app.py: The Streamlit Application script (GUI). It handles the dynamic semantic search (Cosine Similarity thresholding), sociolinguistic filtering ("Lenses"), and programmatic PDF dossier generation.

requirements.txt: List of Python environment dependencies required to execute the pipeline and web application (e.g., pandas, streamlit, sentence-transformers, reportlab).


Scope Included



Temporal Boundary: 1737 to 1828

Geographical Focus: Southern Macro-Region of Brazil (Macrorregião Sul do Brasil).

Data Source: Summaries and metadata extracted from the Arquivo Histórico Ultramarino (AHU) catalogs.


Source of Archival Data

The original unstructured textual summaries used to build this corpus were obtained from the public cataloging efforts of the Arquivo Histórico Ultramarino and the Projeto Resgate. We gratefully acknowledge their continuous work in preserving and indexing the colonial documentary heritage.

Methodology



Extraction: RegEx parsing of raw AHU catalog text to isolate document IDs, dates, and typologies.

Annotation: Zero-shot LLM inference (DeepSeek) to categorize the sender/recipient, vector of communication, and compute the Probability of Vernacularity Score (SPV).

Vectorization: Sentences embedded using HuggingFace's paraphrase-multilingual-mpnet-base-v2.

Curation: Custom Portuguese NLP stemming and fuzzy matching (Levenshtein distance) for typology normalization and strict schema validation.


Methodological Criteria

For a detailed description of the computational pipeline, LLM prompting parameters, the "Forgiving Auditor" extraction logic, and the UI/UX deployment rules (including Semantic Cutoff/Cosine Similarity thresholds), please refer to the forthcoming PhD thesis associated with this repository.

Citation

Currently, this dataset and application are published as an independent computational resource. If you use this corpus, methodology, or code, please cite this repository directly using its Zenodo DOI:

Rocha, Saulo Rogério Pacheco. (2026). AHU Catalog Document Classifier for the Southern Macro-Region of Brazil (Version 1.0) [Data set and Software]. Zenodo. https://doi.org/10.5281/zenodo.18772667

License

This dataset and the associated code are made available under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) license.