DOI: 10.5281/zenodo.17559489
title: RAG-Coder: A Framework for Augmenting Qualitative Analysis in Empirical Software Engineering - Evaluation Dataset

description: 📘 Description

This repository contains the dataset used for the empirical evaluation of the RAG-Coder framework, as described in the paper:



“RAG-Coder: A Framework for Augmenting Qualitative Analysis in Empirical Software Engineering” 2025.


The dataset includes anonymized qualitative survey data and derived analysis files used to assess the reliability, performance, and interpretability of the RAG-Coder approach for coding open-ended responses.

Due to company privacy and compliance requirements, this dataset cannot be publicly distributed.It can, however, be shared with authorized researchers upon reasonable request and with explicit authorization from the partner company.

Requests should be directed to the corresponding author.



📁 Repository Structure

/
├── raw_survey_data.csv              # Original anonymized survey responses (restricted)
├── study1.csv                       # Manually Coded Dataset from Study 1 
├── study2.csv                       # Manually Coded Dataset from Study 2 
├── dataset_varinha_consenso.csv     # Human consensus dataset used for model evaluation
├── adjudication_varinha.csv         # Initial adjudication results between coders
├── adjudication_varinha_erros.csv   # Annotated disagreements subset 
├── rq1_metrics.json                 # Agreement metrics (RQ1)
├── rq2_results-ragcoder.csv         # Framework results (RQ2)
├── rq3_adjudication.csv             # Adjudicated disagreements (RQ3)
├── RAG-coding/                      # RAG-Coder outputs for 775 responses
│   ├── s1/ ... s4/                  # Segment folders containing RAG-generated coding 
└── codebook.csv                     # Code definitions used for analysis




🔍 Contents Summary




File/Folder
Description




raw_survey_data.csv
Raw, anonymized survey responses from practitioners.


study1.csv
Dataset from the first study, now used by the RAG-Coder.


study2.csv
Dataset from the second study, now used by the RAG-Coder.


dataset_varinha_consenso.csv
Consensus codes from human analysts (gold standard).


adjudication_varinha.csv
Adjudication between first and second human coders.


adjudication_varinha_erros.csv
Subset of disagreement cases for detailed examination.


rq1_metrics.json
Cohen’s κ and other inter-coder agreement metrics.


rq2_results-ragcoder.csv
Output codes assigned by RAG-Coder for RQ2 validation.


rq3_adjudication.csv
Adjudicated disagreements between model and human coders (RQ3).


RAG-coding/
RAG-Coder generated code assignments for all 775 responses.


s1–s4/
Subsets representing RAG-Coder batch runs.


codebook.csv
Formal definitions of all qualitative codes.






🔒 Data Availability

The dataset is restricted to protect participant privacy and company confidentiality.Access can be granted upon reasonable request and with company authorization.

Interested researchers may contact the corresponding author to obtain access under a confidentiality agreement.



⚙️ Related Resources



🧠 RAG-Coder Framework (Open-Source): https://zenodo.org/records/17494982




📬 Contact

Corresponding Author:Dr Lidiany CerqueiraEmail: lidianycs@ufs.brORCID: 0000-0002-4989-0986



📄 This dataset is provided exclusively for peer-review and research validation purposes under restricted access conditions.