DOI: 10.5281/zenodo.17297918
title: Attribution Bias in Literary Style Evaluation: Comparing Human and AI Perceptions of Authorship

description: This dataset contains the complete experimental data for a large-scale study investigating attribution bias in literary style evaluation across two complementary experimental designs. Study 1 compares how human evaluators (N=556) and AI language models (N=13) assess identical literary content (the original Queneau vs. GPT-4-generated versions of his exercises) differently based on perceived authorship (human-authored vs. AI-generated). Study 2 expands the content generation across 14 models, then examining bias patterns when evaluators judge material from every AI creator in the experimental matrix.

Both studies employed a three-condition experimental design using Raymond Queneau's "Exercises in Style" as literary stimuli across 30 distinct writing styles. Participants and AI models evaluated the same content under blind, open-label, and counterfactual conditions, revealing systematic attribution bias where identical text receives different quality assessments based solely on authorship labels.

Dataset Contents

This repository includes raw experimental data, quality-filtered datasets, AI model simulation logs, and processed analysis-ready files spanning the complete research pipeline for both experimental studies. Human participant response data can be found here in de-identified form (N=2,780 evaluations from 556 participants), with personally identifiable information removed in accordance with Princeton University IRB protocol #18320.

The compressed data folder (data.zip) should be extracted into this cloned GitHub repository (drop the unzipped folder at the same level of the analysis folder) to enable full replication of analyses through the provided Jupyter notebooks: 

style_and_prejudice/├── analysis/│   ├── 01_data_quality_analysis.ipynb│   ├── 02_human_bias_analysis.ipynb│   └── 03_run_ai_simulation.ipynb            # Study 1: single creator, 13 AI evaluators│   └── 03b_run_ai_simulation_expanded.ipynb  # Study 2: cross-model design, 14x14 creators│   └── 04_ai_bias_analysis.ipynb             # Study 1: analysis│   └── 04b_ai_bias_analysis_extended.ipynb   # Study 2: analysis│   └── 05_comparative_analysis.ipynb         # Study 1: human vs. AI evaluator bias├── data/  # ← COPY HERE!│   ├── literary_materials/│   ├── logs/│   └── responses/└── README.md

Folder Descriptions



responses/ --> core response data organized by evaluator type and processing stage:


==> input for notebook #1 (01_data_quality_analysis.ipynb) 







human_evaluators/raw/ → original participant data (fully anonymized!) collected through web platform



questionnaire.csv → demographics, attention checks, and screening responses

responses.csv → all participant evaluations across experimental conditions






-------------------------------------------------------------------------------------------------------

==> generated by notebook #1 (01_data_quality_analysis.ipynb), used by notebook #2 (02_human_bias_analysis.ipynb)







human_evaluators/processed/ → quality-filtered human participant datasets



valid_participants_dataset.csv → participants meeting inclusion criteria (N=556)

excluded_participants_dataset.csv → excluded participants with reasons






-------------------------------------------------------------------------------------------------------

==> generated by notebook #3 and #3b (03_run_ai_simulation.ipynb, 03b_run_ai_simulation_expanded.ipynb), used by notebook #4 and #4b (04_ai_bias_analysis.ipynb, 04b_ai_bias_analysis_extended.ipynb)







ai_evaluators/raw/ → AI model simulation data



Study 1 (single AI-generator): ai_participant_simulation_20250805_113946.csv → complete AI evaluations across all 13 models (13 models × 30 styles × 3 conditions, 3 separate runs)

Study 1 robustness replications:



ai_participant_simulation_20250915_210237-awardwinningai-t07.csv → alternative labeling ("award-winning AI")

ai_participant_simulation_20250916_195700-symmetricallanguage-t07.csv → symmetric labeling (t=0.7)

ai_participant_simulation_20250917_211949-symmetricallanguage-t00.csv → symmetric labeling, deterministic (t=0)



Study 2 (cross-model design): ai_participant_simulation_expanded.csv → cross-model evaluations (14 evaluators × 14 creators × 30 styles × 3 conditions, 1 run)






-------------------------------------------------------------------------------------------------------

==> generated by notebooks #2 (02_human_bias_analysis.ipynb) & #4 (04_ai_bias_analysis.ipynb), used by notebook #5 (05_comparative_analysis.ipynb)







processed_for_comparison/ → final analysis-ready datasets for cross-evaluator comparison



human_responses_processed.csv → processed human evaluation data

human_summary_for_comparison.csv → Human bias summary statistics

ai_responses_processed.csv → AI evaluation data formatted for comparison

ai_summary_for_comparison.csv → AI bias summary statistics

human_experiment_stats.json & ai_condition_stats.json → experimental condition metadata






-------------------------------------------------------------------------------------------------------



logs/ --> processing logs of AI evaluator simulations:



ai_evaluators/ → API execution logs from AI simulation

 Study 1:



ai_participant_simulation_20250805_113946.log

ai_participant_simulation_20250915_210237-awardwinningai-t07.log

ai_participant_simulation_20250916_195700-symmetricallanguage-t07.log

ai_participant_simulation_20250917_211949-symmetricallanguage-t00.log



Study 2:



ai_participant_simulation_expanded.log






-------------------------------------------------------------------------------------------------------



literary_materials/



Study 1: experimental_stimuli.xlsx → GPT-4 generated stories (30 styles)

Study 2: experimental_stimuli_expanded.xlsx → multi-model generated stories (14 AI creators × 30 styles)




The experimental stimuli files contain structured literary content across 30 style categories, enabling full replication pipeline. The datasets include AI-generated stories from multiple contemporary language models, providing researchers flexibility to experiment with alternative literary materials. For complete replication of the original evaluation study, the placeholder literary content should be replaced with corresponding excerpts from Raymond Queneau's Exercises in Style (2012 New Directions edition, see here). The dataset enables complete reproduction of findings while respecting copyright restrictions on literary materials through placeholder text that demonstrates experimental methodology without redistributing protected content.

-------------------------------------------------------------------------------------------------------Individual files are also provided separately for direct access to specific datasets.

Funding and Support

This work was supported by the Princeton Language and Intelligence (PLI) Seed Grant Program. Research conducted by Wouter Haverals and Meredith Martin, at Princeton University's Center for Digital Humanities.