Reliability of LLM-assisted statistical model selection
Domain: statistics (statistics, model-selection, llm-evaluation)
Topic: Reliability of LLM-assisted statistical model selection under assumption violations
Venue: ARC-Bench Statistics 2026
Headline metric: primary_metric (maximize)

Research briefing:
Large language models are increasingly used to assist with statistical data
analysis, including model selection, test choice, and interpretation. However,
statistical model selection requires matching assumptions to data-generating
conditions, not merely producing plausible-sounding analysis text. An LLM or
LLM-like rule-based assistant may recommend inappropriate tests when data are
skewed, heteroskedastic, non-independent, or affected by outliers.

A credible study of this topic creates a controlled benchmark of small
statistical-analysis tasks with known ground truth. The system should compare
model-selection recommendations against an oracle or rule-based reference
across simulated datasets. The research question is: *can an LLM-assisted
statistical workflow choose appropriate statistical procedures under
assumption violations, and how often do its recommendations lead to invalid
inference?*

Hypotheses:
statement: A simple prompt-only LLM-style recommendation system is more likely to choose standard parametric tests even when assumptions are violated., statement: Adding computed diagnostic statistics, such as skewness, variance ratio, normality tests, and sample-size information, improves statistical procedure selection accuracy., statement: Incorrect procedure selection increases false positive rate or reduces coverage under assumption violations such as heteroskedasticity, skew, or dependence.

Experiment design:
research_question: Can LLM-assisted statistical model selection reliably choose appropriate procedures when assumptions are violated?
task_type: name: statistical_test_selection
description: Given dataset summaries and analysis goals, choose an appropriate statistical test or model.
conditions: name: text_only_recommender
description: Procedure selection using only natural-language task description and variable metadata., name: diagnostics_augmented_recommender
description: Procedure selection using natural-language description plus computed diagnostics., name: oracle_rule_based_selector
description: Reference selector using known data-generating process or explicit rules., name: always_parametric_baseline
description: Baseline that always chooses common parametric procedures such as t-test, ANOVA, or OLS.
statistical_tasks: name: two_sample_mean_comparison
candidate_methods: student_t_test, welch_t_test, mann_whitney_u, permutation_test, name: correlation_analysis
candidate_methods: pearson_correlation, spearman_correlation, robust_regression, name: linear_effect_estimation
candidate_methods: ordinary_least_squares, heteroskedasticity_robust_ols, rank_based_method, permutation_inference
data_generating_conditions: name: normal_equal_variance
description: Parametric assumptions approximately hold., name: normal_unequal_variance
description: Heteroskedastic two-sample setting., name: skewed_lognormal
description: Strongly skewed outcome distribution., name: outlier_contaminated
description: Small fraction of extreme observations., name: nonlinear_monotone_relationship
description: Correlation exists but is nonlinear or rank-based.
metrics: name: selection_accuracy
direction: maximize
description: Fraction of tasks where selected method matches oracle-acceptable method set., name: false_positive_rate
direction: target_nominal
description: Empirical Type I error under null simulations., name: coverage
direction: target_0.95
description: Empirical confidence interval coverage when interval estimates are produced., name: assumption_violation_detection_rate
direction: maximize
description: Fraction of cases where the system correctly identifies relevant assumption violations., name: explanation_grounding_score
direction: maximize
description: Whether the written explanation cites the correct diagnostic evidence rather than generic statistical advice.
implementation_note: If external LLM API access is unavailable, the submission may simulate an
LLM-assisted workflow using templated recommendations, local heuristics, or
stored model outputs. The benchmark should focus on statistical validity of
recommendations and downstream inference, not on proprietary model access.

simulation_settings: sample_sizes: 30, 100, 500
monte_carlo_repetitions: 500
seeds: 0, 1, 2, 3, 4
expected_artifacts: path: src/generate_tasks.py
description: Code for generating synthetic statistical-analysis tasks., path: src/evaluate_selectors.py
description: Code for evaluating selectors and downstream inference., path: results/metrics.json
description: Machine-readable selection accuracy, false positive rate, coverage, and diagnostic-detection metrics., path: results/selector_decisions.csv
description: Machine-readable selector decisions with task metadata, diagnostics, oracle-acceptable methods, and selected procedures., path: results/figures/
description: Plots or tables comparing selectors across task types, data-generating conditions, and assumption violations., path: report/paper.md
description: Paper-style report explaining task generation, selector designs, oracle rules, diagnostic evidence, results, limitations, and per-hypothesis conclusions., path: README.md
description: Setup and run instructions for reproducing the full benchmark from a clean checkout.
dependencies: python: numpy, pandas, scipy, scikit-learn, statsmodels, matplotlib
compute_requirements: gpu_required: no
estimated_wall_clock_sec: 300

Requirements:
type: artifact
description: A canonical results.json file exists at the workspace root with at least the keys: primary_metric (number), metric_key (string), metrics (object with numeric keys), hypotheses (object with h1/h2/h3 entries each carrying a `supported` boolean), summary (non-empty string)., type: artifact
description: A machine-readable metrics artifact (e.g. results/metrics.json) exists reporting selection accuracy and false-positive rate (or Type I error) broken down by selector and data-generating condition, plus a results/selector_decisions.csv with per-task selector decisions and oracle-acceptable method labels., type: numeric
description: results.json metrics MUST contain numeric (non-null, finite) selection accuracy values in [0, 1] for at least the text-only recommender and the diagnostics-augmented recommender, evaluated over more than one assumption-violation condition, so H1 and H2 can be evaluated., type: discussion
description: results.json hypotheses.h1/h2/h3 each MUST have an explicit `supported` boolean AND a `details` string ≥ 40 characters quoting the numerical evidence (specific accuracy / false-positive-rate / coverage values + their source artifact) used to reach the verdict., type: artifact
description: The submission implements or simulates all four selectors — text-only recommender, diagnostics-augmented recommender, oracle / rule-based reference selector, and an always-parametric baseline — and runs the selected procedures to record downstream inference (p-values, coverage, or Type I error)., type: discussion
description: The report clearly distinguishes the analysis goal, data-generating condition, oracle-acceptable method set, selected method, diagnostic evidence, and downstream inference metric. Nice-to-have, not blocking., type: discussion
description: The report states whether an external LLM API was used or whether a templated / heuristic / stored-output recommender stood in for it. Required for honest reporting but not for scientific correctness.

Scoring rubric:
requirements: A credible controlled simulation study evaluating whether LLM-assisted or LLM-like statistical model selection chooses appropriate procedures under assumption violations. The submission should generate statistical-analysis tasks with known reference decisions, compare text-only and diagnostics-augmented recommenders against an oracle or rule-based selector, evaluate downstream inference validity, and connect the numeric findings to the three hypotheses.
judging_note: Score on statistical validity, reproducible task generation, appropriate reference decisions, and honest discussion of downstream inference. Do not require access to an external LLM API; a templated, heuristic, local, or stored-output recommender can receive full credit if it supports the benchmark question. Penalize generic recommendation text that is not tied to diagnostics or data-generating conditions.
sub_tasks: requirements: The code implements a controlled benchmark for statistical procedure selection under several data-generating conditions and selector designs.
sub_tasks: requirements: The submission generates multiple statistical-analysis tasks, including two-sample mean comparison, correlation analysis, and linear effect estimation, with known null or alternative settings and fixed random seeds.
task_category: Code Development
finegrained_task_category: Dataset and Model Acquisition, requirements: The generated tasks cover meaningful assumption regimes, including approximately normal equal-variance data and at least three violation regimes such as heteroskedasticity, skewness, outliers, nonlinear monotone relationships, or dependence.
task_category: Code Development
finegrained_task_category: Experimental Setup, requirements: The submission implements or simulates the required selectors: a text-only recommender, a diagnostics-augmented recommender, an oracle or rule-based reference selector, and a simple parametric baseline.
task_category: Code Development
finegrained_task_category: Method Implementation, requirements: The diagnostics-augmented condition computes relevant diagnostic information, such as skewness, variance ratio, normality checks, outlier indicators, sample size, monotonicity, or heteroskedasticity evidence, and makes those diagnostics available to the selector.
task_category: Code Development
finegrained_task_category: Method Implementation, requirements: The code runs the selected statistical procedures or models and records downstream quantities such as p-values, confidence intervals, Type I error, coverage, or effect estimates where applicable.
task_category: Code Development
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Execution produces machine-readable selector and inference metrics adequate to evaluate statistical reliability.
sub_tasks: requirements: Execution produces a machine-readable metrics artifact, such as results/metrics.json, containing selection accuracy, false positive rate or Type I error, confidence interval coverage where applicable, assumption-violation detection rate, and explanation grounding by selector and data condition.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Reported metrics are aggregated over multiple simulation repetitions, seeds, task instances, or sample sizes, with enough repetition to make selector differences meaningful.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: The executed benchmark includes more than one task type and more than one assumption-violation condition, rather than evaluating a single isolated example.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Execution artifacts include selector decisions, oracle-acceptable method labels, diagnostic summaries, and at least one plot or table comparing selectors across assumption regimes.
task_category: Code Execution
finegrained_task_category: Logging, Analysis & Presentation, requirements: The final paper or report addresses the three model-selection reliability hypotheses with quantitative evidence and a clear statistical narrative.
sub_tasks: requirements: The report evaluates whether the text-only recommender over-selects standard parametric procedures under assumption violations and supports the conclusion with selector-decision metrics.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The report compares diagnostics-augmented and text-only selection and states whether computed diagnostics improve procedure selection accuracy or assumption-violation detection.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The report analyzes whether incorrect procedure selection leads to invalid inference, such as inflated false positive rates, poor coverage, or misleading effect estimates under specific violations.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The writeup clearly distinguishes the analysis goal, data-generating condition, oracle-acceptable method set, selected method, diagnostic evidence, and downstream inference metric.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The README or paper describes task generation, selector designs, diagnostic features, oracle rules, key numeric results, limitations, and per-hypothesis outcomes with appropriate caveats on simulation scope and any use or non-use of external LLM APIs.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation