Bootstrap confidence intervals under non-Gaussian data
Domain: statistics (statistics, resampling-methods, simulation-study)
Topic: Bootstrap confidence interval coverage under light-tailed, skewed, and heavy-tailed data
Venue: ARC-Bench Statistics 2026
Headline metric: primary_metric (maximize)

Research briefing:
Bootstrap confidence intervals are widely used when analytic standard errors
are inconvenient, but their finite-sample coverage can change substantially
with skewness, heavy tails, contamination, and the choice of statistic. A
credible study of this topic should simulate data-generating processes with
known estimands, compare several bootstrap confidence interval methods, and
evaluate empirical coverage and interval width.

The study should not only report whether nominal 95% intervals contain the
truth, but also explain when wider intervals, robust estimators, or
studentized intervals improve reliability. The research question is: *how do
bootstrap interval choices and estimator choices affect empirical coverage
under light-tailed, skewed, and heavy-tailed data?*

Hypotheses:
statement: Percentile bootstrap intervals for the sample mean achieve near-nominal coverage under light-tailed Gaussian data, especially at moderate sample sizes., statement: Percentile bootstrap intervals for the sample mean under-cover under heavy-tailed or contaminated data., statement: Robust location estimators, such as the median or trimmed mean, improve coverage stability or interval behavior under heavy-tailed or contaminated data.

Experiment design:
research_question: How do bootstrap confidence interval methods behave across light-tailed, skewed, and heavy-tailed distributions?
estimands: name: population_mean
description: The true mean of each simulated data-generating process when it exists., name: robust_location
description: A robust target such as the median or trimmed mean, used when heavy tails or contamination make mean-based inference unstable.
conditions: name: normal_theory_mean_ci
description: Classical normal or t-based interval for the sample mean., name: percentile_bootstrap_mean
description: Percentile bootstrap interval for the sample mean., name: studentized_bootstrap_mean
description: Studentized or bootstrap-t interval for the sample mean, or a justified bootstrap alternative., name: robust_estimator_bootstrap
description: Bootstrap interval for a robust estimator such as the median or trimmed mean.
data_generating_conditions: name: gaussian
description: Light-tailed data where standard mean inference should work well., name: lognormal_skewed
description: Skewed data where percentile intervals may show asymmetric finite-sample behavior., name: student_t_heavy_tailed
description: Heavy-tailed data with unstable sample means., name: contaminated_normal
description: Mostly normal data with a small fraction of extreme outliers.
metrics: name: coverage
direction: target_0.95
description: Empirical fraction of intervals containing the true estimand., name: coverage_error
direction: minimize_abs
description: Absolute deviation from nominal 95% coverage., name: interval_width
direction: minimize_given_coverage
description: Average interval width, interpreted jointly with coverage., name: failure_rate
direction: minimize
description: Fraction of simulation runs where an interval could not be computed or produced invalid values.
simulation_settings: sample_sizes: 30, 100, 500
monte_carlo_repetitions: 500
bootstrap_resamples: 1000
seeds: 0, 1, 2, 3, 4
expected_artifacts: path: src/experiment.py
description: Executable simulation code implementing confidence interval methods and data-generating processes., path: results/metrics.json
description: Machine-readable coverage, coverage error, interval width, and failure rate by method, distribution, and sample size., path: results/figures/
description: Diagnostic plots comparing coverage and interval width across methods and data-generating conditions., path: report/paper.md
description: Paper-style report explaining the setup, methods, results, limitations, and per-hypothesis conclusions.
dependencies: python: numpy, pandas, scipy, matplotlib
compute_requirements: gpu_required: no
estimated_wall_clock_sec: 300

Requirements:
type: artifact
description: A canonical results.json file exists at the workspace root with at least the keys: primary_metric (number), metric_key (string), metrics (object with numeric keys), hypotheses (object with h1/h2/h3 entries each carrying a `supported` boolean), summary (non-empty string)., type: artifact
description: A machine-readable metrics artifact (e.g. results/metrics.json) exists reporting empirical coverage AND interval width broken down by CI method, data-generating distribution, and sample size., type: numeric
description: results.json metrics MUST contain at least 3 numeric (non-null, finite) empirical coverage values in [0, 1] for distinct (method, distribution) cells, including at least one Gaussian cell and one heavy-tailed or contaminated cell so H1 and H2 can be evaluated., type: discussion
description: results.json hypotheses.h1/h2/h3 each MUST have an explicit `supported` boolean AND a `details` string ≥ 40 characters quoting the numerical coverage / interval-width evidence (specific values + their source artifact) used to reach the verdict., type: artifact
description: At least one figure file (PNG or PDF, ≥150 DPI for raster) exists under results/figures/ comparing coverage and/or interval width across methods and data-generating conditions, with axes labeled and a legend., type: discussion
description: The report discusses the coverage-versus-interval-width tradeoff — recognising that higher coverage may simply come from wider intervals and that width must be read jointly with coverage. Nice-to-have, not blocking., type: discussion
description: results.json or a sibling reproducibility section names the Monte Carlo repetition count, bootstrap resample count, and at least one explicit random seed. Required for full reproducibility but not for scientific correctness.

Scoring rubric:
requirements: A credible simulation study evaluating whether bootstrap confidence interval choice affects empirical coverage under light-tailed, skewed, and heavy-tailed data. The submission should implement multiple CI methods, simulate several data-generating distributions and sample sizes, report coverage and interval width over repeated trials, and connect the numeric findings to the three hypotheses.
judging_note: Score on statistical substance, correct simulation design, and directional correctness of evidence. Do not require exact repetition counts if the submission is computationally reasonable. Partial but well-motivated simulations deserve partial credit; rigid naming differences should not penalize a substantively correct experiment.
sub_tasks: requirements: The bootstrap simulation code implements meaningful confidence interval comparisons across relevant distributional regimes.
sub_tasks: requirements: The submission implements at least two bootstrap confidence interval methods, typically percentile bootstrap and studentized/bootstrap-t or another justified alternative. Methods should be implemented as distinct code paths rather than cosmetic options.
task_category: Code Development
finegrained_task_category: Method Implementation, requirements: The submission includes a robust location estimator, such as median or trimmed mean, and compares it against the ordinary sample mean under heavy-tailed or contaminated data.
task_category: Code Development
finegrained_task_category: Method Implementation, requirements: The submission simulates multiple data-generating processes covering at least light-tailed and heavy-tailed cases, with skewed data such as lognormal or contaminated data receiving additional credit.
task_category: Code Development
finegrained_task_category: Dataset and Model Acquisition, requirements: The simulation evaluates more than one sample size so that finite-sample behavior can be distinguished from asymptotic behavior.
task_category: Code Development
finegrained_task_category: Experimental Setup, requirements: Execution produces coverage and interval-quality metrics adequate to evaluate bootstrap CI behavior.
sub_tasks: requirements: Execution produces a machine-readable metrics artifact, such as results/metrics.json, containing numeric empirical coverage and interval width by condition, distribution, and sample size. Coverage error or failure rate may supplement these metrics.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Reported metrics are based on repeated Monte Carlo trials and bootstrap resampling. The exact number of repetitions need not match the topic file, but it should be large enough to make coverage comparisons meaningful and should be honestly reported.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: The simulation correctly defines the true estimand for each data-generating process, such as the true mean or true robust location target, and checks whether confidence intervals contain that estimand.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: The final paper or report addresses the three bootstrap hypotheses with quantitative evidence and a clear statistical narrative.
sub_tasks: requirements: The submission evaluates whether percentile bootstrap confidence intervals achieve near-nominal coverage under light-tailed Gaussian data, especially at moderate or larger sample sizes, and states whether H1 is supported, refuted, or inconclusive.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The submission evaluates whether percentile bootstrap intervals for the sample mean undercover under heavy-tailed data and supports the conclusion with empirical coverage numbers.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The submission compares robust location estimators against the ordinary sample mean under heavy-tailed or contaminated data and discusses whether robustness improves coverage stability or interval behavior.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The analysis discusses interval-width tradeoffs, recognizing that higher coverage may come from wider intervals and that width should be interpreted jointly with coverage.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The README or writeup describes the simulation setup, CI methods, data-generating processes, key numeric results, and per-hypothesis outcomes with appropriate caveats on repetition count, bootstrap count, and Monte Carlo uncertainty.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation