Comparing split-conformal, Mondrian conformal, and CQR-style intervals on heteroscedastic regression
Domain: ml (machine-learning, uncertainty-quantification)
Topic: Comparing conformal prediction procedures (split-conformal, Mondrian, CQR) for achieving target coverage on heteroscedastic regression tasks
Venue: ARC-Bench 2026
Headline metric: coverage_gap (minimize)

Research briefing:
Conformal prediction provides finite-sample coverage guarantees with minimal
assumptions, making it attractive for uncertainty quantification in practical
regression. However, standard split-conformal intervals are often globally
calibrated and can be inefficient when noise is heteroscedastic: they may
over-cover low-noise regions and under-cover high-noise regions. This
motivates conditional variants such as Mondrian conformal (group-conditional
calibration) and CQR-style approaches that adapt interval width through
quantile modeling before conformal correction.

In small, CPU-only settings, one can still run a meaningful comparison by
using sklearn-compatible regressors and synthetic data where heteroscedastic
structure is controlled. A rigorous setup should evaluate not only marginal
coverage but also efficiency (mean interval width) and conditional behavior
(coverage gap across strata of predicted difficulty or input regions). These
diagnostics reveal when methods achieve nominal coverage by producing overly
wide intervals versus genuinely adapting to varying noise.

A credible benchmark should include at least one homoscedastic control and at
least one heteroscedastic dataset, then compare split-conformal, a Mondrian
variant (binning by an auxiliary score such as |x| or predicted scale), and a
CQR-style method based on lower/upper quantile regressors plus conformal
adjustment. Repeated random splits (multiple seeds) are needed because
conformal procedures can vary with calibration sample composition.

The key question is whether adaptive procedures can reduce conditional
miscoverage while maintaining near-target marginal coverage and reasonable
interval width under strict runtime constraints.

*Do Mondrian and CQR-style conformal procedures achieve lower conditional coverage error than vanilla split-conformal at comparable marginal coverage on heteroscedastic regression tasks?*

Hypotheses:
statement: On heteroscedastic datasets, at least one adaptive method (Mondrian or CQR-style) attains a lower absolute coverage gap to the 90% target than split-conformal by at least 0.02 on at least 2 of 3 datasets, averaged over ≥5 seeds., statement: At matched target 90% coverage, CQR-style conformal yields mean interval width no larger than split-conformal on at least 2 of 3 datasets (difference ≤ 0.00)., statement: Mondrian conformal reduces worst-bin conditional miscoverage (max over 4 bins of |bin_coverage-0.90|) by at least 0.03 versus split-conformal on at least 2 of 3 datasets.

Experiment design:
research_question: Do adaptive conformal procedures (Mondrian, CQR-style) improve coverage quality and efficiency over split-conformal for heteroscedastic regression under a fixed 90% target coverage?
conditions: name: split_conformal_rf
description: Vanilla split-conformal regression using RandomForestRegressor point predictor and absolute residual conformity scores on a held-out calibration split., name: mondrian_conformal_rf_4bins
description: Mondrian split-conformal with the same base predictor, calibration and test points partitioned into 4 bins by an auxiliary score (e.g., fitted value or |x0| proxy), with per-bin quantiles., name: cqr_gbr
description: CQR-style method using GradientBoostingRegressor quantile models (alpha=0.05, 0.95) and split-conformal correction on calibration residuals of quantile bands., name: naive_quantile_no_conformal
description: Non-conformal baseline using raw quantile regression interval [q0.05, q0.95] without conformal adjustment.
baselines: split_conformal_rf is the primary conformal baseline, naive_quantile_no_conformal is the non-conformal baseline
metrics: name: coverage_gap
direction: minimize
description: Absolute difference between empirical marginal coverage and target 0.90 on the test split, averaged over seeds., name: mean_interval_width
direction: minimize
description: Average prediction interval width on test samples, averaged over seeds., name: worst_bin_miscoverage
direction: minimize
description: Maximum across 4 bins of absolute deviation between bin coverage and 0.90, measuring conditional coverage disparity.
datasets: name: hetero_sine
source: synthetic: y = sin(2πx) + (0.1 + 0.5|x|)ε, x~Uniform(-1,1), n≈2000, name: hetero_friedman1
source: synthetic: sklearn.datasets.make_friedman1 with multiplicative noise scale depending on x0, name: diabetes
source: sklearn.datasets.load_diabetes (tabular regression benchmark)
compute_requirements: gpu_required: no
estimated_wall_clock_sec: 420

Requirements:

Scoring rubric:
requirements: A credible experiment comparing split-conformal, Mondrian conformal, CQR-style, and optionally naive-quantile prediction intervals for ~90% target coverage on heteroscedastic regression: methods are implemented, executed on multiple datasets with multiple seeds, and results address H1/H2/H3 directionally with coverage-gap-focused analysis.
judging_note: Score on scientific substance and directional correctness of evidence, not on exact numeric thresholds. Alternative conformal variants (e.g., jackknife+, locally adaptive) that test the same scientific question should be credited.
sub_tasks: requirements: The conformal and baseline interval-construction conditions are implemented correctly.
sub_tasks: requirements: The submission implements multiple distinct conditions — typically including split conformal, a Mondrian/group-adaptive variant, and a CQR-style or naive-quantile baseline — as separate code paths, not a single shared interval formula.
task_category: Code Development
finegrained_task_category: Method Implementation, requirements: Conformal methods use explicit train/calibration/test separation (or equivalent cross-fit logic) so calibration quantiles are computed on data not used to fit base regressors.
task_category: Code Development
finegrained_task_category: Experimental Setup, requirements: The submission uses multiple datasets (including at least one heteroscedastic synthetic dataset) and prepares regression targets correctly.
task_category: Code Development
finegrained_task_category: Dataset and Model Acquisition, requirements: Execution outputs interval-quality metrics for each condition.
sub_tasks: requirements: Execution produces a metrics artifact containing numeric coverage gap and mean interval width (or equivalents) for each implemented condition on at least one dataset.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Reported metrics are aggregated over multiple random seeds per (condition, dataset) cell with a dispersion estimate. Honest small-seed runs with variance reported are preferable to a single run.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Execution computes a conditional-coverage diagnostic (e.g., worst-bin miscoverage across several bins) for the conformal methods.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Quantitative analysis addresses H1/H2/H3 directionally and discusses trade-offs between coverage, conditional validity, and interval efficiency.
sub_tasks: requirements: The submission compares coverage gap of adaptive methods (Mondrian, CQR-style) against plain split conformal per dataset and conveys whether adaptive methods are meaningfully closer to the nominal coverage — judge directionally against H1.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The submission evaluates whether CQR-style intervals achieve mean interval width comparable to or narrower than plain split conformal on most datasets and conveys an H2 outcome.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The submission compares worst-bin miscoverage for Mondrian conformal vs plain split conformal and conveys whether Mondrian yields a meaningful improvement in conditional coverage (H3).
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The README or writeup describes methods and datasets, reports key metric values (coverage gap, interval width, worst-bin miscoverage), conveys per-hypothesis outcomes (supported / refuted / inconclusive), and notes limitations (finite seeds, binning choice, synthetic-to-real transfer). No strict word-count requirement.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation