PC vs GES vs NOTEARS-linear for recovering small Gaussian linear-SEM DAGs
Domain: ml (causal-inference, machine-learning)
Topic: Comparing causal structure learning algorithms (PC, GES, NOTEARS-linear) on small synthetic linear-SEM DAGs of 5-15 nodes with Gaussian noise
Venue: ARC-Bench 2026
Headline metric: structural_hamming_distance (minimize)

Research briefing:
Causal structure learning methods often claim asymptotic guarantees, but in
practical settings researchers usually face small graphs, finite samples, and
imperfect hyperparameter choices. For Gaussian linear structural equation
models (SEMs), three widely discussed paradigms are constraint-based (PC),
score-based (GES), and continuous optimization (NOTEARS-linear). Even when
data are generated from the model class these methods assume, finite-sample
behavior can differ sharply in edge recovery and orientation errors.

A compact CPU-scale benchmark can isolate these differences by generating
synthetic DAGs with known ground truth over 5-15 nodes, sampling linear-Gaussian
observations, and evaluating structural recovery against the true adjacency.
This avoids data-download overhead and lets the study vary sample size and
graph density in a controlled way. The key metric is Structural Hamming
Distance (SHD), which penalizes missing, extra, and wrongly oriented edges.

A credible study should implement all three algorithms with transparent
assumptions: PC via partial-correlation CI tests, GES via greedy score search
(e.g., BIC), and NOTEARS-linear via a differentiable acyclicity penalty and
L1 regularization. Multiple random seeds and at least two synthetic regimes
(sparser/easier vs denser/harder) are needed to reduce variance and avoid
over-interpreting one-off runs.

Because these algorithms expose different tuning knobs (significance level,
regularization strength, score penalties), the comparison should include one
clear default setting per method plus modest tuning on validation seeds. The
resulting analysis should report SHD and at least one precision/recall-style
edge metric, then map outcomes back to explicit hypotheses about relative
ranking and sample-efficiency.

*On small Gaussian linear-SEM DAGs, which of PC, GES, and NOTEARS-linear most reliably minimizes structural recovery error under realistic finite-sample budgets?*

Hypotheses:
statement: At n_samples=1000, NOTEARS-linear achieves lower mean SHD than PC on at least 2 of 3 synthetic DAG regimes (averaged over >=5 random DAG/data seeds per regime)., statement: At n_samples=300, GES achieves mean SHD <= PC in all evaluated regimes, and strictly lower SHD in at least 1 regime., statement: The best method at n_samples=1000 reduces mean SHD by at least 20% relative to the worst method in at least 2 of 3 regimes.

Experiment design:
research_question: For 5-15 node Gaussian linear-SEM DAGs, how do PC, GES, and NOTEARS-linear compare in SHD and edge recovery across sparse/medium/dense synthetic regimes and finite sample sizes?
conditions: name: pc_partial_corr
description: Constraint-based PC using Gaussian partial-correlation conditional independence tests with significance level alpha (default 0.01, optional small grid)., name: ges_bic
description: Score-based greedy equivalence search using Gaussian/BIC score with forward-backward edge operations., name: notears_linear_l1
description: Continuous optimization NOTEARS-linear with squared loss, acyclicity constraint, and L1 sparsity penalty tuned on a small lambda grid., name: pc_alpha_tuned
description: PC variant selecting alpha from {0.005, 0.01, 0.05} by lowest validation-seed SHD, then evaluated on held-out seeds.
baselines: pc_partial_corr is the classical constraint-based baseline, ges_bic is the classical score-based baseline
metrics: name: shd
direction: minimize
description: Structural Hamming Distance between learned and true DAG adjacency (lower is better), averaged over seeds., name: edge_f1
direction: maximize
description: F1 score on directed edge existence/orientation against ground truth adjacency., name: runtime_sec
direction: minimize
description: Per-run wall-clock time in seconds to assess CPU practicality.
datasets: name: synth_linear_sem_sparse
source: Synthetic 8-node DAGs generated via Erdos-Renyi with expected degree ~1.5; linear weights sampled from [-2,-0.5] U [0.5,2], Gaussian noise., name: synth_linear_sem_medium
source: Synthetic 12-node DAGs with expected degree ~2.5 using same linear-Gaussian SEM generator., name: synth_linear_sem_dense
source: Synthetic 15-node DAGs with expected degree ~3.5 using same linear-Gaussian SEM generator.
compute_requirements: gpu_required: no
estimated_wall_clock_sec: 720

Requirements:

Scoring rubric:
requirements: A credible experiment comparing causal structure learning methods (PC, GES, NOTEARS-linear, or equivalents) on small synthetic Gaussian linear-SEM DAGs: methods are implemented, runs cover multiple synthetic regimes with multiple seeds, and results address H1/H2/H3 directionally using SHD-centered evidence.
judging_note: Score on scientific substance and directional correctness of evidence, not on exact numeric thresholds. Alternative structure-learning methods or linear-SEM variants that test the same scientific question should be credited.
sub_tasks: requirements: Core causal structure learning conditions and synthetic data generation are implemented correctly.
sub_tasks: requirements: The submission implements distinct code paths for multiple structure learning methods — typically PC, GES, and NOTEARS-linear or equivalents — rather than reusing one estimator under different names.
task_category: Code Development
finegrained_task_category: Method Implementation, requirements: A reproducible synthetic linear-Gaussian SEM DAG generator is implemented with moderate node counts, including ground-truth adjacency export and sample generation.
task_category: Code Development
finegrained_task_category: Dataset and Model Acquisition, requirements: At least one tunable hyperparameter is exposed and used for the constraint-based method (e.g., PC's alpha) and the continuous-optimization method (e.g., NOTEARS' sparsity weight), with documented default or grid values.
task_category: Code Development
finegrained_task_category: Hyperparameter Tuning, requirements: The benchmark produces comparable metrics across methods/regimes.
sub_tasks: requirements: Execution covers multiple synthetic regimes and reports results for the primary methods at one or more sample sizes.
task_category: Code Execution
finegrained_task_category: Experimental Setup, requirements: A machine-readable results artifact includes numeric SHD for each (method, regime) and at least one additional metric such as edge-F1 or runtime.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Each reported cell is averaged over multiple random seeds with a dispersion measure. Honest small-seed runs with variance reported are preferable to a single run.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Results analysis addresses H1/H2/H3 directionally.
sub_tasks: requirements: The submission compares NOTEARS-linear vs PC at a reasonable sample size and conveys whether NOTEARS achieves meaningfully lower SHD on most regimes — judge directionally against H1.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The submission conveys qualitative verdicts for H2 and H3 using SHD aggregates, including whether larger sample sizes or denser-graph regimes yield meaningful SHD improvements.
task_category: Result Analysis
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: The README or writeup describes setup, reports key SHD/edge-F1/runtime findings, conveys per-hypothesis outcomes (supported / refuted / inconclusive), and notes limitations (synthetic-only scope, small-node regime, hyperparameter sensitivity, finite seeds). No strict word-count requirement.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation