# idr0071 External Validation: Experiment Summary

## Objective

Demonstrate that gene-disjoint splits prevent biological information leakage on an independent dataset (idr0071 experiment C, Feldman et al. CRISPRko), strengthening the NeurIPS 2026 paper's central claim.

## Why Experiment C

The idr0071 study (Feldman et al. 2019) comprises six experiments (A–F) spanning different cell lines, perturbation modalities, and readouts. Experiment C was selected for external validation based on five criteria:

1. **Independent cell line**: A549 (lung adenocarcinoma) provides cross-cell-line generalization relative to the primary Vesuvius dataset (HeLa). If gene-disjoint leakage holds across cell lines, it strengthens the claim that this is a structural property of the experimental hierarchy, not an artifact of a specific biological system.

2. **Sufficient gene coverage**: The CROPseq NFkB library targets 949 genes with 5,738 sgRNAs (median 6 sgRNAs/gene), providing statistical power for gene-disjoint train/test splits with >100 test genes per seed. Experiments with fewer genes (e.g., experiment F with ~30 targeted genes) would not support robust split evaluation.

3. **CRISPRko modality match**: Experiment C uses the same perturbation modality (CRISPR knockout) as Vesuvius, isolating the cell-line variable while holding the perturbation mechanism constant. This makes the comparison directly interpretable.

4. **Processed data availability**: Per-tile reads.csv files (barcode calling results) are available on the IDR FTP server, enabling cell-level barcode-to-gene mapping without requiring raw image processing or access to the original SBS image analysis pipeline. Other experiments in idr0071 lack these intermediate outputs on the public FTP.

5. **Static immunofluorescence readout**: The p65 antibody staining provides a biologically well-characterized readout (NF-κB pathway activation), which is orthogonal to the broad morphological profiling in Vesuvius. This tests whether leakage manifests regardless of the phenotypic readout.

Experiments A and B use live imaging (time-lapse), which introduces temporal dependencies that complicate cell-level splitting. Experiments D–F use different libraries or are smaller scale. Experiment C is the largest static-readout CRISPRko experiment with processed data available, making it the strongest candidate for a direct comparison.

## Dataset

- **Source**: IDR project 1504 — idr0071-feldman-crisprko, experiment C
- **Cell line**: A549 (lung adenocarcinoma)
- **Screen**: Static p65 antibody, CROPseq NFkB library
- **Wells acquired**: A1, B1 (2 of 6 available)
- **Tiles**: 1,066 (533 per well)
- **Raw data**: 1,358 MB reads.csv files from EBI FTP

## Cell calling

- Library: 5,738 sgRNAs targeting 949 genes + 100 non-targeting controls
- Barcode matching: 9-mer SBS prefix → sgRNA[:9] lookup (no collisions)
- Cell calling: dominant barcode per segmented cell, majority vote
- **Result**: 84,556 cells, 947 genes, 3,731 sgRNAs
- Median cells per gene: 21
- Median barcode purity (fraction): 0.50

## Split protocols

### Gene-disjoint (3 seeds: 42, 123, 456)
- 70/15/15 gene-level split, no gene in >1 partition
- ~620 train / 133 val / 132 test genes per seed
- **Gene overlap: 0.0% (verified)**

### Random cell-level (3 seeds: 42, 123, 456)
- 50/50 cell-level shuffle, genes deliberately shared
- 41,666 train / 41,666 test cells
- **Gene overlap: 99.7–100% (measured leakage)**

## Features

25-dimensional per-cell feature vectors extracted from read quality scores (Q_min, Q_mean, Q_median, etc.) plus barcode purity fraction. These are low-signal features — the point is to test whether leakage creates spurious discriminability, not to achieve high accuracy.

## Results

| Regime | Accuracy (mean ± std) | Macro-F1 (mean ± std) | F1 95% CI |
|--------|----------------------|----------------------|-----------|
| Gene-disjoint | 0.0% ± 0.0% | 0.0% ± 0.0% | [0.0, 0.0] |
| Random cell | 37.6% ± 0.0% | 13.5% ± 0.0% | [13.0, 13.8] |
| Label shuffle | 0.0% | 0.0% | — |

### Interpretation

- **Gene-disjoint = 0%**: Read-quality features carry no true gene-level signal. The kNN classifier correctly fails when train and test genes are disjoint.
- **Random cell = 37.6%**: The same features achieve 37.6% accuracy under random splits — entirely due to gene leakage. When the same gene appears in train and test, the classifier memorizes gene-correlated quality patterns.
- **Label shuffle = 0%**: Confirms no spurious signal from the split procedure itself.

This result directly supports the paper's claim: naive cell-level splits inflate metrics via biological information leakage, while gene-disjoint splits are leakage-free.

## Artifacts

| File | Description |
|------|-------------|
| `tables/cells.parquet` | 84,556 × 12 canonical cell table |
| `tables/genes.parquet` | 947-row gene summary |
| `tables/guides.parquet` | 3,731-row guide summary |
| `tables/mapping_qc.json` | Cell-calling QC statistics |
| `splits/gene_disjoint_seed{42,123,456}.json` | Gene-disjoint splits |
| `splits/random_cell_seed{42,123,456}.json` | Random cell splits |
| `features/per_cell_features.parquet` | 83,332 × 25 feature matrix |
| `results/main_metrics.json` | Full protocol comparison results |
| `results/leakage_audit.csv` | Per-seed gene overlap audit |
| `results/split_qc.json` | Split size and overlap QC |
| `figures/fig_protocol_comparison.{pdf,png}` | Bar plot: macro-F1 by regime |
| `figures/fig_leakage_overlap.{pdf,png}` | Gene overlap by regime and seed |
| `figures/fig_accuracy_vs_leakage.{pdf,png}` | Accuracy vs gene overlap scatter |
| `figures/fig_cells_per_gene.{pdf,png}` | Cell count distribution histogram |
| `checksums_outputs.sha256` | SHA256 checksums for all outputs |

## Reproducibility

All random operations seeded with {42, 123, 456}. Pipeline scripts in `data/idr0071/download/` and `data/idr0071/scripts/`. Raw data from `ftp.ebi.ac.uk/pub/databases/IDR/idr0071-feldman-crisprko/`.