AutoSciDACT: Automated Scientific Discovery through Contrastive Embedding and Hypothesis Testing
Authors: Samuel Bright-Thonney
Christina Reissel
Gaia Grosso
Nathaniel S. Woodward
Katya Govorkova
Andrzej Novak
Sang Eon Park
Eric A. Moreno
Philip Harris
Venue: NeurIPS 2025 poster ()
Keywords: anomaly detection
contrastive learning
scientific discovery
hypothesis testing
automated discovery
Primary area: machine_learning_for_sciences

Abstract: Novelty detection in large scientific datasets faces two key challenges: the noisy and high-dimensional nature of experimental data,
and the necessity of making *statistically robust* statements about any observed outliers. While there is a wealth of literature on anomaly detection via dimensionality reduction, most methods do not produce outputs compatible with quantifiable claims of scientific discovery. In this work we directly address these challenges, presenting the first step towards a unified pipeline for novelty detection adapted for the rigorous statistical demands of science. We introduce AutoSciDACT (Automated Scientific Discovery with Anomalous Contrastive Testing), a general-purpose pipeline for detecting novelty in scientific data. AutoSciDACT begins by creating expressive low-dimensional data representations using a contrastive pre-training, leveraging the abundance of high-quality simulated data in many scientific domains alongside expertise that can guide principled data augmentation strategies. These compact embeddings then enable an extremely sensitive machine learning-based two-sample test using the New Physics Learning Machine (NPLM) framework, which identifies and statistically quantifies deviations in observed data relative to a reference distribution (null hypothesis). We perform experiments across a range of astronomical, physical, biological, image, and synthetic datasets, demonstrating strong sensitivity to small injections of anomalous data across all domains.

AutoSciDACT: Automated Scientific Discovery through Contrastive Embedding and Hypothesis Testing
Samuel Bright-Thonney" Christina Reissel Gaia Grosso Nathaniel Woodward 1,3
Katya Govorkova' Andrzej Novak Sang Eon Park² Eric Moreno Philip Harris
Departmment of Physics, Massachusetts Institute of Technology
2 The NSF AI Institute for Artificial Intelligence and Fundamental Interactions
3 Department of Physics, University of Wisconsin, Madison
Abstract
Novelty detection in large scientific datasets faces two key challenges: the noisy and high-dimensional nature of experimental data, and the necessity of making statistically robust statements about any observed outliers. While there is wealth of literature on anomaly detection via dimensionality reduction, most methods do not produce outputs compatible with quantifiable claims of scientific discovery. this work we directly address these challenges, presenting the first step towards a unified pipeline for novelty detection adapted for the rigorous statistical demands of science. We introduce AutoSciDACT (Automated Scientific Discovery with Anomalous Contrastive Testing), general-purpose pipeline for detecting novelty scientific data. AutoSciDACT begins by creating expressive fow-dimensional data representations using a contrastive pre-training, leveraging the abundance of high- quality simulated data in many scientific domains alongside expertise that can guide principled data augmentation strategies. These compact embeddings then enable extremely sensitive machine learning-based two-sample test using the New Physics Learning Machine (NPLM) framework, which identifies and statistically quantifies deviations in observed data relative to reference distribution (null hypothesis). We perform experiments across a range of astronomical, physical, biological, image, and synthetic datasets, demonstrating strong sensitivity to small injections of anomalous data across all domains.
Introduction
Scientific discovery is often characterized by serendipity: an unexpected observation turns out have a profound impact field, leading rapid progress discovery. Today's data-rich scientific landscape is potentially brimming with curious or unexplained observations, but the scale and complexity of available data increasingly obscures genuine novelties behind statistical noise or incidental fluctuations. As scientific datasets continue to grow, so too grows the challenge of uncovering meaningful unexplained phenomena.
The scientific method traditionally provides a structured framework for discovery, encompassing observation, inquiry, research, hypothesis formulation, experimentation, and conclusion (top of Fig. I] Effective implementation of this method relies on human intuition and domain expertise to identify relevant observables and devise meaningful experiments. However, given the magnitude of modern datasets, the process would significantly benefit from automated tools to efficiently identify the most promising regions for discovery.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Observe lake data un understand
Inquire
Research Search for answers
Hypothesize Famuu hypoth esis
Test Experiment
Conclude Results
Ask a question
Smulation reference
Obsonation
Swirfy galanies loot different?
Expert Input What to look at? reference?
Swirtines
Test
AutoSciDACT
Reduce Simplify with symmetry
Isolate Select subset
Learn Space Organize known signals to create a reference model
Anomaly Detection & Test Compare incoming data with reference access s grificance with a stat. test
Dara
Olseraaion
UNIDERT
Cusa
High Dim. (2/N shown)
Low Dlim.
Low Dim.
Low Dim.
Test Stat.
Pre-trained / dourdation model
Data Supervised contrastive
Resulting ref. model
Search overdensity
anomalous regions
Quantily significance
learning
Figure I: Illustration of the scientific method (top row) and the AutoSciDACT pipeline (bottom row), emphasizing the corresponding methodological steps implemented within AutoSciDACT.
To address this challenge, is essential develop methods that intelligently prioritize informative regions- areas where genuine scientific surprises are most likely to emerge. Traditional feature- engineering approaches are human driven and domain-specific, limiting scalability and generalizabil- ity. Recent advances utilizing agentic Al systems and large language models can partially automate aspects of scientific inquiry [133], yet still lack integrated frameworks capable of rigorous, automated hypothesis testing and validation.
We introduce AuT (Automated Scientific Discovery with Anomalous Contrastive Testing), a pipeline that parallels the scientific method and streamlines scientific inquiry by automating key steps of the scientific discovery process. AutoSciDACT streamlines the phases of data reduction, hypothesis formulation, and statistical testing (bottom row of Fig. D by deploying contrastive learning together with the New Physics Learning Machine (NPLM) 4. Contrastive learning is used to reduce raw, sigh-dimensional datasets into expressive oww-dimesiional feature embeddings while NPLM provides a statistically robust mechanism for identifying and quantifying novel structures within this learned embedding space. A key insight with AutoSciDACT is the effective, automated incorporation of domain expertise as tool reduce the dimensionality to small number of well-behaved features, making it possible to construct robust statistical model. Using NPLM, our pipeline systematically compares incoming data with reference distributions known (background) data, finding the most anomalous regions and quantifying their statistical significance.
We validate our approach using synthetic benchmarks and real-world datasets from astronomy, physics and biomedical domains. AutoSciDACT reliably detects meaningful novelties while remaining robust against spurious variations. Our results demonstrate the promise of combining structured contrastive learning methods with automated statistical hypothesis testing to accelerate scientific discoveries.
Contributions Our main contributions be summarized as follows:
An end-to-end pipeline for novelty discovery in scientific datasets that is readily transferable across domains.
A principled procedure for incorporating scientific simulations, hand-labeled data, and expert knowledge into a contrastive dimensionality reduction pipeline.
A statistically rigorous framework for quantifying the significance of observed anomalies, beyond simply flagging anomalous datapoints.
A realistic demonstration of novelty detection in four disparate scientific domains.

---

2 Related Work
Contrastive Learning Contrastive learning is a powerful tool for learning expressive, low- dimensional data representations. Self-supervised methods such SimCLR (5). MoCo [6] VI- CReg [7]. and Barlow Twins [ use data augmentations (e.g., blurring, cropping) to promote semantically meaningful and well-separated embeddings. Supervised contrastive learning ( uses class labels to define positive pairs, more efficiently capturing semantic relationships between data- points but requiring labeled datasets to train. Its applications have expanded structured data [T0] multimodal inputs [II], and domain-specific tasks [12). demonstrating broad utility.
Contrastive Anomaly Detection Several existing methods leverage contrastive embeddings to search for instances for e.g. industrial applications. In contrast, AutoSciDACT (our method) is tailored for scientific contexts and makes statistical statements about the presence of anomalous data, identifying distribution-level deviations relative a baseline expectation (null hypothesis). Existing approaches include Refs. [3421], all of which rely primarily on the AUROC for identifying OOD points as figure of merit. They do not attempt to statistically quantify observations of OOD data, is required in the scientific context. CADet [13] is nearest to our setup in using a two-sample test, but focuses again on AUROC. In scientific contexts, various combinations of contrastive learning and anomaly detection have been used for domain-specific applications c.g. in astronomy [22 23).
(OOD) data, but primarily focus on identifying individual anomalous
histology, [4 25 and particle physics 266 27 but, to our knowledge, no unified approaches have
been proposed.
Hypothesis Testing for Anomaly Detection Traditional goodness-of- fit (GOF) tests are powerful in univariate settings but struggle beyond that due to the curse of dimensionality. Simple multivariate extensions have limiting factors. The Mahalanobis distance-based test (28). for instance, assumes Gaussianity and is sensitive to the choice of covariance estimation, limiting its applicability in complex data regimes. Recent machine learning-based methods have introduced model-agnostic, data- driven alternatives to enable non- parametric, highly adaptable high -ige--immnsionall tests. To quantify distributional differences, Maximum Mean Discrepancy (MMD) [19-3]] embeds distributions into a reproducing kernel Hilbert space, while the Classifier Two- Two-Sample Test (C2ST) 32 uses a trained
classifier's accuracy. Other methods include density-ratio estimation [33 and generative modeling frameworks that assess sample likelihoods [34]. In this work, we draw from statistical anomaly detection tests developed for high- energy physics, where sensitivity to subtle deviations is crucial. Autoencoders [35 46) and semi-supervised binary classifiers [47] have been widely used to score
anomalies, but not statistically test them. The NPLM algorithm [8 49 was introduced as an end-to- end score-and-test tool, outperforming classic GOF and classifier tests (50). showing sensitivity to a wide class of anomalies, and allowing incorporation of systematic (epistemic) uncertainties (51].
3 The AutoSciDACT Pipeline
The AutoSciDACT pipeline consists of two phases: pre-training and discovery. The aim of the pre-training phase is to learn an expressive, low-dimensional representation of scientific dataset that retains key semantic features while reducing potentially hundreds or thousands of input dimensions to a handful. The discovery phase uses these embeddings in the NPLM anomaly detection and hypothesis testing framework to search for novelty in a scientific dataset. AutoSciDACT is designed for discovering statistically significant anomalies, prioritizing detection of distributional shifts (e.g. overdensities, distortions, outlier clusters) with respect to background-only hypothesis, rather than instance-level anomalies. The power of NPLM (and any statistical test) degrades with data dimensionality, quickly requiring prohibitively large sample sizes to make statistically significant observations of small signals. As such, the reduction in the pre-training phase is critical. The bottom row of Fig. summarizes the key steps and features of AutoSciDACT.
3.1 Pre-Training: Contrastive Embeddings
The backbone of our pipeline is an encoder ʃo : X' + Rd trained with contrastive learning to map raw data from its high-dimensional input space X' to a low-dimensional representation in Id. Contrastive objectives are designed to maximize alignment between like inputs (positive pairs) while separating unlike inputs (negative pairs) in the learned space. We use the SimCLR framework (5), which trains
3

---

an encoder fo alongside projection head 96 (typically small MLP) with the following contrastive loss:
SIMLLR = - log TEB
(I)
Ej exp(sim(z;,
where = (x), (z:; 2,) are positive pair, sim(:,) is the cosine similarity, is configurable temperature, and the sum in the denominator runs over all other pairings in a batch B. Traditionally, positive pairs (x, x.) are constructed on-the-fly from inputs x; using random augmentations that preserve semantic meaning. The projection head Js is discarded after training, with embeddings h (x) used for downstream tasks.
In AutoSciDACT we use supervised contrastive learning (SupCon) 9 which leverages labeled training data to create positive pairs from the same class and negative pairs from different classes. The training objective a simple generalization the SimCLR loss:
exp(sim(zz) 2p)/T)
CsupCon
Σ İEB
log
(2)
P()
Ejjt exp(sim Z,, 2z)/T)²
PEP()
where P(i) is the set of all positive (same-class) pairs of input in the batch. Using labels to define P()) encodes much richer notion of similarity from the full spectrum of a given input class, rather than having to indirectly learn (or fail to learn) important features from views of individual inputs. This also avoids the ill-defined question of identifying the "best" augmentations that promote expressive learned features. Practitioners in many scientific domains have ready access to large quantities of labeled training data from high-quality simulations or expert-labeled databases, so requiring labels is not often a significant bottleneck. When augmentations are desirable to encourage learning scientifically relevant meta-features (e.g. Lorentz invariance for particle physies datasets), or if class labels are unavailable, SupCon can easily incorporate augmented views in the positive set P(i). In conjunction with labels, this offers way to inject additional scientific domain knowledge via tailored augmentations.
In addition the contrastive objective LsupCon include an optional supervised cross-entropy loss fCE. which we found beneficial for learning embeddings with a more regular structure and class separation. Our full loss function is thus
== LSupCon + XCFLCE;
where ÀCE 0.1 0.5 is set make the classification objective sub-dominant.
3.2 Discovery: Anomaly Detection & Hypothesis Testing
In the discovery phase we use the emhedding fa to process unseen datasets and search for anomalous clusters. overdensities, or outliers in the low-dimensional space. The search process is a classic scientific hypothesis test: a reference dataset R composed of known backgrounds is compared to an observed dataset 2 of unknown composition, and seek to accept or reject the null hypothesis that R and 'D are identically distributed (i.e. there are no new phenomena in the observed data). We implement this test with NPLM, which in conjunction with the expressive learned embeddings enables extraordinary sensitivity to new signals.
The NPLM algorithm NPLM builds the classical likelihood ratio introduced by Neyman et al. 52 using a test statistic defined as:
[(x|HLv) [(x Ho)
(D) = 2 max
log
RED
trainable model fw (x: parametrizes family of alternative hypotheses Haw with respect to the null Ho on inputs E Rd with corresponding alternative density of the form:
H.) = (o) exp[fw (z)].
(5)
This formulation enables signal-agosstc approach: instead of specifying a particular signal model, the algorithm learns the deviation directly from data by solving a maximum likelihood problem reframed machine learning task. We follow the model introduced in (4, where the problem is

---

solved as binary classification between the data of interest D, labeled = 1, and the reference sample R, labeled y 0. The model is Nyström approximated kernel method
M
fw
(6)
with M
/ [D] [R] Gaussian kernels K; and trainable mixture coefficients {w; E [R and minimizing a regularized weighted binary cross-entropy
[NPLM[F]] = Σ
[wR(11 - log (1 + e/w) + log (1 + e + Σ w. w,w
(7)
(2.9)
To ensure robustness, the size of the reference sample R| is chosen to be substantially larger than the data |D] and reweighted so that the expected yield under Ho matches the expected experimental one, which may differ from the observed size [D]. This design choice makes the test sensitive to both shape and normalization deviations
Once the training is complete, the test statistic is estimated from the solution fm as
INP = -2 Σ WR (1 y) (efmn(e) -1) =)
(8)
(5.9)
To calibrate the test, we estimate the distribution p(tNP Ho) via pseudo-experiment ("toys"), gen- erating (i.e. sampling from a larger pool) datasets under the null hypothesis and computing the corresponding test statistics form an empirical distribution To. The p-value is then evaluated empirically as:
Σ LET
((D).
p =
To
It can also be estimated asymptotically from the distribution of To. which can be fit to suitable x² distribution [4]. The asymptotic estimate is useful in cases where the deviation of D with respect to R is large (e.g. Z = 5o)), in which case the number of toys required for the empirical estimate would be prohibitively large.
The power of the NPLM test strongly depends on the choice of the kernels' width, as it determines the scale of distortions the model sensitive to. To mitigate this feature and make the model more robust, we adopt an extended version of the algorithm introduced in (53] where multiple widths are considered and combined to obtain a final p-value. The authors [53] explore several options for combining the tests based on "local" p-values, but in this work we choose the average of p-values as a rule. The average score is typically less powerful than the single "optimal" kernel, which can be considered a kind of "look-elsewhere" effect accounting for various kemnel hypotheses.
We consider six different kernel widths for our experiments, with their precise numerical values chosen according to the distribution of pairwise distances between data points in the embedding space. More precisely, the first five values are the 1st, 25th, 50th, 75th. and 99th percentiles of the empirical pairwise distance distribution (computed with subset data points from the training set); the last value twice the 99th percentile, and it ensures sensitivity to -fitt--diitiiiuttin anomalies. This choice means the numerical kernel widths vary among datasets, so we denote them by their corresponding quantiles: oeer E {91, 925, (50, [75: 499, 2499)
The NPLM procedure provides flexible, multivariate, unbinned likelihood-ratio test that is agnostic the source of the anomaly, making it well-suited for unsupervised anomaly detection tasks. Comparisons with alternative GoF approaches presented in [50] show the impressive sensitivity of the method to subtle distortions of the data density distribution. As for any GoF approach relying on density estimation from empirical samples, limitations arise when scaling the data dimensionality. In this work, we target the curse of dimensionality by compressing high-dimensionall raw data with contrastive embeddings.
'This sensitivity is important in contexts where data collection windows determine (D). and deviations in event rates may signal anomalies.
5

---

4 Datasets
We demonstrate AutoSciDACT on a diverse collection of five synthetic, image, and scientific datasets. Each dataset contains a large collection of data from "background" (i.e. well-understood) classes that are used in the contrastive pre-trate mee-trining phase, along with a set of anomalous "signal" data. In the discovery phase we construct datasets 'D with small signal injections and use NPLM to detect the novel component. We briefly describe each dataset below, with additional details available in App. Due to space constraints, novel butterfly hybrids from wing images (App. A.6). and searching for four-lepton decays of the Higgs boson in real LHC data (App.
defer two further studies to the appendix: genomics task identifying
Synthetic Data The synthetic dataset is designed to illustrate the core functionality of AutoSciDACT independent of details specific to scientific datasets. It consists of points X' C RD+M with D meaningful dimensions and M noisy dimensions. The noisy dimensions are sampled from (0, 1). and the meaningful dimensions are populated by N Gaussian clusters (M,,, Eg) (i (i=13 1...., N)
with means μe ~ ((001 1) and covariances Σ; ~ [0, 0.5). The pairwise distances of the Gaussian clusters are adjusted such that 1 % injection of one cluster top of 10k samples of another yields deviation of 3.5 standard deviations. The full D + -idieenniional space is then randomly rotated to obscure the discriminating variables. The contrastive embedding is trained on N 1 clusters with one held out as a signal, with the backbone architecture so being a simple ML..
Astronomy For an astronomical baseline we choose gravitational wave data recorded by the Laser Interferometer Gravitational- Wave Observatories (LIGO) in Hanford, WA and Livingston, LA 154] Although gravitational waves from compact binary systems have been detected [55], many hypothetical sources remain unobserved, making the challenge particularly intriguing for anomaly detection methods. The data consist of 50 ms time-series signals from two channels one for each interferometer - sampled at 4096 Hz (200 measurements per channel) (56] 57). The different classes of data consist of pure background (~~ gaussian noise), "glitches" (periods of short-duration transient instrumental noise), and six observed or hypothetical sources of astrophysical signals. A seventh signal class with "white noise burst" (WNB) waveforms is held out from pre-training and is injected as an anomaly in the discovery phase. The encoder architecture is a dimedimensi ResNet, following the technical setup explored in [58] for identification of binary black holes.
Particle Physics Our particle physics baseline is JETCLASS 55 60), a large dataset consisting of
simulated jets: energetic, collimated streams of O(100) particles that are produced in proton-proton collisions at the Large Hadron Collider (LHC). We use a subset of JETCLASS consisting of jets from quantum chromodynamics (QCD) processes (quark/gluon), top quark decays (t - bqq'). ) and W/Z vector boson decays (V - qq'). We hold out signal jets from boosted Higgs boson decays to bottom quarks (H + bh), inspired by recent measurements of H -> bb in the combined gluon fusion and vector boson production modes by the CMS experiment [6] We use the Particle Transformer (ParT) architecture [60] variant of the Transformer architecture contrastive encoder.
62 adapted for particle physics the
Histology As example from life sciences, we aim to identify abnormal tissue in histopathological images. The abundance of healthy tissue data and the difficulty in collecting samples with various abnormalities render histology particularly well-suited to noomaly-detectionn tasks. We use publicly available optical microscope images from stained tissue samples [63]. Our reference sample contains seven classes of tissue from mice (brain, heart, kidney, liver, lung, pancreas, spleen) and one class of normal liver tissue from rats. We aim to detect anomalous mouse liver tissue caused by non-alcoholic fatty liver disease (NAFLD). Inputs are 256x256 pixel (0.44um/pixel) resolution tissue tiles extracted from the whole slide image with Masson's ' trichome staining. As a backbone, we train the best performing architecture from Ref. (25). EfficientNet-BO (44)
Images We use the CIFAR-IO dataset (63, arbitrarily holding out class as the anomaly and pre-training the other nine classes. In the discovery phase of the pipeline, we use images from CIFAR-5m [66] to supplement the CIFAR-10 test set and expand the number of data points available for hypothesis testing F
We use ResNet-50 encoder backbone with pre-trained weights (67). swapping out only the final fully connected layer with a slightly larger MLP and fine-tuning it on the CIFAR contrastive embedding task.
"CIFAR-5m was introduced in (66) and consists of images generated by diffusion model trained on CIFAR-I0. 10, which were found to be nearly indistinguishable from CIFAR-I0 by pre-trained classifier.

---

QCD
W 44
SineGaustian
Kink
2
KinkXink
bb
Gaussian
Background
Cusp
Giitches
B
B
Y
5
B
B
Y
Figure 2: Contrastive embeddings for Particle physics (left) and Astronomy (right) datasets. The high-dimensional input is projected down to four dimensions (a, B, 7, 8). Background classes are shown in hues of blue and green, while the anomaly is overlaid in hot pink.
Experiments
Embedding We train and evaluate the AutoSeiDACC pipeline each dataset in the same manner, making only small adjustments in pre-training to adapt to the specifics of each dataset (see App. [ for full details). We fix the embedding dimension d 4 for all encoders to put each on equal footing for NPLM, whose performance varies with input dimensionality. The choice of low embedding dimension is made to ensure that statistical tests remain tractable, and demonstrate that it possible to obtain strong anomaly detection performance with very compact representation (see App.BB3. for a study of larger embedding dimensions). Fig. we visualize the learned contrastive embeddings for JetClass and LIGO, with embeddings of the anomalous class which was not included in the training- indicated pink. The anomalous cluster in JetClass manifests as an extended and distinct cluster, while in LIGO it is an overdensity near the latter would be challenging for traditional per-datapoint anomaly detection methods, but we will demonstrate that NPLM detects it as an overdensity.
peltgground-poppllatt region. Flagging
Anomaly detection & hypothesis testing We follow a standardized procedure for signal injection, anomaly detection, and hypothesis testing for each dataset. As described in Sec, 3.2 we compile a reference sample R from the test set composed entirely of the known classes used training. We then construct a "observed data" set D. also from the known classes and in the same relative proportions as R. We mimic the presence novelty in by injecting some number Ns fs| of anomalous signal datapoints from the held out class, where typically Iss X 0.1. For each injection rate S's, we run 500 NPLM pseudo-experiments to populate a distribution of test statisties (DD; fs), re-sampling D and signal injections each time To calibrate the test, we run 500 additional pseudo-experiments with fs = 0 to populate a reference distribution of ((D)fs = 0). The empirical and asymptotic p-value computed. For each dataset, we scan fs across range of injection fractions and plot the resulting Z-scores in Fig. 3 The size and composition of R and D are fixed by the practical limitations of each dataset (i.e. the test set size), and f's is varied in range where NPLM starts to become sensitive to the injected signal. This information is summarized in Table iin App. A Each panel of Fig, also includes results from three baseline statistical tests to compare with NPLM: two supervised tests that incorporate explicit knowledge of the signal, and a test based on the Mahalanobis distance [28].
and Z-score
Supervised baselines We use two fully-supervised baselines as an estimate of best-case anomaly detection performance. We denote them "supervised" and "ideal supervised", distinguishing the extent to which knowledge of the true signal is utilized. For the "supervised" baseline we train an MLP to identify the desired signal in the contrastive embedding space, while for "ideal supervised"
In cases where the test datasets are large enough, re-sample well. Full details are in Appendix A
7

---

Synthetic Dataset
12
Synthetic Dataset
JetClass, H 6ó anomall
Asymppaticc -qy)
Emrpriicll lowr sit
doyropted: MUM lo, mayi
Emppnica Dug kemeial
Eriet sal KILM low -uu
Erre xet RILM (sg tomelst
Manalanoch (tmoirict)
Idral. Supervtedd
Score Gel
Score
Z Score
Tace
Rox Sora wilh Nag 320
0-
sar Los 225 13s Fraction
Number Additional
Injected Anomaly Fraction
(b) Histology, Liver discase anomaly
(c)
LIGO, WNS amaly
CIFAR, Class anomaly
Asymptedic KRLM ldw 2ow)
10
Anymonst MPM aul
Empitca NPL low 20w)
Cmp ricll MPUM low
Empiita NFLM Lhug lemea)
Emp rical karreé
Mahulanoss (tmrpisca)
Supevise
Superssed
pupervios
Suparabed
Meal Superviedd
Iceal S.pertred
7 Score Go)
Z Score [c!
7 Scoore (c)
Eep with Mhyg
Zore #520
Nox Zeoo V Roy 500
is ads
Injected Anomaly Fraction
Injected Anomaly Fraction
Injected Anomaly fraction (%)
(d)
(e)
(I)
Figure 3: Statistical significances (Z scores) of NPLM and other baseline methods for detecting
various fractions of anomalous signals injected into (ekgroundddominatee samples in (a) scanning additional random variables to the same Synthetic toy at fixed fraction of 0.6% with 10k Background, and then for the fraction for (b) Synthetic (2k), (c) particle physics, (d) astronomy, (e) histology, and (f) image datasets. In all cases, NPLM is able to discover very small signals with high confidence. The upper limit of the empirical Z scores is indicated by a gray line roughly 2.88a and set by the fixed number of pseudo-cxperiments (500), so empirical numbers are not quoted beyond this point. The Asymptotic NPLM approximates the large pseudo-experiment limit at large Z scores.
baseline we do the same but first re-train the contrastive embedding with the true signal added to the training set (the encoder has explicit knowledge of the signal). The reference R. and observed 'D datasets are constructed the same way as NPLM, but the hypothesis test relies on more typical statistical methodology. We construct one-dimensional distributions of classifier scores (normalized to the range [). and use the points in R to construct a background-only shape template fR(S) and signal points to construct a signal template fs(s). We then perform a binned maximum likelihood fit to the classifier scores in 'D under the null (Ho D ~ aipR (s)) and alternative (H₃ : D ~ ajpR( (s) + a2ps(() (s)) hypotheses and compute the test-statistic Ax² x². X²I. We compute [68].
empirical and asymptotid p- and Z-scores over many
Mahalanobis baseline As a comparison with analytic anomaly score, we use the Mahalanobis distance metric (28). For each pseudo experiment we compute the mean μı; and covariance Σ, for the embeddings of each background class i in R. The Mahalanobis distance is then dMata (x, R) min;(x ) D (xx p4)) and we define the test statistic as M (D) EXED d (x, D).
Empirical p-values and Z -ccores are computed as before.
'Ax² is asymptotically x² distributed with degree of freedom.
8

---

6 Discussion
6.1 Results
The results in Fig. 3 clearly demonstrate the power of the AutoSciDACT pipeline, with NPLM flagging highly statistically significant deviations (Z ス 3 or S 10-3) with signal fractions as low as 1%. The two supervised baselines provide reasonable upper limit on the sensitivity to the signal given full knowledge of its distribution in the embedded space, and some cases, NPLM performs near this limit. Beyond roughly 50, some trends break down, but at this level of significance (p~ ~ 10 7) discovery is extremely clear.
In all but the synthetic datasets, NPLM significantly outperforms the Mahalanobis baseline due to the flexible range of distortions and overdensities is capable of modeling in the input space. The Mahalanobis test is best suited to cases where each background cluster is roughly normally distributed, and by construction is not sensitive to overdensities near the bulk of any given cluster. Since the synthetic dataset is constructed from Gaussian clusters, Mahalanobis is quite effective in this case. In Fig. 3(a), we leverage the computational efficiency of Mahalanobis distance by running over 100 toy synthetic datasets per point, comparing performance on raw versus embedded inputs. As is clear with the raw performance, additional random variables quickly destroy the sensitivity to hidden signals. Sensitivity is preserved across all numbers of random variables using the exed-dimensional embeddings as inputs, as the large number of noisy dimensions has little impact on the quality of the embedding.
For both the LIGO and JetClass dataset, we approach the supervised limit at a Z- zcoor of 3, which rivals or exceeds all anomaly detection algorithms within their respective domains [69-74]. While as- tronomy and particle physics have long leveraged statistically rigorous anomaly- their application to histology illustrates a successful transfer of methods across scientific disciplines. The results on the histology datasets align with the findings reported in (25], which demonstrate that embedding spaces constructed with label information outperform those based solely on data augmentations. With AutoSciDACT, we introduce a new method capable of detecting localized abnormalities that may be present in only small fraction of tissue, capability that is essential both for early detection of disease and for guiding pathologists' judgments on toxic compounds.
techniques,
6.2 Limitations
Domain knowledge Since AutoSciDACT relies exclusively on domain knowledge in the label information, its performance is highly correlated with the label quality. Although labeling is easy and accurate in some domains (e-g. simulations, or organ labels for histological patches), labeling large training subsets can be laborious or impossible. For all baseline results, we also assume equal distributions from all background classes in the reference distribution for both pre-training and discovery. However, the actual composition of the reference sample during discovery needs to resemble the one in the observed data, and may require additional input from domain experts. These are problems routinely solved by scientists, so they do not pose a major obstacle to implementing the pipeline.
Embedding dimensionality Embedding into a small space (d = 4) limits expressivity, though the features learned in contrastive pre-training will typically be more useful than handpicked variables. This most evident in the LIGO and CIFAR-10 results in Fig- 3 where the "ideal supervised" benchmark falls short of the supervised one when it should in principle do better. This is due to the density of large number of classes, which struggle to be perfectly separated in the four-dimensional space. The "ideal" scenario, including problem. The embedding dimension can be reasonably scaled up (see App. B.3). but beyond a certain point (e.g. hundreds thousands of dimensions) NPLM's sensitivity will degrade substantially due to the sparsity of the data.
additional class in the learned space, exacerbates this
Domain shift and uncertainties In all experiments, we assume that the reference dataset correctly resembles the background distribution of the data. While this an exact assumption in cases where is possible to label subsets of data, the reference sample might contain domain shifts if it is constructed from data recorded under different conditions or from simulation. The impact of domain shift on contrastive embeddings has been studied in (75), and the inclusion of epistemic uncertaintics within

---

both NPLM and the embeddings is possible 76) Extensions of AutoSciDACT, including domain shifts and estimation of the associated epistemic uncertainties, are left for future work.
Conclusion & Future Work
In summary, we have presented what is, our knowledge, the first end-to-end scientific pipeline for novelty discovery arbitrary datasets with a rigorous statistical foundation based on hypothesis testing. We show that using AutoSciDACT, we discover anomalous signals with high statistical signif- icance (2 30) even when the data contains only a percent-level signal fraction and the dimensionality of the data is large. By applying AutoSciDACT to five different datasets from four different scientific domains, we prove the methods' universality and transfcrability, enabled by the strict decor- relation of expert knowledge encapsulated in label information from the actual analysis pipeline. For a comprehensive scientific outcome, incorporating potential domain shifts along with their associated uncertainties is essential. We plan to further extend AutoSciDACT through known extensions of our methods. More generally, by abstracting the scientific method, our approach presents a framework that automates scientific discovery, leading to the possibility of rapid, comprehensive, and rigorous scientific analysis on all data.
7 Acknowledgments
This work is supported by the National Science Foundation under Cooperative Agreement PHY-2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental Interactions, http://iaifi.org CR acknowledges support under a Postdoc.Mobille fellowship from the Swiss National Science Foundation (SNF) (Grant No. 222340). Computations in this paper were the FASRC Cannon cluster supported by the FAS Division of Science Research Computing Group at Harvard University.
Following the publication NeurIPS. It was pointed out that reference [2] Al-hallucinated article. We acknowledge this mistake, which resulted from error in preparing the BibTeX citation using LLMs, prompting the LLM to generate the BibTeX citation with the correct first author and the first word of the title. The updated draft now includes the intended reference.
was linked to an incorrect,
References
[1] Chris Lu, Cong Lu, Robert Tjarko Lange. Jakob Foerster, Jeff Clune. and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint 2xXiv:4488.0622. 2024.
[2] Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist-v2: Workshop- automated scientific discovery via agentic tree search. arXiv preprint arXiv:2504. 08066, 2025.
[3] Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, and Ryutaro Tanno. Towards an ai co-scientist. arXiv preprint 250. 18864, 2025.
[4] Marco Letizia, Gianvito Losapio, Marco Rando, Gaia Grosso, Andrea Wulzer, Maurizio Pierini, Marco Zanetti, and Lorenzo Rosasco. Learning new physics efficiently with nonparametric methods. Eur. Phys. J. C, 82(10):879, 2022. doi: 10. N0.11440cppccs110552-0222 -22-1000----00---
[5] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference Machine Learning (ICML), pages 1597-1607. PMLR, 2020.
[6] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9729-9738, 2020.
[7] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variane-imarrance-ovvrine regular- ization for self-supervised learning, arXiv preprint arXiv:2105.04900 2021.
10

---

[8] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. International conference on machine learning. pages 12310-12320. PMLR, 2021.
[9] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sama, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33: 13:18661111 8661-18673, 2020.
[10] Beliz Gunel, Cheng Du, Alexis Conneau, and Veselin Stoyanov. Supervised contrastive learning for pre-trained language model fine-tuning. In International Conference on Learning Representations (ICLR), 2021.
[11] Alec Radford, Jong Wook Kim, Jack Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin. Miles Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[12] Mehdi Azabou, Mohammad Gheshlaghi Azar, Ran Liu, Chi-Heng Lin, Erik C Johnson, Kiran Bhaskaran- Nair, Max Dabagia, Bernardo Avila-Pires, Lindsey Kitchell, Keith B Hengen, et al. Mine your own view: Self-supervised learning through across-sample prediction. arXiv preprint ar 202::2002..10066 2021.
[13] Charles Guille-Escuret, Pau Rodriguez, David Vazquez, Ioannis Mitliagkas, and Joao Monteiro. Cadet: Fully self-supervised outtt--distributiin detection with contrastive learning, Advances in Neural Information Processing Systems, 36:7361-7376 2023.
[14] Puck de Haan and Sindy Löwe. Contrastive predictive coding for anomaly detection. arXiv preprint arXiv:2107.07888 2021.
[15] Izhak Golan and Ran Yaniv.. Deep anomaly detection using geometric transformations. Advances in neural information processing systems, 31, 2018.
[16] Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning distributionally shifted instances. Advances in neural information processing systems, 33:1 1839-1 1852, 2020.
[17] Hao Liu and Pieter Abbeel. Hybrid cewpiiinativv-geerrtttiii training via contrastive learning. arXiv preprint arXiv:2007.09070, 2020.
[18] Umar Khalid, Ashkan Esmacili, Nazmul Karim, and Nazanin Rahnavard. Rodd: A self- supervised approach for robust ouittf-distribution detection. In 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops (CVPRW), pages 163-170. IEEE, 2022.
[19] Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R Led- sam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, et al. Con- trastive training for improved ofitt--distributiin detection. arXiv preprint arXiv:2007.05566, 2020.
[20] Sina Mohseni, Mandar Pitale, JBS Yadawa, and Zhangyang Wang. Self-supervised learning for generalizable out-of-distribution detection. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04))5216-522) Apr. 2020, doi: 10. w3..:///...........566..66 URL https: //ojs. aaai htgp:/na..pp//AAA//tttttteeeeee 5966
[21] Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for self- supervised outlier detection. arXiv preprini 1rXiv:2103.1205: 2021.
[22] Nicolas Baron Perez, Marcus Brüggen, Gregor Kasieczka, and Luisa Lucie-Smith. Classifi- cation of radio sources through self-supervised learning, 2025. URI //ta//aivvvo////// org/
abs/ 2503..330 19111
11

---

[23] Sara Webb, Michelle Lochner, Daniel Muthukrishna, Jeff Cooke, Chris Flynn, Ashish Mahabal, Simon Goode, Igor Andreoni, Tyler Pritchard, and Timothy M C Abbott. Unsupervised machine learning for transient discovery in deeper, wider, faster light curves. Monthly Notices of the Royal Astronomical Sociery, 498(3):3077-3094 09 2020. ISSN 0035-8711. doi: 10. 1093//mnaas//ta2229555 UtLptts:://doo /tt///..///00 htt.1100//ma//////e /wt.///ra//s/eemmm / staa2395
[24] Benjamin Voigt, Oliver Fischer, Bruno Schilling, Christian Krummow, and Christian Herta. Investigation of semi- and self-supervised learning methods in the histopathological domain. Journal of Pathology Informatics, 14: 100305, 2023. ISSN 2153-3539. doi: https://doi.or// 10. 1p.1016/j.ipi.2022... 100305. URL wwwphhtpps/wweeee hwtp///www.cciecccdrreete.... wttp://teetttccccoooeeeeeeeeeeee
com wttp////cccc///e/cccc/c.cccc/cce articce/
pi1/SS1153353923001199
[25] Igor Zingman, Birgit Stierstorfer, Charlotte Lempp, and Fabian Heinemann. Learning image representations for anomaly detection: Application to discovery of histological alterations
in drug development. Medical Image Analysis, 92:103067, 2024. ISSN 1361-8415 doi:
https://doi.org/11... 1ttp:///....11m...mmmaaaaaaaaaaa sciencce/arriccee////////1115...
103067. URL https: // wttp:wwwccict....... sciencedirect. com/
[26] Barry M. Dillon, Radha Mastandrea, and Benjamin Nachman. Sele-supe anomaly detection for new physics. Phys. Rev. D. 106:056005, Sep 2022. doi: 10. 103/PhysRevD...... aps
056005. URL hwt.hhtttp:/tiiitii wttp:/i...ppp......o/oo/ooo ort.://////o///////o 1100:...11111111 11100///////S11 106. 006005
[27] Kyle Metzger, Lana Xu, Mia Sodini, Thea K. Arrestad, Katya Govorkova, Gaia Grosso, and Philip Harris. Anomaly preserving contrastive neural embeddings for end-to-end model- independent searches at the LHC. 2 2025.
[28] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks, Advances in neural information processing systems, 31, 2018.
[29] Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(Mar):723-773 2012.
[30] Felix Biggs, Antonin Schrab, and Arthur Gretton. Mmd-fuse: learning and combining kernels for two-sample testing without data splitting. NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
[31] Antonin Schrab, Im Kim, Mélisande Albert, Béatrice Laurent, Benjamin Guedj, and Arthur Gretton. Mmd aggregated two-sample test. J. Mach. Learn. Res., 24(1), January 2023. ISSN 1532-4435.
[32] David Lopez-Paz and Maxime Oquab. Revisiting classifier two-sample tests. In International Conference on Learning Representations (ICLR), 2017.
[33] Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation in machine learning. Cambridge University Press, 2012.
[34] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? In International Conference Learning Representations (ICLR), 2019.
[35] Olmo Cerri, Thong Q. Nguyen, Maurizio Pierini, Maria Spiropulu, and Jean-Roch Vlimant. Variational Autoencoders for New Physics Mining at the Large Hadron Collider. JHEPP 05: 036, 2019. doi: 10. 10.1007//HEE05(201))00
[36] Pratik Jawahar, Thea Aarrestad, Nadezda Chernyavskaya, Maurizio Pierini, Kinga A. Wozniak, Jennifer Ngadiuba, Javier Duarte, and Steven Tsan. Improving variational autoencoders for new physics detection at the lhe with normalizing flows. Frontiers in Big Data, Volume 5 5022 2022. ISSN 2624-909X. doi: 2022338////aaa222200222222 803685. URL, hhttps://w/t frontiersin amg/joornals/////m /bgg data/ ceeta/artian http://riicclsssee 3389/ fww.a.aaaaaamaamamaaaaaaaaaaaaaa 2022.
2022.803625 803685
12

---

[37] Ekaterina Govorkova et al. Autoencoders on field-proo programmaee gate arrays for real-time unsupervised new physics detection at 40 MHz the Large Hadron Collider. Nature Mach. Intell., 1614-161, 2022. doi: 10. 0011008/44236--002-0041111
[38] Thorben Finke, Michael Krämer, Alessandro Morandini, Alexander Mück, and Ivan Oleksiyuk. Autoencoders for unsupervised anomaly detection in high energy physics. JHEP, 06:161, 2021. doi: 10.1007/JHEF00((221)))
[39] Abhijith Gandrakota. Realtime Anomaly Detection at the L1 Trigger of CMS Experiment.
PoS, ICHEP2024:1025, 2025. doi: 10.22323/1 1776.1025.
[40] Vasilis Belis, Patrick Odagiu, and Thea Klaeboe Aarrestad. Machine learning for anomaly detection in particle physics. Reviews in Physics, 12:100091, 2024. ISSN 2405-4283. doi: https//ddbbooop///00000//eeevvvp 100091. URL hwtp://www.cciieeedreeeet.... com/
[41] D. Abadjiev et al. Autoencoder-Basse Anomaly Detection System for Online Data Quality Monitoring of the CMS Electromagnetic Calorimeter. Comput. Softw. Big Sci., 8(1):11, 1, 2024.
doi: 10. 10.1007/--1811--24-00116
[42] Oliver Knapp, Olmo Cerri, Guenther Dissertori, Thong Q. Nguyen, Maurizio Pierini, and Jean- Roch Vlimant. Adversarially Learned Anomaly Detection on CMS Open Data: re-discovering the top quark. Eur. Phys. J. Plus, 136(2):236, 2021. doi: 10. 1 109-4.
[43] Sang Eon Park, Dylan Rankin, Silviu-Marian Udrescu, Mikacel Yunus, and Philip Harris. Quasi Anomalous Knowledge: Searching for new physics with embedded knowledge. JNEEP 21:030, 2020. doi: 10.1007/JHEEP6((222))00)
[44] Filip Morawski, Michal Bejger, Elena Cuoco, and Luigia Petre. Anomaly detection in gravitational waves data using convolutional autoencoders. Machine Learning: Science and Technology, 2(4)4045014. jul 2021. doi: 00.0888/6332-2153/abb2 2153/ab3330
URL https
f/áx. doi.ddi.... org/ 10.
[45] Eric A. Moreno, Bartlomiej Borzyszkowski, Maurizio Pierini, Jean-Roch Vlimant, and Maria Spiropulu. Source-agnostic gravitational-waan detection with recurrent autoencoders. Mach. Learn. Sci. Tech., 3(2)00200 2022. doi: 10.1088/2632-211 20503222-2233335553..3
[46] Ryan Raikman et al. GWAK: gravitational-wave anomalous knowledge with recurrent autoen- coders. Mach. Learn. Sei. Tech., 5(2):025020, 2024. doi: 10.1088/2632-2155/addaa..
[47] Eric M. Metodiev, Benjamin Nachman, and Jesse Thaler. Classification without labels: Learning from mixed samples in high energy physics. JHEP, 10:174, 2017. doi: 10.1007/ 1H1)10
[48] Raffaele Tito D' Agnolo and Andrea Wulzer. Learning New Physics from a Machine. Phys. Rev. D, (9( (115041, 2019. doi: 10. 1..110/hyssee.D..o.. D99.0.1.1999..99......
[49] Raffaele Tito D' Agnolo, Gaia Grosso, Maurizio Pierini, Andrea Wulzer, and Marco Zanetti. Learning multivariate physics. Fur. Phys. J. C, 81(1):89, 2021. doi: 10. 140/epjc/ s $10052--21-088533yy
[50] Gaia Grosso, Marco Letizia, Maurizio Pierini, and Andrea Wulzer. Goodness of fit by Neyman- Pearson testing. 2023.
[51] Raffaele Tito d' Agnolo, Gaia Grosso, Maurizio Pierini, Andrea Wulzer, and Marco Zanetti. Learning physics from an imperfect machine. Eur. Phys. J. C, 82(3):275, 2022. doi: (ttppj/ccss002000000 10052-02-- 10226-y.
10. 1 140/epjc/s
[52] Jerzy Neyman, Egon Sharpe Pearson, and Karl Pearson. Ix. on the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 231(694-706): 289-337, 1933, doi: 10. 1098/rsta. 1933. 0009.000 URL, htttpp::/royallsccittppppp org/ doi/ abs///ab/ttt 10. 1098 htt.//ttaaaaaa 1933 9009.0099
13

---

[53] Gaia Grosso and Marco Letizia. Multiple testing for signal-agnostic searches for new physics with machine learning. Eur. Phys. C. 85(1):4, 2025. doi: 10.11400epiccs100022000---111110
[54] J. Aasi et al. Advanced LIGO. Class. Quant. Grav. 32:074001, 2015. doi: 10. (0.108880264-938111 32/7/074001.
[55] R. Abbott et al. GWTC-3: Compact Binary Coalescences Observed by LIGO and Virgo during the Second Part of the Third Observing Run. Phys. Rev. X, 13(4):041039 2023, doi: 10. 10.1103/hhyRR=xx133.440000
[56]
LIGO Scientific. The O3a Data Release, April 2021. URL http:://01..rr//0...77 nfnt Afnt
[57] LIGO Scientific. The O3b Data Release, November 2021. URL https://doi.org/ 1935/pr1s j706
[58] Ethan Marx et al. Machine-leariing pipeline for real-time detection of gravitational waves from compact binary coalescences. Phys. Rev. D, 111(4):042010, 2025. doi: 10. 103/PhysseDD..
111.042010.
[59] Huilin Qu, Congqiao Li, and Sitian Qian. Jetclass: large-scale dataset for deep learning in jet physics, June 2022. URL https://doi.. org/ 10tgg/10.000 5ww.1221112.o... 6619768
[60] Huilin Qu, Conggiao Li, and Sitian Qian. Particle Transformer for Jet Tagging, 2 2022.
[61] Armen Tumasyan et al. A portrait of the Higgs boson by the CMS experiment ten years after the discovery. Nature, 607((917)x60-66 66, 2022. doi: 10.1038/s415866 022- 088--02200489-- [Erratum: Nature 623, (2023)].
[62] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[63] I. Zingman, B. Stierstofer, and F. Heinemann. Nafld pathology and healthy tissue samples, 2022. URL https: htttps://osff.ioogguut//
[64] M. Tan and Q. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. pages 1105-6114 14. International Conference Machine Learning, ICML, 2019.
[65] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
[66] Preetum Nakkiran, Behnam Neyshabur, and Hanie Sedghi. The deep bootstrap framework: Good online learners are good offline generalizers. arXiv preprint arXiv:2010.08127, 2020.
[67] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings the IEFE conference on computer vision and pattern recognition, pages 770-778. 2016.
[68] Herman Chernoff. On the distribution of the likelihood ratio. The Annals of Matheratiiaa Statistics, 25(3)5573-578 1954.
[69] Gregor Kasieczka et al. The LHC Olympics 2020 a community challenge for anomaly detection in high energy physics. Rept. Prog. Phys., 84(12):124201, 2021. doi: 10.1088/1361-66333 ac36b9.
[70] Ryan Raikman et al. A Neural Network-Bare Search for Unmodeled Transients in LIGO- Virgo-KAGRA's Third Observing Run. 12 2024.
[71] Vasileios Skliris, Michael R. K. Norman, and Patrick Sutton. Toward real-time detection of unmodeled gravitational wave transients using convolutional neural networks. Phys. Rev. D) 104004) 2024. doi: (0.1103//hysRevD..I1 10..00044....
[72] Georges Aad al. Weakly supervised anomaly detection for resonant new physics in the dijet final state using proton-protor collisions at /s 13 TeV with the ATLAS detector. 2 2025.
14

---

[73] Georges Aad et al. Dijet resonance search with weak supervision using / = 13 TeV pp collisions in the ATLAS detector. Phys. Rev. Lett., 125(13):13181, 2020. doi: 10.1103/ PhysRevI.ett. 125. 131801.
[74] Vladimir Chekhovsky et al. Model-agnostic search for dijet resonances with anomalous jet substructure in proton-proton collisions at vs 13 TeV. 12 2024.
[75] Fan Yang, Kai Wu, Shuyi Zhang, Guannan Jiang, Yong Liu, Feng Zheng, Wei Zhang, Chengjie Wang, and Long Zeng. Class-aware contrastive semi-supervised learning. 2022. URL https //arxiv..rr org/ abs 2203.2222222 02261
[76] Philip Harris, Jeffrey Krupa, Michael Kagan, Benedikt Maier, and Nathaniel Woodward. Resimulation-base self-supervised learning for pretraining physics foundation models. Phys. Rev. D, 111(3):032010, 2025. doi: 10.1 10.1103/hyyseeD..1 111.032010.
[77] Giacomo Mcanti, Luigi Carratino, Ernesto De Vito, and Lorenzo Rosasco. Efficient hyperpa- rameter tuning for large scale kernel ridge regression. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, 2022.
[78] Giacomo Meanti, Luigi Carratino, Lorenzo Rosasco, and Alessandro Rudi. Kernel methods through the roof: handling billions of points efficiently. In Advances in Neural Information Processing Systems 32, 2020.
[79] Sascha Husa, Sebastian Khan, Mark Hannam, Michael Pürrer, Frank Ohme, Xisco Jiménez Forteza, and Alejandro Bohé. Frequency-domale gravitational waves from nonprecessing black-hole binaries. i. new numerical waveforms and anatomy of the signal. Phys. Rev. D, 93:044006, Feb 2016. doi: 1...://yysee......440000 Utthstts: //ts:://lnnn aps org/ 0t..93.0040 044006
[80] Sebastian Khan, Sascha Husa, Mark Hannam, Frank Ohme, Michael Pürrer, Xisco Jiménez Forteza, and Alejandro Bohé. Frequency-domal gravitational waves from nonprecessing black-hole binaries. ii phenomenological model for the advanced detector era. Phys. Rev. D, 93:044007, Feb 2016. doi: 10. 10.1103/PhyseevD..o 93.044007. URL //tttt:s//innn aps org/ 1tt///0.110///yyaae
93. 044007
[81] LIGO Scientific Collaboration, Virgo Collaboration, and KAGRA Collaboration. LVK Algo- rithm Library LALSuite. Free software (GPL), 2018.
[82] Thibault Damour and Alexander Vilenkin. Gravitational wave bursts from cosmic strings.
Phys. Rev. Lett., 85:3761-3764, 2000. doi: 10. Le0
[83] Thibault Damour and Alexander Vilenkin. Gravitational wave bursts from cusps and kinks on cosmic strings. Phys. Rev. D, 64:064008, 2001. doi: 10. N0.110//hysseeDD64...66 www.seeeee...4..4400000 66.066008.
[84] Yuka Matsui and Sachiko Kuroyanagi. Gravitational wave background from kink-kink colli- sions on infinite cosmic strings. Phys. Rev. D, (00.1235155
2019. doi: 10. 103//hhyReeDDD
[85] B. Abbott et al. Search for gravitational-wav bursts in LIGO data from the fourth science run. Class. Quant. Gruv., 24:5343-5370, 2007. doi: 10. 111088/0226--9811255//009999 [Erratum: Mlass.Qunt.GGrrt 25, 039801 (2008)).
[86] Florent Robinet, Nicolas Arnaud, Nicolas Leroy, Andrew Lundgren, Duncan Macleod, and Jessica Mclver. Omicron: tool to characterize transient noise in gravitational-wave detectors. SoftwareX, 12:100620, 2020. doi: 10.10166j.sottx.222000066
[87] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URI httpps:://ppeereviee htt.:ppperreiiieeeeeeforrrrrr net/forum! id-Bkg6Ricqy7
[88] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.01001, 2017.
15

---

[89] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009. doi: 10.1 N0.1099/CVRR.220552266
[90] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 13:6840-6851, 2020.
[91] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, Augusi 23-28, 2020, Proceedings, Part V 16, pages 491-507. Springer, 2020.
[92] Anniina Mattila, Chris Jiggins, and Ian Warren. University of Helsinki butterfly collection Anniina Mattila bred specimens, February 2019. Ut///ttts////d.ooo//00055//////2 2555086
[93] Camilo Salazar, Gabriela Montejo- Kovacevich Chris Jiggins, Ian Warren, and Imogen Gavins. Camilo Salazar and Cambridge butterfly wing collection batch 1, May 2019. URL https
//doi org/ 10. 5281 2081/zeoooo.. .el.oo......0006....
[94] Ian Warren and Chris Jiggins. Miscellaneous Heliconius wing photographs (2001-2019) Part
2, February 2019. URL https: orgg /0/ 5281/zenodo 2553501
[95] Gabriela Montejo-Kovaceviee Quentin Paynter, and Amin Ghane. Heliconius erato cyrbia. Cook Islands (New Zealand) 2016, 2019, 2021, September 2021. URL https://://do org/
[96] Gabriela Kontejo-Kovacevice Eva van der Heijden, and Chris Jiggins. Cambridge butterfly collection GMK Broods Ikiam 2018, November 2020. URL https://doi. 10t.lii..rr/1..52222 5281/
zenodo 4291095
[97] Gabriela Montejo-Kovacet Kovacevich, Chris Jiggins, Ian Warren, Eva Wiltshire, and Imogen Gavins. Cambridge butterfly wing collection batch 9, May 2019. URL https://doi. 0rt.:oi.rrg11..22222.2 12.11.11.11
zenodo Genodo.2714333 2714333
[98] Gabriela Montejo-Kove Kovacevch.. Chris Jiggins, Ian Warren, and Eva Wiltshire. Cambridge butterfly wing collection batch May 2019. URL https://adl.oor//00.555///eee
2707828
[99] Erika Pinheiro de Castro. Christopher Jiggins, Karina Lucas da Victor Lucei Freitas, Marcio Zikan Cardoso, Eva Van Der Heijden, Joana Meier, and Ian Warren. Brazilian Butterflies Collected December 2020 to January 2021, February 2022. URL
Andre
attps://zenodo.. wtt.://..rro//eeerer/r5es55
[100] Patricio Salazar, Gabriela Montejo-Kovaceviide Ian Warren, and Chris Jiggins. Cambridge butterfly wing collection Patricio Salazar PhD wild and bred specimens batch 2, January 2019. htt. http:///....rrr/...525////////// 0tw.://...2222eeedee 2548678
[101] Gabriela Kontejo-Kovacevice Chris Jiggins, Ian Warren, and Eva Wiltshire. Cambridge butterfly wing collection batch 7, May 2019. URL https://ddoo....... 222.0/220oo...11000.2222222oooo 2702457
5281/zenodo
[102] Patricio Salazar, Gabriela Nontejo-Kovaceviel Ian Warren, and Chris Jiggins. Cambridge butterfly wing collection Patricio Salazar PhD) wild and bred specimens batch 1, December 2018. URI htRlssttss:/// //doi. 10. 5281/2enooo...... 0t0/://.......///oooooo.oo..o 1748277
[103] Gabriela
Chris Jiggins, Ian Warren, Camilo Salazar, Marianne Elias, Imogen Gavins, Eva Wiltshire, Stephen Montgomery, and Owen McMillan. Cambridge and collaborators butterfly wing collection batch 10, May 2019. URL http:http:///ett org/10
5281/zenodo. 2813153
2813115
[104] Gabriela Montejo-Kovacevve Chris Jiggins, and Ian Warren. Cambridge butterfly wing collec- tion batch 1- version 2, May 2019. URL https: :/oooi org 5281//enoo.. 3082688
16

---

[105] Joana I. Meier, Patricio Salazar. Gabriela Kovacvic,,,,,,, Ian Warren, and Chris Iggins. Cambridge butterfly wing collection Patricio Salazar PhD wild specimens batch 3, October 2020. UtI.https::/ org/ wttp://rrr/....8//eeeeooooo
4153502.
[106] Chris Jiggins and Ian Warren. Cambridge butterfly wing collection Chris Jiggins 2001/2 broods batch January 2019. UtL_pcpge///a...ogg/0000//////12 022.:///oo.00//oood...o. 2550099
[107] Chris Jiggins and Ian Warren, Cambridge butterffy wing collection Chris Jiggins 2001/2 broods batch 1, January 2019.
2549524
[108] Gabriela Montejo-Kovacevich Chris Jiggins, lan Warren, and Eva Wiltshire. Cambridge butterfly wing collection batch 6, May 2019. URL htt.:ppp//aa...or///00.5///////2 2686762
[109] lan Warren and Chris Jiggins. Miscellaneous Heliconius wing photographs (2001-2019) Part
3, February 2019. URL https:s::///oi.oorr 10. w2w.00000002.0......... 2553977
[110] Ian Warren and Chris Jiggins. Miscellancoous Heliconius wing photographs (2001-2019) Part
1, February 2019. URL https: 10. 5081/2220000 2552371
[111] Gabriela
Chris Jiggins, Ian Warren, and Eva Wiltshire. Cambridge butterfly wing collection batch 5, May 2019. URL 2684906
[112] Chris Jiggins, Gabriela
Ian Warren, and Eva Wiltshire. Cambridge butterfly wing collection batch 3. May 2019. URL 2682458
[113] Gabriela Montejo- Kovacevich, Chris Jiggins, and Ian Warren. Cambridge butterfly wing collection batch 2, May 2019. ttt.ttttss//doo..om http://orr.....2.///.......0070.
[114] Patricio A. Salazar, Nicola Nadeau, Gabriela Moonneej-Kooacevidde and Chris Jiggins. Sheffield butterfly wing collection Patricio Salazar, Nicola Nadeau, Ikiam broods batch and November 2020. URL hattps:///o/ org 1tt.:/..281111oeo 4288311
[115] Gabriela Montejo Montejo-Kovacevichh Eva van der Heijden, Nicola Nadeau, and Chris Jiggins. Cam- bridge butterflyy wing collection batch 10, November 2020. URL / 5281/zenodo.42880
4289223
[116] Samuel Stevens, Jiaman Wu, Matthew J. Thompson, Elizabeth G.Campolongo, Chan Hee Song, David Edward Carlyn, Li Dong, Wasila M. Dahdul, Charles Stewart, Tanya Werger-Wolf, Wei-Lun Chao, and Yu Su. Bioclip: A vision foundation model for the tree of life. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), page 19412-19424, 2024.
[117] DC Dowson and BV666017 Landau. The fréchet distance between multivariate normal distributions. Journal of multivariate analysis, 1203):450-455, 1982.
[118) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[119] Antoine Chatalic, Marco Letizia, Nicolas Schreuder, and Lorenzo Rosasco. An efficient permutation-based kernel two-sample test. arXiv preprint arXiv:2502. 13570, 2025.
[120] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize imagenet? In International conference machine learning, pages 5389-5400 5400. PMLR, 2019.
[121] Feng Liu, Wenkai Xu, Jie Lu, Guangquan Zhang, Arthur Gretton, and Danica Sutherland. Learning deep kernels for non-parametric two-sample tests. In International conference machine learning, pages 6316-6326. PMLR, 2020.
[122] Serguei Chatrchyan et al. Observation of a New Boson at a Mass of 125 GeV with the CMS Experiment at the LHC. Phys. Lett. B, 716:30-61, 2012. doi: 10. 00.10166.phyxeeeb2222......
17

---

[123] Georges Aad et al. Observation of new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC. Phys. Lett. B, 716:1-29, 2012. doi: 10. ph0t://..pyyseeeh2112.......
[124] Cms open data. URL https:
ch/
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction claim that we have introduced a new pipeline for novelty detection in scientific datasets, which is described in Sec. [ We claim strong sensitivity to small injections of novel signals into background-ong datasets, which is supported by the experiments (Sec. 5
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We discuss how our method is better suited finding distributional anomalies via GoF, rather than finding single point anomalies. We also discuss how domain shift could be tackled in future work.
Justification: We directly address limitations in the conclusion, Sec. 6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
19

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: We do not derive any novel theoretical results. All statistical theory we discuss related to NPLM is established in prior work, which is cited.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: In sections and we describe detail the contrastive learning setup we use, the statistical hypothesis testing framework, and the experiments we run. Most datasets we use are open source, with the only exception being synthetic dataset that we describe how to generate. We describe the datasets, model architectures, and training setups in detail in App. A
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
20

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Our repository is publicly hosted on GitHub, and will write detailed instructions about running code and reproducing experiments in the README. We will include the GitHub link the final version of this paper, after double-blind review.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips.cr
Seallic/giies//oooe /wode/Sumiissiioool11111..
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
polpliscc/uullicguudddeses setpredeeee/ooeSeeeeeeeeeeeeeeee wttpstubbiissiiiiii www.iioiiioiooooooooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: These discussed for each dataset in Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The foundation of our results are carefully considered and statistically rigorous, as this paper's main focus scientific hypothesis testing. This is discussed in great detail in Secs. B and
21

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This is discussed in App. A
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: ttttpps://eeuriii wwwwrrp...cppuulllpeee hcc/ppblicc/tthhccccieeee.l....
Answer: [Yes]
Justification: Our work is focused on hypothesis testing and discovery in large scientific datasets. We do not anticipate any potential harms caused by our research or as result of
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
22

---

Answer: [NA]
Justification: Our paper is about scientific hypothesis testing and discovery existing datasets. We do not anticipate any immediate societal impact of finding outliers in large datasets from e.g. physics and astronomy, aside from extremely broad-scope and minor potential impacts related to uncovering new understanding of the universe. In bio-sciences context, this could potentially lead to some new medical understanding or discovery, but it is nearly impossible to accurately forecast the occurrence or impact of such discoveries. Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We use standard, publicly available datasets and machine learning architectures. Our pipeline is designed be run on scientific datasets. We anticipate irresponsible use of our code or data.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-us should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
23

---

We use and cite open-source datasets and code from open source machine learning libraries, and plan make all of our code public.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pappesswi thcode danasatss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We release new datasets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We use no crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

---

Answer:
[NA]
Justification:
We do not do any studies inolving human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: Our LLM usage is limited to editing the written words in this draft.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM for what should or should not be described.
25

---

Dataset
[
[2]
fs range (%)
Synthetic
10,000
2,000
0.5 10
Astronomy
20,000
4,000
0.5 10
Particle Physics
50,000
10,000
0.1
Histology
6,296
500
-
Images (CIFAR-10)
9,000
1,800
1-10
Genomies
1,764
100
-0
Table I: The reference dataset size, observed dataset size, and range of injected anomaly fractions (relative to the background component in D) for each dataset considered in our study. These parameters are used when running NPLM pseudo-experiments across a range of signal fractions to produce the results shown in Fig.
A Dataset, Training, and Evaluation Details
All of the experiments presented in this paper were run on an academic computing cluster. The contrastive trainings were run on a single NVIDIA A100 GPU in all cases, and none took more than a few hours to compute. The kernel-based NPLM tests used the GPU-accelerated Falkon package 77 78 and also ran on single GPU, with a typical set of 100 toys with [R] 10, 000 and [D] 2, 000 taking 10-20 minutes. All other metrics such as the Mahalanobis test were computed on CPU nodes and were not a significant computational bottleneck.
Table summarizes the reference dataset sizes |R| observed dataset sizes [D] and signal fractions fs used in the experiments presented in Sec. 5 These numbers typically limited by the test set sizes for each dataset, and by requirement that [R] be significantly larger than [D] for NPLM. We fix the ratio at | 5| except for histology and genomics where the datasets are very small.
As mentioned in Sec. 3.2, the kernel size a is configurable hyper-parameter performance varies somewhat as the kernel width changes. In practice, all NPLM pseudo-experiment are run with six different variations of the kernel width o = [0.1, 1.5, 2.6, 3.6, 4.9, 9.8). The four dimensional imput data standardized according the mean and standard deviation of the reference sample R, so these widths refer to a common scale. The smallest-width kernels are best at adapting capture excesses or
NPLM, and the
small, local features and distortions in the data, while the widest outlier far in the tails away from the bulk background distribution. Fig. 3 of the main text we present the asymptotic and empirical NPLM Z scores corresponding to the best-performing kernel width, but we also show average empirical Z of all six kernels in black. We plot full results for all kernel widths, both asymptotic and empirical Z scores, App.
A.1 Synthetic Data
The synthetic dataset aims to broadly look at challenging datasets that are largely overlapping and nigh-dimensiooal. As part of that, insisted series of core elements to ensure robust construction. Namely, the chosen mixture Gaussians
signals are fully reproducible,
the minimum pairwise optimized significance between clusters was 3.5 standard deviations; this was computed through the computation of cumulatíve distribution about the mean of the Gaussian, assuming scenario of percent background in a sample size of 10000 events.
All discriminating variables were randomly mixed among non-discriminating variables
Gaussian means and sigmas are bounded in ranges [0. 1] and [0.02, 0.5). respectively.
For each dataset, we generated 10k events each of the data classes. Datasets ranging from separate classes 20 were generated, along with additional random variables ranging from 0 to 30 variables. Additionally, to address the variation of models, we generated roughly 100 separate random models for each point. The total number of models utilized is 3870.
With each model, two trainings are performed, using a simple 4-layer MLP with separate MI.P classifier with no output activation and projector to compute the contrastive loss; there a total
26

---

Synthetic Dataset
Synthetic Dataset
NN Mahalanobis
NN Mabalanobis
Raw Mahalanobis
Raw Supervised
Z Score (a)
Raw Supervised
Score (a)
Supervised
Supervised
Ideal Supervised
Ideal Supervised
2
8
10
12
14 16
18
10 15
25
Number Signals
Number of Additional Dim
Figure 4: Z-score detection statistic for an anomaly using Mahalanobis distance, and supervised learning on the raw and learned embedded space. (a) We observe the impact of additional signals on sensitivity to find an anomaly. (b) We observe the change sensitivity vs number of additional random variables obtained from the average of 100 toys, adding an additional signal model and discriminating variables with each increment on the x-axis. The green corresponds to Mahalanobis distance on raw inputs, the orange corresponds to fully supervised algorithm on raw inputs, and the red corresponds to a supervised algorithm trained on the ideal contrastive space, where the signal is known. The blue shows the result of supervised training (knowing signal) contrastive space where the signal is unknown; the brown shows the result the Mahalanobis distance on the same space.
of 12k trainable parameters. The loss function used was SimCLR with r = 0.01 and ACE = 0.5, learning rate 0.001 with batch size of 1000 used along with a cosine annealing. Trainings CPU. Similarly, for the supervised
are performed over 50 epochs and take roughly 5 minutes algorithms, 4-layer MI.P of roughly the same size was utilized.
Figure 4 (a) presents the result of scanning over the toy models, computing the asymptotic Mahalanobis distance for the embedded space, the raw space, and applying supervised algorithm on both, and with an additional supervised algorithm on the embedded space trained with the hidden signal. The left plot adds an additional signal and an additional diseriminating dimension with each variable. Here, we observe that at least four signals are needed to span the space, and high sensitivity is observed, which gradually goes down the confusion and density from so many signals make hard for specific point to separate itself. The right plot shows the impact of additional random dimensions on the data. We observed that either embedding or supervised algorithm sufficient to overcome loss of sensitivity present from just adding random, fluctuating dimensions.
A.2 Astronomy
We utilize the AutoSeiDACT pipeline to identify anomalous gravitational wave sources. Our dataset comprises time-series recorded by the two advanced LIGO detectors [54] Hanford, Washington, and Livingston, Louisiana- spanning the third observing run (O3) from April 2019 to March 2020;
this data publicly available [56] (7). Our data preparation and labeling process closely follows the setup described in [46] [70]. Class-balanced reference sets are constructed by injecting simulated signals into real background; the background class is simply segments with injections. We inject compact-binary coalescences including binary black hole mergers from phenomenological model IMRPhenomPv2 (79 80) (sine-) Gaussian signals (81), signals from cusps
(82). kinks (83),
double kink events 184, and signals from band-limited white noise bursts (WNBs) 85 Moreover, a dedicated "glitch" class is obtained using Omicron's 's criteria [86).
Each input consists of two sequences, one for each detector, with each sequence containing 200 points, corresponding to 50 ms long time series sampled at 4096 Hz. Exemplary time-series can be
27

---

BBH
WhiteNoiseBurst
normalized elongation
Livingston
normalized elongation
2
Hanford
-1
2
Livingston
-2
Hanford
10
20
30
40
50
0
10
20
30
40
50
time [ms]
time [ms]
Figure 5: Example LIGO signal waveforms: Signals from binary black holes (left) and white noise burst (right).
found in Fig. 5 The dataset comprises total of about 530,000 samples, with an near-uniform class balance in the pre-training. To enhance data processing and support network training, the data are normalized to have a unit standard deviation on a per-sample basis. Of the nine total classes, the WNB class is withheld from pre-training and treated as the held-out anomaly for discovery. White-noise bursts are deliberately model-agnostic: they represent correlated, band-limited stochastic fluctuations whose spectra are flat over the analysis band. As such they emulate the "worsttcasse burst burst-lacking distinctive phase evolution or chirp structure, providing a stringent test of the pipeline's capacity to disentangle subtle, structure- poor signals from detector noise.
The available data split into a training, validation and test dataset with the test dataset not only utilized for testing the pre-training performance, but also for constructing the reference RR and data distribution ID for the hypothesis test.
The optimization of the backbone encoder fa one-dimensional ResNet with about 7.2M trainable weights uses the combined loss objective (SimCLR temperature Adam W optimizer [87] with an initial learning rate of 0.001 and 350 batch size. To facilitate improved convergence and generalization, a cosine annealing learning rate schedule employed. The training is set up for maximum of 25 epochs, with early stopping in case the validation loss does not decrease for more than five epochs.
0.5, ACE = 0.5) and the
A.3 Particle Physics
JetClas is an open- open-source particle physics dataset introduced in [60]. The training set consists of
100M jets from 10 classes (10M per class), of which we use five for total of 50M training samples: QCD (quark/gluon), top quark (I - byg'), ) W boson (W - qg') ), boson (Z - qq) and Higgs (H + bD). The validation set consists of 500,000 jets per class and is used during training to monitor performance. The test set consists 2M jets per class and is used to construct reference and data samples for all NPLM hypothesis tests presented in the main text.
For the encoder, we use the particle transformer architecture nearly exactly as described in 60 using a particle embedding of dimension 128, eight self self-attention layers with eight heads, and two class-attention layers with the final 4-dimensional embedding derived from the final class token plus a fully connected layer. We use the same 17 per-parne per-particle input features described in [60], including information on the particle's energy/monentumm, trajectory, and particle type. We cap the input size at 64 particles per jet. We train the encoder with SimCLR temperature = 0.1 and a classifier strength of ÀcE = 0.1, running for 100 epochs with an initial learning rate x 10-4 annealed to 10 cosine schedule and using the AdamW optimizer (88). We use a batch size of 512.
28

---

heart
liver
NAFLD
50
50
50
pixel index
plxe Index
pixel index
100
100
100
150
150
150
200
200
100
200
100
200
100
200
pixel Index
pikel Index
pixel Index
Figure 6: From left to right: exemplary image patches of heart tissue, liver tissue and liver tissue with non-alcoholic fat Non-Alcoholic Fatty Liver Disease (NAFLD) 63
A.4 Histology
Exemplary image patches from the publicly available histological dataset 663 are shown in Fig. The training dataset balanced and includes samples from mouse tissues the brain, heart, kidney, liver, lung, pancreas, and spleen and a separate class containing normal liver tissue samples from rats. Each class comprises approximately 6,300 samples. The dataset is split into training, validation, and testing sets with a ratio of 70% In addition, independent test set consists of approximately 2,300 samples normal mouse liver tissue and an equal number of samples with non-alcoholic fatty liver disease (NAFLD).
Due to statistical constraints, the reference distribution is constructed from the training data, with prior validation to ensure the absence of overfitting. The data distribution 'D for normal tissue is derived from both the training set and the independent dataset containing only healthy mouse liver samples.
The backbone encoder fa. EfficientNct-B0 64 with approximately 4.8M trainable parameters, is optimized using combined loss objective. incorporating SimCLR contrastive loss (temperature 0.5) and cross-entropy loss with weighting ÀCE 0.5. Optimization is performed with the Adam W optimizer, using an initial learning rate of 0.001 and a batch size of 32. To promote stable convergence and improved generalization, a cosine annealing learning rate schedule is employed. Training is conducted for a maximum of 25 epochs, with early stopping triggered if the validation loss fails improve for more than five consecutive epochs. Each training run takes between one and two hours single NVIDIA A A100 GPU.
A.5 Images
We use the CIFAR-10 665
dataset, applying the standard resize to 232 232 with interpolation,
crop to 224 x 224, and standardizing using the ImageNet [89] mean and standard deviation. For the encoder, we use the fytorch-providee connected layer with an MLP of hidden dimensions [512, 256, 128] and output dimension of 4. Only these final MLP weights are floated during training.
pre-trained 5e 667 weights and replace the final fully
We use the 50,000 CIFAR- 10 training images to pre-train the encoder, using only 45,000 in practice because class 1 is held out as the anomaly. When evaluating with NPLM we introduce 100,000 images from CIFAR- -mm [66] in order to boost the number of points available for demonstrating our method. The approximately million images in CIFAR-5m were generated by an unconditional denoising diffusion probabilistic model (DDPM) 190] trained 98.5% accurate Big-Trie Tia-srrnsfer model (9I.. Trainings took about hour on single A A100 GPU, and ran for 50 epochs with a learning rate of 10 annealed to 10
C1FAR-10, then labeled by the
cosine schedule and with batch
size 512. The SimCLR temperature set to T = 0.1 and the cross-entropy strength to ÀCE = 0.5.
htttp::/eenodoo.rrgreeetttt//s/
29

---

Figure 7: From left to right: Exemplary images of two different butterfly subspecies (left, center) and a hybrid offspring of those species (right).
A.6 Genomics
As an example from genomics, we evaluate our approach on the dataset provided in the "Butterfly Hybrid Detection" challenge Butterflies come different subspecies characterized by visual differences in color and pattern the wings and sometimes mix and produce offspring. We aim to detect these anomalies, so called hybrids, with AutoSciDACT.
The dataset consists of 1991 colour images of size 224 x 224 pixels, annotated to belong to one of 14 classes, with a highly imbalanced class distribution (some classes contain almost 100 times more instances than others) [15]. Example images of two different species and hyrbid are shown in 7 We adopt a fixed split of 80% training, 10% validation and 10% test sets. Our model uses a backbone encoder based on the BioClip architecture (16), with approximately 430k trainable weights. Training employs the AutoSciDACT combined loss: a contrastive self self-suuervised term following the SimCLR formulation with temperature parameter
0.5, and a supervised cross-entropy term, weighted with ÅCE = 0.5. Optimization is performed using the Adam optimizer at a learning rate of 0.001 and batch size of 32. We apply cosine annealing learning-rate scheduling over 100 epochs and minimal value 0.0001. Training is terminated by early stopping when no improvement is observed for five consecutive epochs; in our experiments this is the case after 43 epochs.
The contrastive embedding resulting from the projection of image data into four- niu--dimessionl latent space is shown in Fig. [ (left). Among all experiments, the anomaly in the genomics case shows the clearest separation from the background classes. The performance of AutoSciDACT in this setting is presented in Fig. (right), which displays the statistical significance (Z-scores) obtained by NPLM and various baseline methods for detecting varying proportions of hybrid butterfly species images injected in background-dominntce samples. We omit asymptotic results, since the underlying assumption that the test statistic is x" -ddittributeed, is invalid in light of the limited statistics of the dataset. The limited statistics of this dataset is also the reason why it is not included in the main results. Notably, the empirical Mahalanobis distance baseline achieves the strongest performance due to the excellent separation of the anomalous class in the contrastive embedding space. Conversely, the Maximum Mean Discrepancy (MMD) test yields the weakest performance, potentially due to suboptimal kerell width choice (see Appendix B.2 for further discussion).
B Additional Experiments
B.1
Impact of NPLM kernel width
The choice of NPLM kernel width has significant impact on the sensitivity of the test, and there is no a priori choice that can optimize sensitivity to potentially anomalous features. The results in Fig. 3 included 7-scores from the best-performing kernel and the average over all kernel widths considered. For completeness, we plot Z-scores from all kernel width settings in Figures 9 (asymptotic) and I0
(empirical). These figures exactly mirror Fig. 3 displaying results for each of the five datasets in the five panels (a)-(e). In Fig. 10 we also show the kernel-averaged Z-score in black. In general, intermediate to larger kernels do the best, with only fairly small variations among the top performers. The empirical Z-scores cannot exceed roughly 2.88a due to the number of pseudo-experiments (500).
30

---

Genomics, Hybrid butterflies anomaly
Empirical NPLM
Mahalandbis (Empirical)
3.5
(Empirical)
3.0
Sccore (o)
Max Zeep with Nags
2.0
N
1.5
0.5
4
10
16
20
Injected Anomaly Fraction (%)
Figure 8: Results for the genomies experiment: Contrastive embeddings in four dimensions (left).
Background classes are shown in hues of blue and green, while the anomaly is overlaid in hot pink. Results of the statistical tests (right).
B.2
Comparisons with additional baselines
We further contextualize AutoSciDACT's performance by comparing with two additional anomaly detection baselines: Maximum Mean Discrepancy (MMD) 22 and Fréchet Inception Distance (FID)
[18].
Nyström approximated Maximum Mean Discrepancy (N-MMD). The Maximum Mean Discrep- ancy (MMD) is a kernel-based statistical test used to determine whether two datasets come from the same distribution [29]] works by mapping the data into a high- nigh-dimmnsional reproducing kernel Hilbert space (RKHS) via chosen kernel (c.g. Gaussian) and computing the distance between the mean embeddings of the two distributions in that space. MMD has been shown be sensitive to subtle differences between distributions, making it particularly effective in high-dimensional settings. However, its computational cost, quadratic in the number of examples, limits applicability small sample sizes or low- oow-dimensional problems. To address this, [119] introduced a scalable variant of the MMD test based on a Nyström approximation of the kernel matrix, enabling its use on larger datasets while maintaining statistical power. Since the NPLM test employed in this work is also built upon the Nyström approximation, we perform a direct comparison between Nyström- wystrmmMMD and NPLM under matched settings, using the number of centroids and same kernel bandwidth.
Fréchet distance (FD). The Fréchet distance has emerged as popular metric for comparing probability distributions, particularly in the context of generative modeling [117 [18]. Under the assumption that both distributions are Gaussian, it admits a closed-form expression involving only their means and covariances, making it computationally efficient and interpretable. However, this assumption can limit its effectiveness when the underlying distributions exhibit significant non- Gaussian behavior, such as heavy tails or multimodality. Despite this, the Fréchet distance remains widely used due to its robustness high-dimensional settings and its ability to capture both mean and covariance differences.
In Fig II we reproduce Fig. with empirical MMD and FID Z-scores included for the particle physics, astronomy, histology, and image datasets. NPLM outperforms FID/MMD for the particle physics and image datasets, approximately matches them for astronomy, and, surprisingly, underperforms in histology. This wide range of outcomes makes it difficult to draw unambiguous conclusions, but broadly suggests that the "best" anomaly detection method depends strongly on the structure of the embedding space and the size of the dataset at hand. NPLM's sensitivity scales with available sample size, and the histology dataset has by far the smallest available test datasets among all experiments in the main body of the paper (see Ta In such data-constrained cases, other methods may perform better as they do not require likelihood ratio estimation. In other cases, the structure of the data embeddings may also confer advantage (e.g. if clusters are approximately Gaussian). However,
31

---

LIGO, WNB Anomaly
Synthetic Dataset
jetciss. H6 anomaly
Rur Ro
Row -aa
Pur 00
Bua 925
Our 4po
Due
Por the
- Pun -20
Asymptetic z Sccrre (o)
qio
Our
Moymgtotiic 2.50cre (o)
Noymphotic Score
bjected Anomaly fraction (%)
bijected Aromaly Fraction (84
02
Injected An Anomaly Fraction (%)
(a)
(b)
(c)
Histoloyy, Liver disease anomaly
CIFAR, Class anomally
10
<lar- Ois the
16
OL
sha- D
Our
2it
10
2:00
Asymplotie mommtettic Z Score C
Asymploticc z Score
25
23
160 125
125
26.
23
56 15 360
L25 35. 260
Injected Aromayy Traction 196)
Injected Anomaly Traction
(c)
Figure 9: Asymptotic NPLM Z scores function of injected signal yield for all six kernel choices [0.1, 1.5, 2.6, 3.6, 4.9, 9.8) used in pseudo experiments. We show results for our five benchmark datasets: Synthetic (a), JetClass (b), LIGO (c), Histology (d), and CIFAR-10 (e).
NPLM's strong performance in all but the most data-constrained cases positions it as a reliable and competitive choice.
B.3 Varying embedding dimension
use a very low contrastive embedding dimension (d = 4) for the main results this paper. This choice was made to ensure the tractability of our statistical anomaly detection technique (NPLM), which relies on statistical hypothesis testing that can quickly lose sensitivity in high dimensions. In this section explore the impact of increasing the embedding dimension, extending experiments up to = 3 Figure [2] shows the Z-score as a function
for the CIFAR, JetClass, and LIGO datasets, where the best-performing kernel widths used and the signal injection fractions are set such that NPLM has good but not fully-saturated sensitivity at the default setting d = 4.
There are no unambiguous trends for any of the anomaly detection methods, with all methods performing relatively stably up to d = 32. NPLM's sensitivity declines very modestly the CIFAR and JetClass examples, but slightly improves in the LIGO example. The MMD and Fréchet metrics are similarly stable. Interestingly, the Mahalanobis distance appears to almost always benefit from larger dimensionality. This could be explained by class-specific clusters having more "room" to spread out and condense under the contrastive objective in higher-dimen dinherr-imessionall space. As Mahalanobis distance is sensitive to the distribution of these clusters, this would likely improve performance (assuming that the anomalous cluster separated from the rest, i.c not an overdensity in an existing cluster). The absence of clear trends for NPLM is encouraging, suggesting AutoSciDACT could be applied to problems requiring digher-dimensional latent spaces. Where possible, however, we vouch for keeping the dimensionality low this is beneficial for classical statistical analysis and uncertainty quantification.
"This is still modest relative to standard embedding sizes in computer vision or natural language, but reasonable choice for many scientific applications.
32

---

Synthelic Dataset
JetCisss .b6 anemaly
UGO WNN aromaly
Gur 0.
On
Our
-0
On
Gar Ga
due 200e
dur 20ha
Guy the
Average
-4- dur th
Average
day O.
Mar Ri
Empisiall Z S.Sorre dodl
Empinical 2 Grt
Empincall Scoore
22-
2.0
15
66
Injected Anomaly Fraction 196)
Aramaly
Irjected Anomaly
(a)
(b)
(c)
Histology, Liver disease ancomaly
CIFAR, anomaly
Gar
3.5
2.0
Empincal z Score (o)
Empiniaal 22 Score
0.
Injected Anomaly traction (%)
Injected Anoma Fraction (%)
(d)
Figure 10: Empirical NPLM Z scores as a function of injected signal yield for all six kernel choices a 1.5, 2.6, 3.6, 4.9, 9.8] used in pseudo experiments. We show results for our five benchmark datasets: Synthetic (a), JetClass (b), LIGO (c). Histology (d), and CIFAR-10 (e). We also show the average empirical Z score from all six kernels in black.
B.4 Noisy labels
Many real-world scientific datasets are labeled by hand or by algorithms with known error rates, both of which result in some fraction of mislabeled training data. To assess our sensitivity to such cases, we train variants of our JetClass contrastive embedding model with 1, 2, 5, and 10% label noise, meaning 2% of samples the training datasets are randomly mislabeled. The results are shown in Fig. [3 where again the best-performing kernel width is chosen and the injected signal fraction is set such that NPLM has good performance at zero noise. As expected, sensitivity drops as label noise increased, falling from 4o to about 20 at 10% noise.
C Comparison to previous results from histology
Our setup for the histology study makes use of the same dataset as in 225 In 25 the authors proposed a contrastive learning method for anomaly detection. The learned embedding returns a 320-dimensional representation, and a standard one-class SVM with the Radial Basis Function (RBF) kernel and margin error / 0.1 is then trained on anomaly-free data to construct an anomaly score. Applying a threshold that ensures zero false positives on the training set, the tiles constituting data sample are tagged as either standard or anomalous. The final anomaly metric used for comparisons is an average across the tags of all tiles in the sample. This statistical test assumes the anomaly is out-of-distribuu and the distribution of the background class in the chosen representation is well clustered. While these assumptions are met in (25), out-of -f-triiiibittiinnn is not ensured for arbitrary representation space agnostic to the anomaly source, which motivates our choice for using NPLM as universal statistical test.
Since 25 does not present results in terms of p-value, we implement the one-class SVM trained on our four-dimensional embedding space for comparison purposes. We find that the performance
33

---

LIGO, WNB anomaly
JetClass, bò anomaly
10
Asymptotic NPLM 24sa)
Asymptotic NPLM (d,a 99p)
Empirical NPLM (dur 4sa)
Empirical NPLM (Over 20sl
Empirical NPLM (Mg kernels)
Empirical NPLM (Avg kernels)
Mahalanotis (Empinical)
Mahalanobis (Empirical)
Supervised
MMD (Empirical)
Supervised
Frechet (Empirical)
MMD (Empirical)
Supervised
Z Score (d)
(Empirical)
Ideal Supervisge
Z Score
Max Zamp with Nans 500
Max Zamp with Mays 500
0.25
0.50
1.00
1.50
Injected Anomaly Fraction (%)
Injected Anomaly Fraction (%)
(b)
Histology. Liver disease anomaly
CIFAR, Class anomaly
10
10
Asymptotic NPLM (Owr 9sal
Asymptotic NPLM (Oue = Qoo)
Empirical NPLM (Øtr qsal
Empirical NPLM (Ooe qua)
Empirical (Avg kernels)
Empirical NPLM (Avg kernels)
Mahalanobis (Empirical)
Mahalanobis (Empirical)
Supervised
Supervised
Ideal Supervised
Ideal Supervised
Z Score (d)
MMD (Empirical)
Z Score (c)
MMD (Empirical)
Frechet (Empirical)
Frechet (Empirica)
Max Zomp with Nuys 500
Max Zams With 500
2.5
5.0
7.5
10.0
12.5
15.0
17.5 20.0
Injected Anomaly Fraction (%)
Injected Anomaly Fraction (%)
(c)
(d)
Figure 11: reproduction of Fig. including the Maximum Mean Discrepancy (MMD) and Fréchet Inception Distance (FID) baselines for the JetClass (a), LIGO (b), Histology (c), and CIFAR-10 (d) datasets.
34

---

CIFAR, 5.0% Signal Injection,
Jefciass, 1.0% Signal Injection,
LIGO, 4.0% Signal Injection, 24px
Anymprat Uhur dul
Sopacel LPE Sianeeed
PMO Eiginoal
qui
dng foll Mill Pua
Natalanctia [trpe
Siprabed
NMG [Emoi
moceet ltrrcn
Scane (c)
Scare
Contrastive Embedding Dimension
Contrastive Embedding Dimension
Contrastive Embedding Dimension
(a)
(b)
(c)
Figure 12: score as a function of embedding dimension for NPLM and baseline anomaly detection
methods in the CIFAR-10 (a), JetClass (b), and LIGO (c) datasets. The best-performing kernel widths are chosen for presentation, and signal injection fractions are set such that NPLM has good but not fully-saturated sensitivity at the d 4 baseline.
JetClass, 1.0% Signal Injection, Oker 950
Asymptotic NPLM (Øker - Qsp)
Empirical NPLM (dee - Qs0)
Mahalanobis (Empirical)
Supervised
Supervised
10-
MMD (Empirical)
Frechet (Empirical)
Score
10
Training Label Noise (%)
Figure 13: Z score as a function of label noise (%) dimension for NPLM and baseline anomaly detection methods for the JetClass dataset. The best-performing kernel widths are chosen for presentation, and signal injection fractions are set such that NPLM has good but not fully-saturated sensitivity at the zero-noise baseline.
of the test highly depends on the threshold set on the one-class output score. With a threshold that allows for 10% false positive rate, the one-class SVM performs comparably to the NPLM version. The one-class SVM has no discriminative power when the threshold set to 1% false positive rate, corresponding to
subtle anomalous contribution. The reason for such variance is the fact that the latent representation, the anomaly does not necessarily lie outside the distribution. Additional tests run the astronomy data (where the signal highly overlaps with one of the background classes) show an even more striking example of detection failure.
In conclusion, these studies reassure our strategy of using the NPLM to compute two-sample test rather than focusing on out-of-distribution detection because two-sample tests are sensitive to wider range of anomalous behavior. This is particularly relevant for the histology data since more than one anomalous tile expected per sample, and capturing collective behaviors enables the detection of more subtle anomalies, e.g., detect diseases manifesting in tissuc earlier.
35

---

D Discriminating CIFAR-10.1 and CIFAR-5m
To further demonstrate the capabilities of AutoSciDACT for detecting distributional shifts, we apply to the problem of quantifying the difference between CIFAR- 10 and the synthetic CIFAR- 5m dataset [66], as well as the independently curated CIFAR-10.1 dataset [20]] CIFAR-5m was discussed the main text and App.AA and consists of approximately 5 million images generated by a diffusion model trained on CIFAR- 10. CIFAR-10. was introduced in [200, where it was specifically collected and curated to be as close to CIFAR- 10 as possible (e-g. drawing from similar sources of images). The idea was to test whether classifiers trained CIFAR- 10 readily generalized to data outside of CIFAR- 10 but in principle distributionally identical. It was found that performance dropped substantially when evaluating
CIFAR-IO. 10, indicating some non-trivial shift in the dataeset. Recent work in Liu et. al. [121 and Bacllee et. al. [3] has applied some versions of two-sample tests to distinguish CIFAR-10 and CIFAR-10.1, both showing strong evidence of distributional shift between the two.
In Fig. 14 we tackle this question with AutoSciDACT. We train four-dimensional contrastive embedding on the full CIFAR-10 training dataset without holding out any classes, as individual classes are not considered anomalous in this context. We use the same architecture and training procedure as for the CIFAR-I0 results in the main text. We use this encoder to embed the CIFAR- 10m test set, the C1.AR-10.1 set, and 100,000 randomly selected images from the CIFAR-5m set with the same class proportions as CIFAR-10 test and CIFAR- 1. We run 500 NPLM pseudo experiments for each of the six kernel widths [0.1, 1.5, 2.6, 3.6, 4.9, 9.8] to produce the following distributions of test statistics:
1. Null hypothesis: in each experiment R is composed of 8500 randomly sampled CIFAR-10 test set images, and D from the remaining 1500.
2. CIFAR-10.1; in each experiment R. is composed of 8500 randomly sampled CIFAR-10 test set images, and D from 1500 randomly sampled CIFAR-1001 images.
3. CIFAR-5m: in each experiment R is composed of 8500 randomly sampled CIFAR-10 test set images, and D from 1500 randomly sampled CIFAR-5m images.
Distributions of the NPLM test statistic for each scenario and each kernel width plotted in Fig. [4] with the corresponding asymptotic and empirical 7 scores for CIFAR-10.1 and CIFAR-5m relative to the CIFAR-10-only null hypothesis indicated in the legends. Even the smallest and worst- performing kernel a = 0.1 distinguishes CIFAR-IO1 10.1 from CIFAR-10 10 at the 2.2g level, while the remaining larger kernels distinguish it extremely easy beyond even the 10g level. This indicates a clear distributional shift between CIFAR- 10 and CIFAR- 10.1, and presents one of the first (to our knowledge) statistically rigorous quantifications of this discrepancy.
The discrepancy between CIFAR-10 10 and CIFAR-5m is notably much smaller, saturating near 2.3a for the best performing kernels. This is exactly in line with what one would expect, given that CIFAR-5m is generated from a diffusion model trained on CIFAR-10. A well-trained diffusion model is able to model its training distribution exceedingly accurately, and the relatively small deviation we observe here underscores this fact. More interestingly, this hints at an unexpected but fascinating potential use case for AutoSciDACT method for evaluating the quality of generative models.
E Searching for the Higgs boson in LHC Data
To demonstrate how AutoSciDACT might be used in a more realistic setting, we use it search for evidence of the Higgs boson (H) in a dataset of real proton- proton collision data collected by the Compact Muon Solenoid (CMS) experiment at the Large Hadron Collider (LHC). We specifically target the four-lepton final-state, where the Higgs boson decays to four electrons, four muons, or two electrons and two muons (pp - H e+c e-= -u p 4* or p e*e). The Higgs boson was discovered through the observation of an excess in predominantly these final states and the di-photon final state [122 [23), with Higgs-like events isolated using hand-tuned selections physics-motivated on physics-mi motivated variables that were reconstructed from observed data. Here, we replace the majority of this selection with the AutoSciDACT pipeline, keeping only a loose "pre-selection" " of events designed to suppress large, well-known background processes.
36

---

SO V OTAR OFAR-O0 Sur
KPL:: OTAR.SM, GPAR-20
187
140
123
Countis
49
33
120
320
(a)
(b)
(c)
14.34: (3:44 10ss. OHAN Sm, CPANJ01
NRMM CIAR CITAR-TT CITI
MP..6 OI-4 30 OFAN
CRAN
292
Gral 111
153
Counts
Counts
133
75
1.33
25-
150
270
230
300
350
302
150
200
250
369
300 1.50
250
TuRc
(d)
(e)
(f)
Figure 14: NPLM tests comparing CIFAR-10 CIFAR-5m and CIFAR-10.1 using four dimensional contrastive embedding. The reference is built from the CIFAR-10 test set, and test stastics for the null hypothesis are shown in gray. Test statistics comparing against CIFAR-5m are shown in red, while those comparing against CIFAR-10. are shown in blue. Panels (a)-(f) corresponding to the six different NPLM kernel widths a [0.1, 1.5, 2.6, 3.6, 4.9, 9.8].
run AutoSciDACT on both real and simulated events from the CMS Open Data 124 for 2011 and 2012. We impose similar data pre-selection to the standard CMS search, requiring at least well-identified and loosely isolated electrons or muons with transverse momenta (pr) greater than GeV, the most energetic of which ("leading lepton") having PT > 20 GeV. Additionally, we require one opposite-sign pair of electrons or muons with invariant mass (mz++-) greater than 12 GeV. the first stage of AutoSciDACT, dominant background processes (Z boson pair production with decays to the 4e, 4μ, and 2e2p final states). The encoder is small MLP that takes input the full kinematic information (Pz: Pys P=, E) and particle ID (electron or muon) of the four leptons in the event, for a total of 20 inputs. As in the main paper, train four-dimen -iunniimeniional embedding space.
pre-train the contrastive encoder using simulated events from the
In the search phase, we follow the standard procedure for CMS data analysis and compute expected p-values based on simulated data, then measure the observed p-value in real data. Expected p-values are computed empirically by running many toys for the background-only and background + Higgs hypotheses to obtain distributions of test statistic (either the NPLM test statistic from a direct to the four-lepton invariant mass, see below). For each NPLM toy, we sample the reference (R) and observed (D) datasets from simulated background events, adding additional simulated Higgs events to D for toys under the background signal hypothesis. Finally, a single test statistic is computed using the true data (i.e. observed in CMS), and the observed p-value is computed relative to the test statistic for background-ond toys.
obtained
We perform three different tests using the statistical procedure described above:
Baseline Discovery We assume full knowledge of the Higgs boson, and perform a one- dimensional hypothesis test using the single most discriminating variable: the four-lepton invariant mass mag. The Higgs boson signal should manifest itself localized "bump" or peak in histogram of observed m4e ma mH (approximately 125 GeV). We construct
37

---

Mat Fit
4D Contrastive Space, with Higgs
Trys Sim, dkg Only Trys Sim. dkgg 5gg observed
Taya. Sem. Bkg Only Taya, 2mm, Bkg 59 obsurraed
[(eep)
plobs) 0.098
60
Toys
Toys
60
plexp) 0.056 plobos 0.03
40
A Log(c)
NPLM Score
(a)
(b)
4D Contrastive Space
4D Contrastive Space
18m
Expoctes Bockin S PUURd Tild 15mm)
observed
DdP
Anomaly Score
125
8
Toys
0.162
plobs)
50-
Data
Medium Anomaly Score
Tight Anomaly Score Cut
10
NPLM Score
20
Mass(GeV)
349
190
(c)
(d)
Figure 15: Top row: test-statistic distributions for simulated background-only (blue) (orange) sim- ulated background Higgs signal (orange), along with observed test statistic (red), for a direct fit to mar (left) and AutoSciDACT with Higgs included in training (right). Bottom left: Expected and observed test statistics/p-values for vanilla AutoSciDACT trained without knowledge of the Higgs. Bottom right: Distributions of m4e versus AutoSciDACT NPLM anomaly score in simulated backgrounds (filled) and observed data (points), where the most anomalous observed data lies near the true Higgs mass (125 GeV).
background-only (B) and signal (S) m4e histogram shape templates using simulated events, then fit the observed mss histogram to the background (B) or signal + background (S' + B) hypotheses. The statistical significance of an observation then computed from the delta log-likelihood test statistic obtained from the two fits: A log C = log Cs+B B - log Cp. We view this as the baseline sensitivity for discovery.
AutoSciDACT NPLM We run NPLM on the four-dimensional embedding space obtained from the contrastive encoder. We expect the presence of domain shift between the simulated and observed data (i.e. mis-modeling) to somewhat weaken the sensitivity of discovery.
Supervised AutoSciDACT NPLM We re- re-train our contrastive encoder with simulated Higgs events included in the training set, that it explicitly learns about the signal of interest. The Higgs signal should be well-separated from backgrounds in the learned embedding space, expect sensitivity in this case to be par with the baseline obtained from the mae fit.
We present results from our analysis in Fig. The top row shows expected test statistic distributions and expected/observed p-values for the baseline (left) and supervised AutoSciDACT (right) methods. Both methods achieve similar expected performance, with the observed p-values differing somewhat in either direction due to known domain shifts between simulated and real LHC data. These shifts lead larger spread observed p-values, and given that we are not properly accounting for systematic uncertaintics in our analysis, they have noticeable effect. The expected p- valuee are thus more
38

---

informative, as they provide a clear impression of how effectively each method finds Higgs signal without the impact of domain shift.
Figure (5(e) shows results using standard AutoSciDACT (i.e. contrastive space trained without knowledge of the Higgs). As expected, it is somewhat less sensitive than the baseline methods. However, even the baselines do not achieve highly significant p-values, largely due to the small dataset size and the fact that only consider one Higgs decay mode To better interpret these results, in Fig. d) we plot two-dimensional distributions of m4 versus NPLM anomaly score for simulated backgrounds (filled) and observed data (points). We observe that the most anomalous points in real data are clustered near 120-133 Ge near the known mass of the Higgs (125 GeV), We draw two horizontal lines depicting example medium/tight thresholds on the NPLM score, showing how these Higgs-like events could be isolated in a dataset by selecting on anomaly score. Note that full analysis using this selection would likely increase sensitivity, since would use both the 4-lepton mass and the anomaly score.
F Embedding Space Visualizations
For reference, we include visualizations of the four-dimensional contrastive embedding spaces for the CIFAR-10, JetClass, LIGO, and histology datasets in Figure 16
'The expected significance in this channel was only 2o in the original CMS paper [12 with discovery claimed only by combining results from several channels.
39

---

Airplane
Frog
QCD
04
Bird
Horse
Top
Z
Ship
H-
Deer
Truck
Dog
Automobile
B
Y
Y
a
B
(a)
(b)
Lung
SineGaussian
Kink
Heart
Pancreas
BBH
KinkKInk
Kidney
Spleen
Gaussian
Background
Liver
Liver (rat)
Cusp
Giltches
Liver Disease
White Bursts
α
V
α
B
(c)
(d)
Figure 16: Corner plots showing the four-dimensional contrastive embedding spaces for CIFAR-I0 10 (a), JetClass (b), histology (c), and LIGO (d). The turqoise clusters correspond to the classes used in training the encoder, and the pink cluster shows the distribution of "anomalous" " signal in the learned space.
40


Reviews:
[{"review_id": "PRt8MzRyFC", "reviewer": "Reviewer_2DCU", "summary": "This article presents an anomaly detection approach that combines contrastive learning with hypothesis testing. The authors propose a general-purpose, two-stage pipeline for detecting novelty in scientific data, aimed at enabling end-to-end automatic scientific discovery. The pipeline is evaluated on datasets from diverse domains, including astronomy, particle physics, histology, and CIFAR datasets.", "questions": "* To clarify the technical details: for the histology data and CIFAR images, is the embedding dimension fixed to 4? If so, I’m concerned that this setting may not be optimal and could negatively impact the experimental results.", "limitations": "See \"Strengths And Weaknesses\".", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 1, "strengths_and_weaknesses": "The paper introduces a two-stage pipeline combining contrastive learning and hypothesis testing for anomaly detection in scientific data. While the approach is promising, it falls short of demonstrating true generalist capabilities for automated scientific discovery. The methodology and experimental setup raise concerns about generalization, applicability, and evaluation rigor. Several technical choices and omissions limit the strength of the conclusions.\n\n* The method is presented as a generalist approach for Automated Scientific Discovery, but in practice, it appears to be a specialized anomaly detection pipeline targeting a narrow class of anomalies.\n* The paper lacks a comprehensive set of baseline comparisons. Several strong and commonly used anomaly detection methods are omitted, making it difficult to assess the relative performance of the proposed approach.\n* Some settings and claims (e.g., universality across domains) are not well-motivated or empirically validated.\n* The related work section is incomplete and does not adequately position the proposed method within the broader landscape of existing anomaly detection research.\n* The choice of a simple MLP as the supervised baseline is insufficient for meaningful comparison, especially when stronger models could offer a better performance benchmark.", "was_revised": true, "final_justification": "The paper lacks rigorous technical justification and thorough model evaluation. The proposed \"baselines\" consist of overly simplistic anomaly scores that do not engage with the contrastive training framework introduced in the first stage. Due to these methodological shortcomings, I cannot justify increasing my score.", "rebuttal": "We thank the reviewer for their feedback. We are concerned that some of the reviewer's critiques inaccurately reflect the content of our paper or miss crucial details. Some others make broad claims but without detailed justification or explanation, making it difficult to craft a detailed response. We will attempt to clarify some misunderstandings, and do our best to respond point-by-point to the reviewer’s comments.\n\n***\n\n> **Pipeline target[s] a narrow class of anomalies” and “some settings and claims (e.g., universality across domains) are not well-motivated or empirically validated.**\n\nWe demonstrate our pipeline on data from five distinct domains (one synthetic, one image-based, and three scientific; elaborated in section 4 and appendix A). Data modalities vary widely across these domains, and the anomaly takes a different form in each dataset. In section 6, we present results for all datasets (including time series data, images, point clouds, and tabular data), showing sensitivity to very small fractions of anomalous signals in each domain. We disagree with the reviewer’s characterization of these domains as a “narrow class of anomalies”, and point to our extensive experiments in each domain as empirical validation of our approach’s wide applicability.\n\n***\n\n> **The paper lacks a comprehensive set of baseline comparisons. Several strong and commonly used anomaly detection methods are omitted[...]**\n\nAs discussed in our Related Work section (Sec. 2), our anomaly detection approach differs from most standard approaches in its statistical rigor, with a p-value/Z-score being the primary figure of merit. To our knowledge, this approach has not been extensively explored in prior work. Standard approaches typically report e.g. AUROC and make no claims of statistical significance, making direct comparisons difficult. We include three baselines where a statistically rigorous anomaly score is possible: two “fully supervised” methods (MLP classifiers trained to find the anomaly) and the Mahalanobis distance. \n\nAdditionally, we have performed studies with two additional baselines: Maximum Mean Discrepancy (MMD) and Frechet Distance (FD). We have evaluated these baselines for the gravitational wave, particle physics, and CIFAR datasets. We use the “empirical” method to compute p-values/z-scores with these metrics, as their asymptotic distributions are unknown. The results vary somewhat between datasets, but in all cases the empirical MMD/FD performs as well as or below empirical NPLM (our method) with the default $d = 4$ contrastive embedding space. MMD/FID and NPLM perform more equally at lower anomaly injection fractions, while NPLM becomes significantly better for larger injections. Interestingly, we do not observe consistent trends in the relative performance among the three non-NPLM methods (Mahalanobis, MMD, FID), other than that they tend to lag behind NPLM. This highlights the complementarity of the various statistical anomaly detection methods we compare with NPLM. **This study will appear in the updated camera-ready version of our paper, though unfortunately its results (i.e. plots) cannot be shared here.**\n\n\n***\n\n> **The choice of a simple MLP as the supervised baseline is insufficient for meaningful comparison[...].**\n\nWe agree that an MLP classifier would be insufficient if trained on the raw data inputs, but we argue this is not the relevant setting for discussing baselines for our approach. The foundation of our pipeline is the contrastive embedding, which vastly reduces the dimensionality of a dataset into a handful of expressive features. The anomaly detection step proceeds using these features rather than the raw inputs, which is crucial for allowing us to make statistically robust claims about the presence of anomalies (making statistically rigorous claims from high-dimensional raw inputs would require a prohibitively large number of samples). \n\nGiven that dimensionality reduction is critical for the statistical methods used, we argue that it makes the most sense to compare anomaly detection methods applied to the embeddings rather than raw inputs. With this in mind, using an MLP as a supervised baseline is sensible: the data are vectors in a low-dimensional space, so the classification problem doesn’t require a more sophisticated architecture.\n\n***\n\n> **The related work section is incomplete and does not adequately position the proposed method within the broader landscape of existing anomaly detection research.**\n\nWe find the statement too vague to address meaningfully. No specific works, subfields, or methodological gaps are cited, making it difficult for us to act on this critique. Our related work section (section 2) was written with careful attention to representative categories in anomaly detection relevant for our work, including contrastive learning, contrastive anomaly detection, and hypothesis testing for anomaly detection, and we explicitly cite and discuss key works from each category. While most of the current literature focuses on identifying anomalous data, our work focuses on giving a meaningful statistical interpretation of said anomalies in a scientific context.\n\n***\n\n> **To clarify the technical details: for the histology data and CIFAR images, is the embedding dimension fixed to 4? If so, I’m concerned that this setting may not be optimal and could negatively impact the experimental results.**\n\nThe embedding dimension is fixed to 4 for experiments presented in the main text, as noted in Section 5. We have performed additional studies increasing the dimensionality of the embedding up to 32. We find no significant change in detection performance. In realistic applications where the signal is not known a priori, tuning the dimensionality of the embedding is not possible, and a priori design choices must be made. As discussed in the paper, a low-dimensional embedding was selected to make the statistical component of the anomaly detection tractable. In addition, low-dimensional representations allow the deployment of classical statistical methods and benefit uncertainty quantification.\n\nOur additional studies with dimensions $d = 8, 16, 32$ held all other factors (training time, NPLM kernel size, etc) the same as in the main results (Fig 3). At modest signal injections – where NPLM and any of the other baselines are able to detect an anomaly with $\\sim2-3\\sigma$ significance or higher – we observe fairly mild variation in NPLM’s performance up to $d = 32$. The sensitivity degrades slightly as dimension is increased, but by a nearly negligible amount (0 to 0.5$\\sigma$) and not in all cases. The MMD and FD metrics are similarly stable as a function of dimension, varying only slightly and sometimes non-monotonically up to $d=32$. The Mahalanobis distance, on the other hand, tends to perform slightly better with increasing dimensionality. We hypothesize that this is due to more dimensions helping the background clusters be pushed further apart under the SimCLR objective. Since Mahalanobis is defined relative to these cluster centroids, a collection of tighter/better-separated clusters would be more sensitive to outliers.\n\n***\n\n**Summary**\nWe thank the reviewer again for their feedback, and sincerely hope that our explanations and clarifications above have helped improve the paper and address the reviewer's concerns. We are happy to follow up with additional details during the discussion period!"}, {"review_id": "uAT7F35SNR", "reviewer": "Reviewer_drbe", "summary": "The authors introduce AutoSciDACT (Automated Scientific Discovery with Anomalous Contrastive Testing), a new\ntwo-part tool to detect and statistically validate anomalies in large and complex scientific datasets. The tool helps\nsolve two critical challenges in scientific discovery: the high dimensionality and noise typical in real-world data, and\nthe need for rigorous, interpretable statistical testing of potential outliers.\nDuring pre-training, AutoSciDACT utilizes supervised contrastive learning to compress raw data into\nlow-dimensional, semantically meaningful embeddings. This will incorporate some domain knowledge by using\nlabeled data or scientifically relevant augmentations, which provide the necessary complexity reduction while\nkeeping the salient features of the raw data.\nIn the discovery phase, the pipeline applies the New Physics Learning Machine (NPLM), which is a kernel-based,\nsignal-agnostic statistical hypothesis testing methodology. NPLM allows us to compare an \"observed\" dataset to a\n\"reference\" background dataset in order to identify any distributional shifts (e.g., clusters or overdensities) that may\nrepresent new phenomena. The method outputs statistically calibrated significance scores (Z-scores, p-values),\nsupporting robust claims of discovery.\nThe authors evaluated AutoSciDACT against five datasets well within astronomy, particle physics, biology, and\ncomputer vision , demonstrating its domain-agnostic performance and sensitivity to subtle signals—even at sub-1%\nanomaly rates. The authors have also discussed its potential towards detection of dataset shift (e.g., CIFAR-10 vs.\nCIFAR-10.1), suggesting broader utility in model evaluation and data quality assessment.", "questions": "1. How robust is AutoSciDACT to label noise or partial supervision in the pre-training phase?\nSince the pipeline relies on supervised contrastive learning using labeled data, it's important to understand how\nperformance degrades with noisy or incomplete labels, especially in domains where precise annotation is difficult.\nSuggestion: Consider including experiments (even synthetic) that measure sensitivity to label corruption or using\nsemi-supervised SupCon variants.\nImpact: Demonstrating robustness (or proposing mitigation strategies) would increase confidence in the method’s\napplicability to less curated domains.\n\n2. Can the method handle domain shift between simulated/reference data and observed data?\nThe pipeline assumes the background data (R) and the test data (D) are drawn from the same distribution under the\nnull hypothesis. However, simulation-to-real gaps or data collection shifts are common in real scientific settings.\nSuggestion: Provide a discussion or ablation showing how mild domain shift affects NPLM outputs or embeddings,\nand whether techniques like domain adaptation or uncertainty modeling can be integrated.\nImpact: If addressed well, this would significantly strengthen the generalizability and practical utility of\nAutoSciDACT.\n\n3. How sensitive is NPLM’s performance to the fixed 4D embedding choice?\nWhile low-dimensional embeddings are necessary for statistical tractability, this constraint may lead to loss of\ninformation, especially for datasets with many complex classes (e.g., CIFAR-10).\nSuggestion: Include a sensitivity study on the embedding dimension (e.g., comparing d = 4, 8, 16) and how it affects\nboth detection performance and computational cost.\nImpact: Clarifying this trade-off can help users balance interpretability vs. expressiveness and would inform how to\nscale AutoSciDACT to larger problems.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nQuality\n\n* The paper is technically sound, with well-justified methodological choices. It combines contrastive learning\nand New Physics Learning Machine (NPLM) into a coherent, end-to-end pipeline.\n* Experimental validation is comprehensive across diverse domains (astronomy, physics, histology, synthetic\ndata, and images), demonstrating both robustness and generalizability.\n* Statistical claims are backed by empirical and asymptotic p-values and Z-scores, following good scientific\npractice.\n* The authors are transparent about their assumptions, limitations (e.g., embedding dimension, domain\nshifts), and the scope of the method.\n\nClarity\n\n* The paper is well-organized and readable, particularly for readers familiar with ML.\n* Diagrams, dataset descriptions, and appendix details contribute to a clear presentation of the pipeline and\nexperiments.\n* The method's purpose (scientific novelty detection with statistical rigor) is clearly distinguished from\ngeneral anomaly detection approaches.\n\nSignificance\n\n* This work addresses a critical and underexplored need in machine learning for science: tools that not only\ndetect anomalies but quantify their statistical significance in a scientifically interpretable way.\n\n* AutoSciDACT offers a domain-agnostic framework applicable to any field with high-dimensional, labeled\ndata—making it valuable for physicists, biologists, astronomers, and potentially even ML safety\nresearchers.\n\nOriginality\n\n* While the core components (contrastive learning, NPLM) are based on existing work, their integration into\na unified pipeline tailored for scientific discovery is novel and well-motivated.\n\n* The use of Supervised Contrastive Learning for dimensionality reduction that respects domain labels is an\neffective and underused idea.\n\n* The pipeline is also shown to be extensible: e.g., in Section C, it is repurposed for evaluating dataset shift\nand generative model fidelity, suggesting a broader set of applications beyond discovery.\n\n**Weaknesses**\n\nQuality\n\n* The performance of AutoSciDACT relies heavily on the availability and quality of labeled background\ndata. In real-world domains with scarce or noisy labels, this assumption may limit applicability.\n\n* Although the authors mention potential solutions (e.g., augmentations, domain knowledge), there is no\nquantitative study on robustness to label noise or partial supervision.\n\n* The embedding dimension is fixed to 4 for NPLM compatibility. While this keeps the statistical test\ntractable, it may limit expressiveness, especially in highly multimodal data like CIFAR-10.\n\nSignificance\n\n* Despite strong empirical results, AutoSciDACT currently assumes that the reference and observed\ndistributions are aligned. In real-world applications, domain shift between simulated (or historical)\nreference data and observed data is common, and this limitation is only addressed as future work.\n\n* The application to scientific discovery is exciting, but no real-world discovery example is included—only\nsimulated signals in known datasets. Including or proposing such a use case would strengthen the impact.\n\nOriginality\n\nThe core components (e.g., SupCon, NPLM) are re-used from prior work, and while the combination is\nnovel and useful, the method itself is not radically innovative in terms of algorithmic design.", "was_revised": true, "final_justification": "The authors appropriately acknowledge and clarifies the issues I raised. My score stands.", "rebuttal": "We sincerely thank the reviewer for their thoughtful and encouraging feedback. We are pleased to hear that our method was found to be “novel and well-motivated”. We especially appreciate your recognition that our pipeline not only detects anomalies but also quantifies their statistical significance in a scientifically interpretable way, as this was the central motivation of this work. Below, we respond to the specific weaknesses, suggestions, and questions you raised, and we clarify a few points to strengthen the paper further. We are confident that the revisions based on your feedback will help improve the manuscript further.\n\n***\n\n> **The performance of AutoSciDACT relies heavily on the availability and quality of labeled background data. In real-world domains with scarce or noisy labels, this assumption may limit applicability.**\n\n> **How robust is AutoSciDACT to label noise or partial supervision in the pre-training phase?**\n\nThese are great observations and questions! This is certainly a concern for scientific domains that rely on human- or algorithmically-labeled datasets where ground-truth is inaccessible. We agree that substantial label noise would degrade the quality of the learned embedding space, but NPLM’s sensitivity to subtle tails or overdensities may preserve its performance as long as a suitable background-only reference can be identified.\n\nTo address this question, we have performed an additional study exploring the impact of label noise in the pre-training data using the JetClass dataset. We trained variants of the embedding with 1%, 2%, 5% and 10% label noise, where X% refers to the probability of a training sample’s label being randomly changed to a different value. During training, we observed that high label noise caused the models to learn poorly separated and less structured embeddings of the background classes. With a 1% injection of Higgs signal, we observed that the empirical NPLM significance across 500 toys degraded by about $1\\sigma$ from 0% to 10% label noise, from $3\\sigma$ to $2\\sigma$. (The $3\\sigma$ at 0% noise is limited by the choice of 500 toys; the asymptotic significance at this injection was around 4$\\sigma$ and decreased to $2\\sigma$ by 10% label noise). The Mahalanobis metric was more stable, but significantly less sensitive ($<2\\sigma$). **This study will appear in the updated camera-ready version of our paper, though unfortunately its results (i.e. plots) cannot be shared here.** \n\n***\n\n> **How sensitive is NPLM’s performance to the fixed 4D embedding choice?**\n\nThis is a good question, and highlights some of the key design choices that went into developing AutoSciDACT. We agree that, by conventional representation learning standards, a four-dimensional embedding is relatively small. As you point out, this choice is made to keep the statistical analysis tractable. Exploring higher (but still modest) dimension embeddings is a great idea and will strengthen the claims made in our paper. \n\nTo this end, we have performed additional studies exploring our pipeline’s performance with larger embedding dimensions $d = 8, 16, 32$, with all other factors (training time, NPLM kernel size, etc) kept the same as in the main results (Fig 3). At modest signal injections – where NPLM and any of the other baselines are able to detect an anomaly with $\\sim2-3\\sigma$ significance or higher – we observe fairly mild variation in NPLM’s performance up to $d = 32$. The sensitivity degrades slightly as dimension is increased, but by a nearly negligible amount (0 to 0.5$\\sigma$) and not in all cases. The Mahalanobis distance, on the other hand, tends to perform better with increasing dimensionality. We hypothesize that this is due to more dimensions helping the background clusters be pushed further apart under the SimCLR objective. Since Mahalanobis is defined relative to these cluster centroids, a collection of tighter/better-separated clusters would be more sensitive to outliers. **This study will appear in the updated camera-ready version of our paper, though unfortunately its results (i.e. plots) cannot be shared here.**\n\n***\n> **In real-world applications, domain shift between simulated (or historical) reference data and observed data is common, and this limitation is only addressed as future work.**\n\n> **Can the method handle domain shift between simulated/reference data and observed data?**\n\nThis is an important observation - we fully agree with the reviewer on the importance of managing domain shifts when applying the methodology to real data. Our goal is to make this work directly applicable to real-world datasets. In light of that, we have chosen a variety of datasets as our benchmarks, some of which include real data (histology, gravitational-wave astronomy), while others are built on simulation. \n\nThe histology datasets are “real” datasets, labelled by humans according to which organ they belong to and whether the fat-liver disease was present. While there may be elements of domain shift in the dataset due to variations in the data acquisition, such as staining, these elements are corrected through human labels. The choice of supervised contrastive learning then effectively mitigates the domain shift by explicitly using these labels. In the gravitational wave study, backgrounds are taken from signal-free regions of real data, and thus implicitly feature many of the realistic variations that might be present in another sample of real data analyzed using our pipeline. The signals are simulated and injected into the backgrounds, and reflect state-of-the-art modeling that is used in mainstream gravitational wave analyses. While domain shift is not *explicitly* addressed in either of these cases, we want to highlight that training dataset curation can indirectly have an impact. Significant domain adaptation strategies may not be necessary to perform anomaly detection searches if the training and reference datasets can be (partially) taken from real data,\n\nLastly, we should mention that there are variants of NPLM that can handle systematic uncertainties (see e.g. Ref. [51] in our paper), and these uncertainties can help account for the impact of domain shift. Including this variant of NPLM is beyond the scope of this work, but we hope to continue building on our ideas in AutoSciDACT to extend it towards domain adaptation.\n\n***\n\n**Summary**\n\nWe thank the reviewer again for their thoughtful, positive, and constructive feedback. We hope that our responses have clarified key details of our work and addressed the reviewer's concerns. Please let us know if we can further clarify, explain, or expand upon any specific points - we would be quite happy to continue a dialog during the discussion period!"}, {"review_id": "htvzroYUZq", "reviewer": "Reviewer_jLRS", "summary": "The authors observed that in scientific discovery, novel findings are tied to data points that are not in the background distribution. Based on this observation, they utilize anomaly detection with contrastive learning, using huge amounts of good quality simulation data to find anomalous data patterns. With the help of the New Physics Learning Machine (NPLM), they can then test the statistical significance of these findings to determine if the data is truly out-of-distribution. This significance score serves as an important tool in scientific research. All in all, the pipeline is a proper use case of anomaly detection and contrastive learning to help scientific discovery.", "questions": "1. In Lines 201-202, the description of the synthetic dataset mentions rotating 'The full M × N dimension space.' This phrasing was slightly confusing, as my understanding is that N represents the number of distinct data clusters, while the data itself exists in a (D+M)-dimensional space. Could you please clarify if the text was intended to state that the full (D+M)-dimensional space is rotated to obscure the signal?\n\n2. In Figure 2 (right panel), in the 2D projection, the anomalous signal class (pink) appears to have a significant visual overlap with one of the background classes (blue). This raises a question about the pipeline's effectiveness in this specific case. Could you elaborate on how detection is achieved here? For example, is the separation clearer in other, un-plotted dimensions of the embedding, or is the NPLM test sensitive enough to detect a subtle overdensity even within this overlapping region?\n\n3. In Figure 3 (d) and (f), in these two panels, the 'supervised' baseline (blue line) consistently outperforms the 'ideal-supervised' baseline (red line), which is the opposite of the trend seen in the other datasets. It's notable that both of these are image-based tasks. Could you offer some insight into this counter-intuitive result?", "limitations": "The authors disscussed the limtations.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper's primary strength is its statistical rigor. The entire pipeline is grounded in a formal hypothesis testing framework that quantifies the significance of any detected novelty through p-values and Z-scores, providing a robust foundation for scientific claims.\n2. A key contribution is the paper's insightful framing of scientific discovery as a statistical anomaly detection problem. This provides a clear and actionable paradigm for automating the search for novel phenomena in large datasets.\n\nWeaknesses:\n1. The empirical validation of the method is limited. Despite showing success on 3 tasks (exclude toy data and CIFAR-10), the applications are drawn exclusively from the domains of physics and astrophysics, which is insufficient to substantiate the paper's broad claims of general scientific utility. Demonstrating efficacy in disparate fields, such as genomics, chemistry, or climate science, would be essential for this.\n2. The paper's conceptualization of \"scientific discovery\" is limited, which narrows the framework's scope and makes its title potentially misleading. The term \"discovery\" often implies not just the detection of a new phenomenon, but also the generation of an explanatory hypothesis for it. The AutoSciDACT framework is tailored for the first step—identifying observational novelty—but its scope does not extend to the crucial subsequent step of building models or inferring the causal mechanisms that would explain the detected anomaly.", "was_revised": true, "final_justification": "The author provided very detailed reply adressed my concerns. I decided to raise my mark to 4.", "rebuttal": "We thank the reviewer for their detailed review and thoughtful feedback! We particularly appreciate their acknowledgment of our paper’s core contribution: a statistically rigorous treatment of anomaly detection, which makes it compatible with scientific hypothesis testing and robust claims of discovery. We are also pleased that our pipeline “provides a clear and actionable paradigm for automating the search for novel phenomena in large datasets.”\n\nBelow, we respond to the reviewer’s specific comments and questions:\n\n***\n\n> **The empirical validation of the method is limited. Despite showing success on 3 tasks (exclude toy data and CIFAR-10), the applications are drawn exclusively from the domains of physics and astrophysics, which is insufficient to substantiate the paper's broad claims of general scientific utility. Demonstrating efficacy in disparate fields, such as genomics, chemistry, or climate science, would be essential for this.**\n\nWe agree that demonstrating our pipeline’s efficacy across disparate fields is important! A small point of clarification: we included baselines from three distinct scientific disciplines: particle physics (Figs. 2(a), 3(c), 7(b), 8(b)), astrophysics (LIGO, Figs. 2(b), 3(d), 5, 7(c), 8(c)), and histology (Figs. 3(e), 6, 7(d), 8(d)). Histology is distinct from physics and astrophysics. \n\nMore importantly, these datasets span diverse **data modalities**: time-series (LIGO), point clouds (particle physics), and images (histology, CIFAR). This is reflected in the diversity of architectures used for the contrastive embeddings (transformers, resnets). We argue that, for our pipeline, this distinction is equally as important as conceptual distinctions between scientific domains. AutoSciDACT interacts with a discipline at the level of datasets, meaning a variation in modality is arguably the relevant distinction between scientific domains. Higher-level distinctions between domains are relevant insofar as they impact how well data from a particular domain can be embedded into a low-dimensional contrastive space, which is related to intrinsic qualities of data (e.g. information content, diversity of classes, availability of relevant augmentations, etc.) With this in mind, our experiments across modalities are broader in scope than the qualitative distinction between e.g. particle physics and astrophysics.\n\nWhile we argue that the presented datasets already span a broad range of scientific applications, we have conducted an additional study using a genomics dataset comprised of images of butterfly hybrids paired with genetic information. We trained AutoSciDACT using pictures of non-hybrid butterflies with labels identifying the butterflies as belonging to one of 12 subspecies classes. Butterflies form new hybrids in the wild, and we used our pipeline to detect \"anomalies\" (i.e. new hybrids) in the testing dataset. **This study will appear in the updated camera-ready version of our paper, though unfortunately its results (i.e. plots) cannot be shared here.**\n\n***\n\n> **The paper's conceptualization of \"scientific discovery\" is limited, which narrows the framework's scope and makes its title potentially misleading. The term \"discovery\" often implies not just the detection of a new phenomenon, but also the generation of an explanatory hypothesis for it. The AutoSciDACT framework is tailored for the first step—identifying observational novelty—but its scope does not extend to the crucial subsequent step of building models or inferring the causal mechanisms that would explain the detected anomaly.**\n\nAs correctly pointed out by the reviewer, this work focuses on the statistical strategy to detect significant anomalous behavior. It does not provide a solution for the interpretation and signal hypothesis generation that follow the detection stage. However, the NPLM algorithm outputs not only a scalar test but also a model shaping the log-density-ratio between the reference sample and the sample of interest. This information can be used to get insights on the shape and location of the anomaly: similarly to standard classifier-based tests, the model $f_{\\hat{w}}$ resulting from the NPLM training can be sigmoided to obtain a classification score that can be used to select the most anomalous data points in the dataset; the reference set reweighted by the exponent of the model $f_{\\hat{w}}$ represents the learnt density distribution of the data, giving information on the shape of the generative process. While these insights do not provide a unique theoretical interpretation for the signal, they can guide experts in hypothesis formulation.\n\nTo demonstrate how this can be used in a real-data example, we have added a new appendix to the draft that uses particle collider data from the Higgs boson discovery (Open Data from the CMS experiment). In this study, we show that the most significant NPLM scores can point to the most anomalous events and indicate a Higgs boson. In particular, we find that the mass of the Higgs boson can be hinted at by examining correlations between the NPLM score and the four-lepton invariant mass among events flagged as most anomalous. Crucially, the learned embedding did not use four-lepton invariant mass as an input feature, but was still capable of flagging a mass overdensity via NPLM. It was trained instead on the raw inputs of the four leptons in an event (20 inputs in total). **This study will appear in the updated camera-ready version of our paper, though unfortunately its results (i.e. plots) cannot be shared here.** \n\n***\n\n> **In Lines 201-202, the description of the synthetic dataset mentions rotating 'The full M × N dimension space.' This phrasing was slightly confusing, as my understanding is that N represents the number of distinct data clusters, while the data itself exists in a (D+M)-dimensional space. Could you please clarify if the text was intended to state that the full (D+M)-dimensional space is rotated to obscure the signal?**\n\nThanks for catching this! You're right: the text should refer to the full D+M-dimensional space being rotated to obscure the signal. We corrected the typo in the manuscript.\n\n***\n\n> **In Figure 2 (right panel), in the 2D projection, the anomalous signal class (pink) appears to have a significant visual overlap with one of the background classes (blue). This raises a question about the pipeline's effectiveness in this specific case. Could you elaborate on how detection is achieved here? For example, is the separation clearer in other, un-plotted dimensions of the embedding, or is the NPLM test sensitive enough to detect a subtle overdensity even within this overlapping region?**\n\nThis is a great observation, and highlights one of the most important aspects of our pipeline. Indeed, in this example, the signal strongly overlaps with the background, imposing severe challenges for standard anomaly detection via e.g. flagging outliers. NPLM (our method), is explicitly designed as a two-sample test, which compares the dataset containing anomalous points to a reference dataset that is anomaly-free by construction at a distributional level. The test is sensitive to collective (distribution-level) anomalies that can manifest in many ways: as shape deviations/distortions/overdensities *in the bulk of the background-populated region*, or as excesses in the tails. In this case, we are sensitive to a distortion in the bulk. Since our method is sensitive to even these very subtle anomalies, it is a powerful general-purpose tool for discovering novelty in scientific datasets.\n\n***\n\n> **In Figure 3 (d) and (f), in these two panels, the 'supervised' baseline (blue line) consistently outperforms the 'ideal-supervised' baseline (red line), which is the opposite of the trend seen in the other datasets. It's notable that both of these are image-based tasks. Could you offer some insight into this counter-intuitive result?**\n\nYou're right that the supervised baseline outperforms the ideal supervised baseline in some cases, which is indeed counterintuitive. As a small point of clarification: panel (f) refers to the CIFAR dataset, which is indeed an image-based dataset, while panel (d) shows the results of the gravitational-wave dataset, which consists of time series (see Appendix A2, Figure 5).\n\nThe difference between the two baselines lies in the contrastive pretraining; for the supervised baseline, the anomalous signal information is omitted, while in the ideal supervised setting, the anomalous signal is added as an additional class. Rather than the data modality, the number of classes drives the difference: if the number of classes is already high relative to the embedding dimension (as it is for the CIFAR-10 and gravitational wave datasets), embedding an additional class in the low-dimensional embedding ($d=4$) is more challenging and might be suboptimal, leading to the observed difference. \n\nWe have also performed some additional studies exploring our pipeline’s performance with larger embedding dimensions $d = 8, 16, 32$, with all other factors (training time, NPLM kernel size, etc) kept the same as in the main results (Fig 3). In these studies we find that the \"ideal supervised\" performance quickly equalizes with or exceeds that of \"supervised\" as $d$ is increased. **This study will appear in the updated camera-ready version of our paper, though unfortunately its results (i.e. plots) cannot be shared here.**\n\n***\n\n**Summary**\n\nWe thank the reviewer again for their thoughtful, constructive feedback and insightful comments on our paper. We hope our response has clarified key details of our work and sufficiently addressed the reviewer's questions and concerns. Please let us know if we can further clarify, explain, or expand upon any specific points - we would be quite happy to continue a dialog during the discussion period!"}, {"review_id": "cZLCCGlcYH", "reviewer": "Reviewer_qPQa", "summary": "This paper introduces AutoSciDACT, a two-stage pipeline for scientific discovery. The two stages are: (1) Constructing a low-dimensional embedding using contrastive learning, and (2) employing the New Physics Learning Machine (NPLM) framework, performing a likelihood ratio test comparing a background distribution to another dataset with unknown composition. Extensive experiments on data sets from four diverse application domains (and one synthetic data set) demonstrate the effectiveness of the approach.", "questions": "- Why are the supervised baselines considered an upper limit (see line 285)?\n- Is it possible to compare the method to baselines like maximum mean discrepancy [1] or the Fréchet distance [2, 3]?\n- What is the justification for using a single, tied variance for all classes instead of a full covariance matrix for the Mahalanobis distance baseline (like, e.g., in [4])?\n- Have augmentations been used in the pre-training stage in the individual domains? If so, which augmentations?\n- How was the embedding dimension $d$ selected? $d = 4$ seems rather small compared to SOTA models.\n\n**References**\n\n[1] Gretton et al., A Kernel Two-Sample Test, JMLR 2012\n\n[2] Dowson et al., The Fréchet Distance between Multivariate Normal Distributions, Jounral of Multivariate Analysis 1982\n\n[3] Heusel et al., GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium, NeurIPS 2017\n\n[4] Lee et al., A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks, NeurIPS 2018", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- The paper is well-written. The process of scientific discovery establishes a good motivation for the paper.\n- AutoSciDACT employs a statistical test for discovery, which allows the user to assess significance levels.\n- The work includes a very extensive set of experiments on four diverse domains (astronomy, particle physics, histology, and images), showcasing the wide applicability of the pipeline.\n\n**Weaknesses**\n\n- The suggested pipeline is merely a combination of two existing methods (Contrastive Learning + NPLM), which diminishes the methodological contributions of the paper.\n- The paper is very dense, making certain sections rather hard to follow (e.g., the section introducing the supervised baselines - at least a more detailed discussion in the Appendix would have been beneficial, in my opinion)\n- The selection of baselines could be extended (see Questions).", "was_revised": true, "final_justification": "The authors have added additional baselines and their responses were satisfactory.", "rebuttal": "We thank the reviewer for their detailed and constructive feedback. We are pleased that they found the framing around the process of scientific discovery provides a strong and well-motivated foundation for our approach. We appreciate the recognition of AutoSciDACT’s broad applicability and domain generalization. We address each of the weaknesses, suggestions, and questions that the reviewer raised and offer clarifications. We believe that the revisions have significantly strengthened the manuscript.\n\n***\n\n> **The suggested pipeline is merely a combination of two existing methods (Contrastive Learning + NPLM), which diminishes the methodological contributions of the paper.**\n\nWe thank the reviewer for raising this concern. While the individual components of our approach may not be novel in isolation, we believe their combination is both original and impactful. It enables the application of statistical anomaly detection techniques to complex, high-dimensional, and structured datasets commonly encountered in scientific domains. \n\nRecent work on representation learning tends to favor high-dimensional embeddings to preserve rich semantic information. However, these high-dimensional spaces are often difficult to interpret and analyze statistically, posing challenges for uncertainty quantification and hypothesis testing, which are central in the scientific domain. The NPLM method and other statistical methods degrade in their ability to find statistically meaningful overdensities when random, non-discriminating variables are added to the dataset (see figure 4 in Appendix A).  Our contribution shows that contrastive learning, when guided by domain knowledge (i.e. class labels), can extract low-dimensional representations that retain sufficient information to enable anomaly detection in a statistically robust manner, removing information that is redundant or non-critical. Beyond anomaly detection, this approach demonstrates how previously developed statistical methods, widely trusted and used in the scientific community, can be successfully integrated with state-of-the-art machine learning techniques for representation learning. This opens new avenues for scientifically grounded, data-driven discovery.\n\n***\n\n> **The paper is very dense, making certain sections rather hard to follow (e.g., the section introducing the supervised baselines)**\n\nThank you for this feedback! We acknowledge that the paper is dense and agree that the section introducing the supervised baselines could be more explicit. Our goal was to concisely present multiple baselines within limited space, but we understand that this may have made the section difficult to follow.\n\nThe goal of the supervised baselines is to identify \"best case\" performance for anomaly detection given the contrastive embeddings, and to compare this to our anomaly detection algorithm (NPLM). The distinction between \"supervised\" and \"ideal supervised\" is whether or not the *true signal* was included as a class in the supervised contrastive training. Including it creates a space that is designed with the signal in mind, allowing \"ideal\" separation from background. We will add these clarifications to the revised version of the paper.\n\n***\n\n> **The selection of baselines could be extended**\n> **Is it possible to compare the method to baselines like maximum mean discrepancy or the Fréchet distance?**\n\nThank you for these helpful suggestions! Following the reviewer’s suggestion, we have performed studies with two additional baselines: Maximum Mean Discrepancy (MMD) and Frechet Distance (FD). We have evaluated these baselines for the gravitational wave, particle physics, and CIFAR datasets. We use the “empirical” method to compute p-values/z-scores with these metrics, as their asymptotic distributions are unknown. The results vary somewhat between datasets, but in all cases the NPLM (our method) performs equally well or outperforms MMD/FD with the default $d = 4$ contrastive embedding space. MMD/FID and NPLM perform more equally at lower anomaly injection fractions, while NPLM becomes significantly better for larger injections. Interestingly, we do not observe consistent trends in the relative performance among the three non-NPLM methods (Mahalanobis, MMD, FID), other than that they tend to lag behind NPLM. This highlights the complementarity of the various statistical anomaly detection methods we compare with NPLM. **This study will appear in the updated camera-ready version of our paper, though unfortunately its results (i.e. plots) cannot be shared here.**\n\n> **Why are the supervised baselines considered an upper limit (see line 285)?**\n\nThank you for highlighting this point and allowing us to clarify! The idea of the supervised baselines is to probe best-case performance for identifying signals *given the contrastive embedding*. They differ in whether or not the true signal was included in training the space (\"ideal\"), with \"ideal\" being close to the best low-dimensional representation one could have with a particular signal in mind. In contrast, the “supervised” baseline represents a more realistic intermediate case, where some signal information is used only at the inference stage. Since data compression is a core part of our pipeline, we argue that it is natural to baseline our methods in this compressed space. \n\nComparing these two baselines also offers insight into the learned representations: if the “supervised” and “ideal supervised\" match, it indicates that the representation learned through signal-agnostic supervised contrastive learning is well-suited for downstream anomaly detection, even without prior knowledge of the signal.  Additionally, comparing our approach directly to the “supervised” baseline quantifies the impact of lacking prior signal information on test performance. Finally, the “ideal supervised” case provides a meaningful reference point for assessing the intrinsic difficulty of the anomaly detection task itself. Notably, the signal injection rate alone does not fully capture the problem's complexity, as the detectability of rare perturbations also depends on the separation between signal and background in the embedding space.\n\n> **What is the justification for using a single, tied variance for all classes instead of a full covariance matrix for the Mahalanobis distance baseline (like, e.g., in [4])?**\n\nWe thank the reviewer for spotting the difference and have adopted the computation as suggested in [4]. The text in the manuscript is changed to reflect the update. Switching to this version of the Mahalanobis distance had negligible impact on the results we obtain in our studies, and the manuscript has been updated with the new numbers.\n\n> **Have augmentations been used in the pre-training stage in the individual domains? If so, which augmentations?**\n\nWe do not make use of any augmentations. We elaborate on this in Sec. 3.1, where we justify our choice of using labels with the richer domain knowledge they encode compared to augmentations, tailored view of inputs. Choosing augmentations also typically requires specific domain knowledge, and it may be difficult to design augmentations that cover as rich a variety of positive pairs as labels encode.\n\nWhile we do not use augmentations, they can be easily added to the contrastive embedding if desired, for example, in cases where labels are unavailable or to impose certain invariances in the embedding. The downstream anomaly detection and hypothesis testing part of AutoSciDACT is unaffected.\n\n> **How was the embedding dimension d selected? d=4 seems rather small compared to SOTA models.**\n\nThis is a very important aspect of our method and we thank the referee for calling attention to it! In principle, the optimal choice of $d$ for some signal cannot be known without *a priori* knowledge of the signal. The choice of $d$ is instead guided by another crucial constraint: *statistical tractability*. Performing statistically robust anomaly detection and producing calibrated test statistics is very difficult in high dimensions, quickly requiring a prohibitive number of samples as $d$ grows. We choose $d$ to allow for a powerful and tractable test while retaining enough flexibility to learn a good contrastive embedding.\n\nTo further address this point, we have performed additional studies exploring our pipeline’s performance with larger embedding dimensions $d = 8, 16, 32$, with all other factors (training time, NPLM kernel size, etc) kept the same as in the main results (Fig 3). At modest signal injections – where NPLM and any of the other baselines are able to detect an anomaly with $\\sim2-3\\sigma$ significance or higher – we observe fairly mild variation in NPLM’s performance up to $d = 32$. The sensitivity degrades slightly as dimension is increased, but by a nearly negligible amount (0 to 0.5$\\sigma$) and not in all cases. The MMD and FD metrics are similarly stable as a function of dimension, varying only slightly and sometimes non-monotonically up to $d=32$. The Mahalanobis distance, on the other hand, tends to perform better with increasing dimensionality. We hypothesize that this is due to more dimensions helping the background clusters be pushed further apart under the SimCLR objective. Since Mahalanobis is defined relative to these cluster centroids, a collection of tighter/better-separated clusters would be more sensitive to outliers. **This study will appear in the updated camera-ready version of our paper, though unfortunately its results (i.e. plots) cannot be shared here.**\n\n***\n\n**Summary**\nWe thank the reviewer again for their thoughtful and constructive feedback. We hope that our responses have clarified key details of our work and addressed the reviewer's concerns. Please let us know if we can further clarify, explain, or expand upon any specific points - we would be quite happy to continue a dialog during the discussion period!"}]

Author rebuttal:

Meta-review: The work proposes a method for detecting anomalous sample set of scientific artifacts given an established dataset of normal artifacts of the same kind. The method combines dimensionalitybreduction via contrastive learning and a recently published statistical hypothesis testing technique called NPLM. Most reviewers note that the paper is well written and contains an extensive empirical evaluation. 

However, despite voting for acceptance, the reviewers point out that the advertises scientific discovery *in datasets* is not addressed in the paper, and the paper explorers comparison *berween* dataset. The technique appears to be quite standard on the high level: dimensionality reduction and hypothesis testing in low dimensional space. Actual scientific discovery has not been demonstrated in the case studies, and was instead simulated by injection of perturbed instances or instances of a missing class.

My known reading also raises certain concerns on the soundness of the statistical analysis. Z score in four dimensions does not have the same meaning  as in one dimension. Even under isotropic normal distribution, the confidence intervals are different.
Decision: Accept (poster)