Title: AutoSciDACT: Automated Scientific Discovery through Contrastive Embedding and Hypothesis Testing
Venue: NeurIPS 2025
Accepted: yes
Keywords: anomaly detection

contrastive learning

scientific discovery

hypothesis testing

automated discovery
Abstract: Novelty detection in large scientific datasets faces two key challenges: the noisy and high-dimensional nature of experimental data,
and the necessity of making *statistically robust* statements about any observed outliers. While there is a wealth of literature on anomaly detection via dimensionality reduction, most methods do not produce outputs compatible with quantifiable claims of scientific discovery. In this work we directly address these challenges, presenting the first step towards a unified pipeline for novelty detection adapted for the rigorous statistical demands of science. We introduce AutoSciDACT (Automated Scientific Discovery with Anomalous Contrastive Testing), a general-purpose pipeline for detecting novelty in scientific data. AutoSciDACT begins by creating expressive low-dimensional data representations using a contrastive pre-training, leveraging the abundance of high-quality simulated data in many scientific domains alongside expertise that can guide principled data augmentation strategies. These compact embeddings then enable an extremely sensitive machine learning-based two-sample test using the New Physics Learning Machine (NPLM) framework, which identifies and statistically quantifies deviations in observed data relative to a reference distribution (null hypothesis). We perform experiments across a range of astronomical, physical, biological, image, and synthetic datasets, demonstrating strong sensitivity to small injections of anomalous data across all domains.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces AutoSciDACT, a two-stage pipeline for scientific discovery. The two stages are: (1) Constructing a low-dimensional embedding using contrastive learning, and (2) employing the New Physics Learning Machine (NPLM) framework, performing a likelihood ratio test comparing a background distribution to another dataset with unknown composition. Extensive experiments on data sets from four diverse application domains (and one synthetic data set) demonstrate the effectiveness of the approach.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The paper is well-written. The process of scientific discovery establishes a good motivation for the paper.\n- AutoSciDACT employs a statistical test for discovery, which allows the user to assess significance levels.\n- The work includes a very extensive set of experiments on four diverse domains (astronomy, particle physics, histology, and images), showcasing the wide applicability of the pipeline.\n\n**Weaknesses**\n\n- The suggested pipeline is merely a combination of two existing methods (Contrastive Learning + NPLM), which diminishes the methodological contributions of the paper.\n- The paper is very dense, making certain sections rather hard to follow (e.g., the section introducing the supervised baselines - at least a more detailed discussion in the Appendix would have been beneficial, in my opinion)\n- The selection of baselines could be extended (see Questions).\n\nQUESTIONS:\n- Why are the supervised baselines considered an upper limit (see line 285)?\n- Is it possible to compare the method to baselines like maximum mean discrepancy [1] or the Fréchet distance [2, 3]?\n- What is the justification for using a single, tied variance for all classes instead of a full covariance matrix for the Mahalanobis distance baseline (like, e.g., in [4])?\n- Have augmentations been used in the pre-training stage in the individual domains? If so, which augmentations?\n- How was the embedding dimension $d$ selected? $d = 4$ seems rather small compared to SOTA models.\n\n**References**\n\n[1] Gretton et al., A Kernel Two-Sample Test, JMLR 2012\n\n[2] Dowson et al., The Fréchet Distance between Multivariate Normal Distributions, Jounral of Multivariate Analysis 1982\n\n[3] Heusel et al., GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium, NeurIPS 2017\n\n[4] Lee et al., A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks, NeurIPS 2018\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors observed that in scientific discovery, novel findings are tied to data points that are not in the background distribution. Based on this observation, they utilize anomaly detection with contrastive learning, using huge amounts of good quality simulation data to find anomalous data patterns. With the help of the New Physics Learning Machine (NPLM), they can then test the statistical significance of these findings to determine if the data is truly out-of-distribution. This significance score serves as an important tool in scientific research. All in all, the pipeline is a proper use case of anomaly detection and contrastive learning to help scientific discovery.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper's primary strength is its statistical rigor. The entire pipeline is grounded in a formal hypothesis testing framework that quantifies the significance of any detected novelty through p-values and Z-scores, providing a robust foundation for scientific claims.\n2. A key contribution is the paper's insightful framing of scientific discovery as a statistical anomaly detection problem. This provides a clear and actionable paradigm for automating the search for novel phenomena in large datasets.\n\nWeaknesses:\n1. The empirical validation of the method is limited. Despite showing success on 3 tasks (exclude toy data and CIFAR-10), the applications are drawn exclusively from the domains of physics and astrophysics, which is insufficient to substantiate the paper's broad claims of general scientific utility. Demonstrating efficacy in disparate fields, such as genomics, chemistry, or climate science, would be essential for this.\n2. The paper's conceptualization of \"scientific discovery\" is limited, which narrows the framework's scope and makes its title potentially misleading. The term \"discovery\" often implies not just the detection of a new phenomenon, but also the generation of an explanatory hypothesis for it. The AutoSciDACT framework is tailored for the first step—identifying observational novelty—but its scope does not extend to the crucial subsequent step of building models or inferring the causal mechanisms that would explain the detected anomaly.\n\nQUESTIONS:\n1. In Lines 201-202, the description of the synthetic dataset mentions rotating 'The full M × N dimension space.' This phrasing was slightly confusing, as my understanding is that N represents the number of distinct data clusters, while the data itself exists in a (D+M)-dimensional space. Could you please clarify if the text was intended to state that the full (D+M)-dimensional space is rotated to obscure the signal?\n\n2. In Figure 2 (right panel), in the 2D projection, the anomalous signal class (pink) appears to have a significant visual overlap with one of the background classes (blue). This raises a question about the pipeline's effectiveness in this specific case. Could you elaborate on how detection is achieved here? For example, is the separation clearer in other, un-plotted dimensions of the embedding, or is the NPLM test sensitive enough to detect a subtle overdensity even within this overlapping region?\n\n3. In Figure 3 (d) and (f), in these two panels, the 'supervised' baseline (blue line) consistently outperforms the 'ideal-supervised' baseline (red line), which is the opposite of the trend seen in the other datasets. It's notable that both of these are image-based tasks. Could you offer some insight into this counter-intuitive result?\n\nLIMITATIONS:\nThe authors disscussed the limtations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce AutoSciDACT (Automated Scientific Discovery with Anomalous Contrastive Testing), a new\ntwo-part tool to detect and statistically validate anomalies in large and complex scientific datasets. The tool helps\nsolve two critical challenges in scientific discovery: the high dimensionality and noise typical in real-world data, and\nthe need for rigorous, interpretable statistical testing of potential outliers.\nDuring pre-training, AutoSciDACT utilizes supervised contrastive learning to compress raw data into\nlow-dimensional, semantically meaningful embeddings. This will incorporate some domain knowledge by using\nlabeled data or scientifically relevant augmentations, which provide the necessary complexity reduction while\nkeeping the salient features of the raw data.\nIn the discovery phase, the pipeline applies the New Physics Learning Machine (NPLM), which is a kernel-based,\nsignal-agnostic statistical hypothesis testing methodology. NPLM allows us to compare an \"observed\" dataset to a\n\"reference\" background dataset in order to identify any distributional shifts (e.g., clusters or overdensities) that may\nrepresent new phenomena. The method outputs statistically calibrated significance scores (Z-scores, p-values),\nsupporting robust claims of discovery.\nThe authors evaluated AutoSciDACT against five datasets well within astronomy, particle physics, biology, and\ncomputer vision , demonstrating its domain-agnostic performance and sensitivity to subtle signals—even at sub-1%\nanomaly rates. The authors have also discussed its potential towards detection of dataset shift (e.g., CIFAR-10 vs.\nCIFAR-10.1), suggesting broader utility in model evaluation and data quality assessment.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nQuality\n\n* The paper is technically sound, with well-justified methodological choices. It combines contrastive learning\nand New Physics Learning Machine (NPLM) into a coherent, end-to-end pipeline.\n* Experimental validation is comprehensive across diverse domains (astronomy, physics, histology, synthetic\ndata, and images), demonstrating both robustness and generalizability.\n* Statistical claims are backed by empirical and asymptotic p-values and Z-scores, following good scientific\npractice.\n* The authors are transparent about their assumptions, limitations (e.g., embedding dimension, domain\nshifts), and the scope of the method.\n\nClarity\n\n* The paper is well-organized and readable, particularly for readers familiar with ML.\n* Diagrams, dataset descriptions, and appendix details contribute to a clear presentation of the pipeline and\nexperiments.\n* The method's purpose (scientific novelty detection with statistical rigor) is clearly distinguished from\ngeneral anomaly detection approaches.\n\nSignificance\n\n* This work addresses a critical and underexplored need in machine learning for science: tools that not only\ndetect anomalies but quantify their statistical significance in a scientifically interpretable way.\n\n* AutoSciDACT offers a domain-agnostic framework applicable to any field with high-dimensional, labeled\ndata—making it valuable for physicists, biologists, astronomers, and potentially even ML safety\nresearchers.\n\nOriginality\n\n* While the core components (contrastive learning, NPLM) are based on existing work, their integration into\na unified pipeline tailored for scientific discovery is novel and well-motivated.\n\n* The use of Supervised Contrastive Learning for dimensionality reduction that respects domain labels is an\neffective and underused idea.\n\n* The pipeline is also shown to be extensible: e.g., in Section C, it is repurposed for evaluating dataset shift\nand generative model fidelity, suggesting a broader set of applications beyond discovery.\n\n**Weaknesses**\n\nQuality\n\n* The performance of AutoSciDACT relies heavily on the availability and quality of labeled background\ndata. In real-world domains with scarce or noisy labels, this assumption may limit applicability.\n\n* Although the authors mention potential solutions (e.g., augmentations, domain knowledge), there is no\nquantitative study on robustness to label noise or partial supervision.\n\n* The embedding dimension is fixed to 4 for NPLM compatibility. While this keeps the statistical test\ntractable, it may limit expressiveness, especially in highly multimodal data like CIFAR-10.\n\nSignificance\n\n* Despite strong empirical results, AutoSciDACT currently assumes that the reference and observed\ndistributions are aligned. In real-world applications, domain shift between simulated (or historical)\nreference data and observed data is common, and this limitation is only addressed as future work.\n\n* The application to scientific discovery is exciting, but no real-world discovery example is included—only\nsimulated signals in known datasets. Including or proposing such a use case would strengthen the impact.\n\nOriginality\n\nThe core components (e.g., SupCon, NPLM) are re-used from prior work, and while the combination is\nnovel and useful, the method itself is not radically innovative in terms of algorithmic design.\n\nQUESTIONS:\n1. How robust is AutoSciDACT to label noise or partial supervision in the pre-training phase?\nSince the pipeline relies on supervised contrastive learning using labeled data, it's important to understand how\nperformance degrades with noisy or incomplete labels, especially in domains where precise annotation is difficult.\nSuggestion: Consider including experiments (even synthetic) that measure sensitivity to label corruption or using\nsemi-supervised SupCon variants.\nImpact: Demonstrating robustness (or proposing mitigation strategies) would increase confidence in the method’s\napplicability to less curated domains.\n\n2. Can the method handle domain shift between simulated/reference data and observed data?\nThe pipeline assumes the background data (R) and the test data (D) are drawn from the same distribution under the\nnull hypothesis. However, simulation-to-real gaps or data collection shifts are common in real scientific settings.\nSuggestion: Provide a discussion or ablation showing how mild domain shift affects NPLM outputs or embeddings,\nand whether techniques like domain adaptation or uncertainty modeling can be integrated.\nImpact: If addressed well, this would significantly strengthen the generalizability and practical utility of\nAutoSciDACT.\n\n3. How sensitive is NPLM’s performance to the fixed 4D embedding choice?\nWhile low-dimensional embeddings are necessary for statistical tractability, this constraint may lead to loss of\ninformation, especially for datasets with many complex classes (e.g., CIFAR-10).\nSuggestion: Include a sensitivity study on the embedding dimension (e.g., comparing d = 4, 8, 16) and how it affects\nboth detection performance and computational cost.\nImpact: Clarifying this trade-off can help users balance interpretability vs. expressiveness and would inform how to\nscale AutoSciDACT to larger problems.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article presents an anomaly detection approach that combines contrastive learning with hypothesis testing. The authors propose a general-purpose, two-stage pipeline for detecting novelty in scientific data, aimed at enabling end-to-end automatic scientific discovery. The pipeline is evaluated on datasets from diverse domains, including astronomy, particle physics, histology, and CIFAR datasets.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper introduces a two-stage pipeline combining contrastive learning and hypothesis testing for anomaly detection in scientific data. While the approach is promising, it falls short of demonstrating true generalist capabilities for automated scientific discovery. The methodology and experimental setup raise concerns about generalization, applicability, and evaluation rigor. Several technical choices and omissions limit the strength of the conclusions.\n\n* The method is presented as a generalist approach for Automated Scientific Discovery, but in practice, it appears to be a specialized anomaly detection pipeline targeting a narrow class of anomalies.\n* The paper lacks a comprehensive set of baseline comparisons. Several strong and commonly used anomaly detection methods are omitted, making it difficult to assess the relative performance of the proposed approach.\n* Some settings and claims (e.g., universality across domains) are not well-motivated or empirically validated.\n* The related work section is incomplete and does not adequately position the proposed method within the broader landscape of existing anomaly detection research.\n* The choice of a simple MLP as the supervised baseline is insufficient for meaningful comparison, especially when stronger models could offer a better performance benchmark.\n\nQUESTIONS:\n* To clarify the technical details: for the histology data and CIFAR images, is the embedding dimension fixed to 4? If so, I’m concerned that this setting may not be optimal and could negatively impact the experimental results.\n\nLIMITATIONS:\nSee \"Strengths And Weaknesses\".", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-14"}