Uncertainty-aware machine learning to predict non-cancer human toxicity for the global chemicals market

Domain: physics
Published: 2026-01-07
Editorial outcome: minor_revision

Abstract
Humans are exposed to thousands of chemicals, yet limited toxicity data hinder effective management of their impacts on human health. High-performing machine learning models hold potential for addressing this gap, but their uncharacterized prediction performance across the wider range of chemicals undermines confidence in their results. We develop uncertainty-aware models to predict reproductive/developmental and general non-cancer human toxicity effect doses. Our well-calibrated models provide uncertainty estimates aligned with observed prediction errors and chemical familiarity. We predict toxicity with 95% confidence intervals for >100,000 globally marketed chemicals and identify toxicity and uncertainty hotspots. These results can be applied to inform decisions aimed at reducing potential human health impacts and guide targeted data generation and modeling efforts to reduce prediction uncertainty. Here, we show that enhancing transparency in prediction uncertainty provides key insights for building confidence in toxicity predictions, supporting the sound integration of machine learning-based predictions in chemical assessments.

Introduction
Every day, humans are exposed to a wide range of chemicals through the air, water, food, and countless consumer products. These chemicals can have substantial, yet still largely unknown, toxicity-related impacts on human health1. Safeguarding human health requires a sound scientific basis to assess chemical hazards, establish safe exposure limits, regulate harmful substances, and promote the use of safer alternatives. However, the lack of chemical data for the majority of more than 100,000 marketed chemicals registered worldwide has hampered effective chemical assessment2,3. A key metric in these assessments is the toxicological point of departure (POD), the lowest dose on the dose-response curve where adverse effects begin to manifest. Lower PODs indicate more potent chemicals, as they require a smaller exposure dose to elicit toxic effects. Previous studies derived non-cancer PODs using statistical and probabilistic analysis of numerous in vivo animal studies, providing PODs for a few hundred to several thousand chemicals4–6. This leaves most globally marketed chemicals without data and hence without appropriate estimates of their non-cancer toxicity potential to harm human health.

In silico tools, particularly machine learning (ML) models, are promising approaches to fill chemical toxicity data gaps3,7. They achieve high predictive performance by identifying patterns from large, complex datasets using flexible algorithms such as support vector machines, random forests, gradient boosting machines, and deep learning approaches. However, several challenges hinder the widespread adoption of ML models in chemical assessments. Their performance relies heavily on the quantity and quality of available training data8 and the chosen algorithms and training features9. ML models have limited extrapolative capabilities, restricting their applicability to the domains represented by the training data10,11. Their high flexibility can also lead to overfitting, where models fit to noise or erroneous data rather than underlying patterns12, thereby achieving extraordinarily high prediction accuracy on training data, but significantly lower accuracy on validation datasets. As a result, prediction performance can vary significantly across models and chemicals. However, most ML models lack mechanisms to recognize their own limitations or quantify the uncertainty of their predictions. These limitations undermine confidence in predictions, hindering their integration into decision support tools like chemical risk and health impacts assessments13,14, chemical substitution studies15,16, safe and sustainable-by-design approaches17, and lifecycle assessment of products and technologies18.

To improve confidence in ML predictions, substantial research has focused on understanding and quantifying uncertainty in ML models. ML uncertainty is generally divided into two principal types: aleatoric and epistemic uncertainty19. Aleatoric uncertainty refers to data-related uncertainty arising from the inherent randomness and variability in the available training data, for example, the uncertainty in PODs derived from small samples of observed effects with substantial biological and experimental variability20. Epistemic uncertainty refers to model-related uncertainty caused by a lack of knowledge, which introduces variability in the model-building process, e.g., due to incomplete training data, choice of training features, and uncertain model parameters19. A wider range of uncertainty estimation methods have been developed to date. Some focus on aleatoric uncertainty, such as quantile regression, which provides prediction intervals from empirically estimated quantiles. Other target epistemic uncertainty, for example, ensemble methods that estimate variance across point predictions from a large set of predictors. Individually, these uncertainty estimates can be overconfident as they fail to adequately account for both types of uncertainty. In addition, despite these advancements, uncertainty quantification remains underused in ML model development, particularly outside healthcare and autonomous systems as critical applications21,22.

In this work, we identified uncertainty-aware ML methods (UAMs) that provide quantified confidence intervals with every prediction and developed models for predicting non-cancer PODs for oral exposure. Here, we separate reproductive/developmental from general non-cancer effects to account for the difference in severity at which human lives are impaired23. We aimed to develop UAMs that quantify both aleatoric and epistemic uncertainty, generating robust predictions across a wide range of chemical structures, including unfamiliar or challenging compounds, with confidence intervals that accurately reflect these limitations. To demonstrate the broad applicability of this approach, we used the best-performing models to predict reproductive/developmental and general non-cancer PODs for a large set of >100,000 marketed chemicals, providing initial toxicity estimates for widely used chemicals currently lacking POD data. In addition, we analyzed hotspots of high predicted toxicity and related uncertainty to identify trends across chemical classes and highlight priority classes associated with particularly high non-cancer toxicity potency or low prediction confidence. While prior studies have focused on improving predictive accuracy, few have quantified and validated prediction uncertainty, a key barrier to trust in and uptake of predicted data. This study provides the first comprehensive demonstration of uncertainty-aware toxicity modeling at a chemical space of this scale, enabling transparent predictions that support confidence-building in machine learning–based chemical assessments. Unlike prior studies, which have focused mainly on predictive accuracy, we quantify and validate prediction uncertainty, a key barrier to the uptake of predicted data in decision-making tools.

Results
Uncertainty-aware ML prediction performance and calibration
We compared the prediction performance and uncertainty calibration of two uncertainty-aware ML methods (UAMs) based on frequentist conformal prediction (CP) and probabilistic Bayesian neural networks (BNNs) for estimating two human toxicity endpoints, namely reproductive/developmental (rd) and general non-cancer (nc) PODs, using datasets covering 2357 and 1845 chemicals, respectively, and comparing four different sets of molecular descriptors as training features.

We found that the random forest-based CP models achieved higher prediction performance on test chemicals with existing data and were better calibrated to quantify uncertainty for these chemicals in comparison to the BNN models based on 10-fold cross-validation (see Fig. 1 for CP models trained with RDKit descriptors, results for other models are shown in Supplementary Figs. 7–13). Overall, models with lower prediction errors (higher prediction accuracy) showed lower average prediction uncertainty, demonstrating the integral relationship between accurate predictions and confidence in these predictions. This link is not always directly used to derive uncertainty estimates: While CP models construct uncertainty estimates from observed prediction errors using a calibration set of chemicals with available data, BNN models derive uncertainty from the posterior distribution of model weights, with higher uncertainty making prediction errors more likely. Despite using different methods and descriptors, predictions across models were highly correlated, with Pearson r2 values ranging from 0.61–0.92 for median predictions and 0.17–0.64 for uncertainty estimates (see Supplementary Figs. 14–18). This shows that while models generally produce similar median predictions, their uncertainty estimates can vary more substantially. Still, the mean prediction uncertainty across models was correlated with the variability in median predictions across models (Pearson r2 = 0.41 rd and Pearson r2 = 0.49 nc, see Supplementary Fig. 19). This suggests that the factors contributing to uncertainty within individual models may also drive differences observed across models.

For both endpoints, CP models outperformed BNN models as shown by lower prediction errors and higher coefficients of determination (Fig. 1a, b). Both methods showed lower root mean squared error (RMSE) for POD (RMSE = 0.63, RMSE = 0.69) than for POD (RMSE = 0.73 and RMSE = 0.84), while the coefficients of determination were slightly higher for POD (R2 = 0.55, R2 = 0.40) compared to POD (R2 = 0.41, R2 = 0.30). Thus, model predictions captured a larger fraction of the variance in reported POD, which also exhibited higher variance (1.17) compared to POD (0.69). The lower variance in reported POD stems from fewer chemicals exhibiting substantial reproductive/developmental effects compared to the wider range of general non-cancer effects in our training dataset (see Supplementary Fig. 1). However, it is important to note that the training data did not indicate the absence of toxicity, e.g. for reproductive/developmental effects, and as regression models, the models will always predict a quantitative POD value.

The overall distribution of prediction uncertainty for POD and POD showed median 95% confidence intervals (CIs) spanning 2.2 and 2.7 log10-units (i.e. orders of magnitude) for CP models compared to 2.9–3.7 log10-units for BNN models (Fig. 1c, d). This highlights the greater overall confidence of CP models in their predictions compared to BNN models which aligns with the lower prediction errors observed. For most chemicals, CP models in particular quantified uncertainty that is lower than the general variability observed in the data, where 95% of reported POD and POD values fell within intervals of 3 and 4 log10-units, respectively. The maximum prediction uncertainty for both models (9.7–9.9 and 8.4–9.4 log10-units for POD and POD , respectively) remained near the total reported POD value range (11.4 and 9.3 log10-units). The highest and lowest outer uncertainty bounds were within 0.6 log10-units of the highest and lowest reported POD values, indicating that uncertainties are contained within a reasonable range suggested by the data.

The uncertainty quantification provided by both methods was well calibrated to capture different aspects of uncertainty demonstrated by their confidence-based, error-based, and distance-based calibration.

Confidence-based calibration assessed how often the reported value fell within the predicted confidence interval for the test chemicals with available data and whether this matched the expected confidence level, for example, that 95% of the reported points fell within the predicted 95% CI. Both methods showed strong alignment between the expected and observed fractions of reported data points falling within the predicted CI at different confidence levels ranging from 1 to 99% (Fig. 1e, f). The CP models, designed to guarantee a desired confidence level, performed particularly well with an expected calibration error (ECE), describing the mean absolute deviation between expected and observed fractions, of 1.2% (rd, nc). The BNN models showed consistently higher observed fractions than the expected confidence level with an ECE of 5.0% (rd) and 5.1% (nc), indicating slight underconfidence in their predictions, thus consistently overestimating uncertainty.

Error-based calibration assessed whether the models’ prediction uncertainty (described by root mean uncertainty, RMU) aligned with the observed prediction error (described by RMSE) across sorted batches of increasing prediction uncertainty containing 42–49 chemicals each. Both methods exhibited the expected trend of increasing RMU with increasing RMSE, although the correlation was less pronounced for BNNs predicting POD (Spearman’s ρ = 0.44). The expected normalized calibration error (ENCE), describing the mean absolute deviation between the prediction uncertainty and the observed prediction error, was 16.6% (rd) and 13.4% (nc) for CP models compared to 21% (rd) and 20% (nc) for BNN models (Fig. 1g, h). This demonstrated that for both methods, the models’ confidence in their predictions reflected their actual prediction accuracy, thus capturing overall trends in prediction errors. The BNN models’ trend fell below the identity line, further indicating a more pronounced overestimation of uncertainty.

Distance-based calibration assessed whether higher uncertainty was associated with predicting chemicals that are unlike chemicals seen during training. Both methods showed an increase in chemical dissimilarity (given by the mean Jaccard distance [0,1] to the five nearest training chemicals using Morgan fingerprints) across sorted batches of increasing prediction uncertainty (described by RMU) containing 42–49 chemicals each (Fig. 1i, j). This shows the models’ ability to recognize the increase in epistemic uncertainty in regions with little or no precedent data. The shoulder of the trend suggests that, as a rule of thumb, confident predictions are mainly achievable for chemicals with mean Jaccard distances below 0.5, i.e., chemicals with an average 50% similarity to their five nearest training chemicals.

All in all, while both methods provided well-calibrated uncertainty quantification, the random forest-based CP models were able to fit the observed data better than the BNN models, leading to higher prediction performance and model confidence, thus lower overall uncertainty. This was likely not caused by a lower capacity of BNN’s to fit the data, but rather attributable to their weak assumptions requiring larger amounts of data to find a good approximation19, resulting in larger epistemic uncertainty. In addition, the BNN’s performance may have been constrained by limited exploration of network architectures and hyperparameter tuning, settings to which NN performance is more sensitive than random forest-based models. However, increasing network complexity did not consistently improve prediction performance for conventional NNs, supporting the conclusion that data limitations, rather than model capacity, constrain the NN’s ability to learn a better approximation on our datasets (see Supplementary Fig. 5). Based on the model performance and calibration results, we applied the CP models trained on RDKit descriptors in our further analysis.

Uncertainty-aware ML predictions for more diverse chemicals
Adopting uncertainty-aware ML can be hindered by concerns about sacrificing prediction performance, as these models need to balance reducing prediction errors with providing accurate confidence intervals. We compared the performance of our models with high-performing conventional ML prediction models. For this purpose, we trained five conventional ML algorithms with four types of molecular descriptors and obtained high-performing consensus predictions across these ML models without uncertainty quantification, achieving RMSE = 0.59 [0.53, 0.64] and RMSE = 0.66 [0.63, 0.69]. As another example, Kvasnicka et al. (2024) achieved similar performance with RMSE = 0.58 [0.54, 0.72] and RMSE = 0.69 [0.64, 0.76] by applying random forest models, which were trained using predicted physicochemical properties as features, to a comparable set of standardized chemicals with reported PODs24. The results indicate that the choice of an uncertainty-aware approach led to a minor decrease in prediction accuracy. For POD , the mean difference in squared error (0.05) was not statistically significant (t(9)=1.12, p=0.26, Cohen’s d=0.03, 95% CI [−0.04, 0.15]), indicating comparable predictive performance. For POD , the mean difference in squared error (0.09) represented a statistically significant improvement, although effect sizes remained small (t(9)=2.54, p=0.01, Cohen’s d=0.08, 95% CI [0.02, 0.16]). At the same time, UAMs enabled us to generate predictions with quantified uncertainty for a larger proportion of chemicals than those typically considered within the model’s applicability domain. For instance, based on the threshold of a mean Jaccard distance below 0.5 observed in our results, only 34% of cross-validated chemicals with POD and 28% with POD would fall within the applicability domain.

To investigate the limits of our UAMs’ potential for predicting a more diverse chemical space than applicable for conventional ML models while accurately reflecting varying uncertainty levels, we assessed their performance on a dataset comprising inorganic, metal, and organometallic compounds ‒ chemical classes typically excluded during pre-processing of training datasets due to their non-standard bonding and valencies, which typical molecular features, designed for drug-like chemicals that rarely contain metals, fail to encode. Predictions for these chemicals are thus expected to be highly uncertain. However, if UAMs quantify this uncertainty appropriately, it may reduce the need to exclude non-standardized chemicals during training. This would allow the use of larger, more diverse datasets, further expanding applicability and potentially prediction performance, while also indicating limitations in current training features through uncertainty estimates. We constructed an alternative training dataset that included these non-standardized chemicals, totaling 2873 (rd) and 2225 (nc) compounds. Models were trained using this expanded dataset, incorporating both standardized and non-standardized chemicals, to evaluate their ability to predict uncertainty across a more diverse chemical space. Figure 2a, c show that the predicted POD generally aligned with reported POD data for both standardized and non-standardized chemicals, but the models failed to capture a larger fraction of non-standardized chemicals with a reported POD below 1 mg/kg-d (22% vs. 14% for standardized chemicals), contributing to higher prediction errors (see Supplementary Fig. 20 for POD ). Fig. 2b illustrates that UAMs addressed the challenge of predicting more diverse datasets by effectively adjusting their prediction uncertainty for standardized chemicals (RMU = 0.82) and non-standardized chemicals (RMU = 1.00), reflecting these differences in observed prediction error. While the prediction uncertainty for standardized chemicals tended to be slightly underconfident compared to the observed error (RMSE = 0.72), it was slightly overconfident for the non-standardized chemicals (RMSE = 1.04). The error- and distance-based calibration on non-standardized chemicals exhibited a coherent trend in uncertainty estimation (Fig. 2e, f), comparable to what has been observed for the standardized chemicals in Fig. 1. In addition, the prediction performance for the standardized chemicals was nearly identical to that of models trained on the standardized datasets, demonstrating that training ML models on the expanded dataset allowed them to extract additional information for non-standardized chemicals without compromising standardized chemicals’ predictions though mean prediction uncertainty for standardized chemicals increased slightly from 2.9 to 3.1 log10-units (Fig. 2d). In comparison, when predicting non-standardized chemicals as external test set using the models trained only on standardized datasets, mean prediction uncertainty was 3.9 log10-units as opposed to 3.7 log10-units, despite the models never having observed their limited prediction performance for these chemicals (see Supplementary Figs. 21, 22). In this case, higher prediction uncertainty can be attributed to lower structural similarity between the non-standardized test set and the standardized training set, with a mean Jaccard distance of 0.81 compared to 0.57 among standardized chemicals. This further demonstrates that UAMs can adjust their uncertainty estimates based on both training observations and structural unfamiliarity. However, prediction errors for non-standardized chemicals were substantially higher when predicted with models trained on standardized datasets, leading to overconfident uncertainty estimates and weaker error-based calibration, despite the models increasing their prediction uncertainty. This suggests that uncertainty estimates farther from the training domain should be interpreted as indicative, as the models are forced to extrapolate beyond what they can reliably quantify. It also shows that, although the training features were generally less informative for non-standardized chemicals, they still captured enough structural information, for example, from atom counts, atom fractions, or basic connectivity indices, to enable some degree of predictive learning for these chemicals with the expanded datasets. Nevertheless, prediction uncertainty remained substantially higher for non-standardized chemicals compared to standardized chemicals, limiting the practical utility of these predictions. This highlights that while UAMs can fairly robustly characterize prediction uncertainty for non-standardized chemicals beyond the applicability domain of conventional ML models, tailored strategies are needed for improving prediction performance for these challenging compounds.

Toxicity and uncertainty hotspots for >100,000 marketed chemicals
Based on our findings we finally applied our CP models trained with the standardized and expanded datasets to predict reproductive/developmental and general non-cancer PODs for a large set of >100,000 globally marketed chemicals3. Of these, 126,060 were standardized chemicals, and 8054 were non-standardized chemicals. Results for standardized chemicals are presented here, while provisional results for non-standardized chemicals, marked by high yet potentially underestimated prediction uncertainty, are provided in Supplementary Figs. 24, 25. To spot patterns across marketed chemicals we performed a chemical space analysis following von Borries et al. 3, visualizing predictions in a 2D spatial map using a t-distributed stochastic neighbor embedding (t-SNE)25, that clusters similar chemical structures closely together (see Fig. 3).

Overall, the median toxicity of our set of marketed chemicals was predicted at 43 mg/kg-d for POD and 11 mg/kg-d for POD which aligned with the median toxicity in the reported POD data used for training the models. The prediction uncertainty was lower for POD with the mean 95% CI spanning 2.6 log10-units compared to 3.1 log10-units for POD , reflecting the validated prediction performance (2.2 and 2.8 log10-units, respectively). The slight shift towards higher uncertainty can be explained with a shift towards lower chemical similarity relative to the cross-validated test data (see Supplementary Fig. 26). However, similarity remained within a comparable range and did not approach the extreme unfamiliarity observed during external testing with non-standardized chemicals. Therefore, uncertainty estimates are generally expected to align with the calibration established during cross-validation. The prediction uncertainty was distributed asymmetrically between lower and upper bounds, whose distances from the median PODs were on average −1.7 to −1.8 and +1.0 to +1.3 log10-units. Though most marketed chemicals displayed moderate toxicity with >80% predicted POD and POD falling between 1 and 100 mg/kg-d, these uncertainties allow differentiating toxicity across chemicals compared to the total variability of 6.5 and 8 log10-units across predicted POD and POD , respectively.

Both endpoints showed several clusters associated with elevated predicted toxicity. Notably, two clusters containing organothiophosphates (Fig. 3a, b, clusters 2 and 3) showed high toxicity with predicted POD and POD < 1 mg/kg-d. Additionally, two large clusters containing per- and polyfluorinated alkyl substances (PFAS) (1) and steroids (8) exhibited elevated toxicity < 10 mg/kg-d, in particular for POD . The clusters containing organothiophosphates and PFAS were also predicted with elevated uncertainty with 95% CIs spanning up to 5 log10-units, while many steroids displayed very high uncertainty with 95% CIs exceeding 6 log10-units (Fig. 3c, d). Organothiophosphates, PFAS, and steroids are thus priority classes both for consideration in chemical assessments aimed at limiting potential harm to human health due to their high predicted toxicity, and as targets for improving model confidence, given their high prediction uncertainty.

Several clusters of polyhalogenated compounds were also predicted to have elevated general non-cancer toxicity (Fig. 3b). These include polybrominated and polychlorinated biphenyls (PBBs, PCBs) and diphenyl ethers (PBDEs, PCBEs) (4, 5), chlorinated benzodioxins and benzofurans (6), and polychlorinated cycloaliphatics (7). The first three clusters showed very high prediction uncertainty, with 95% CIs spanning > 6 log10-units (Fig. 3d). These polybrominated and polychlorinated chemical classes are thus also priority classes both for consideration in chemical assessments and as targets for improving model confidence, as they combine high toxicity potency with high prediction uncertainty.

Two additional clusters with notably elevated prediction uncertainty are related to various natural products (9) like alkaloids, polyketides, phenylpropanoids, and lignans, as well as heterocyclic phenothiazines (10). While these natural product classes show only slightly elevated toxicity potency, the substantial prediction uncertainty makes them a priority class for model development efforts aimed at improving model confidence to better assess toxicity potency, particularly given the known biological activity of some chemicals in these clusters like morphinan alkaloids.

Class-based toxicity ranking for >100,000 marketed chemicals
The chemical space analysis showed that there are substantial trends in both toxicity and uncertainty related to different chemical classes. For a systematic chemical class ranking, we used the ClassyFire chemical taxonomy, grouping chemicals by their lowest available classification level (subclass, class or superclass). Fig. 4 highlights the top 25 chemical classes covering at least 30 marketed chemicals ranked by median predicted toxicity. Table 1 details several top-ranked chemical classes highlighting example chemicals, main uses, and observed adverse effects relevant for humans. The top-ranked chemical classes aligned well with highlighted clusters identified in Fig. 3.

Organothiophosphorus compounds, including thiophosphoric acid esters and dithiophosphate o-esters, ranked high for both toxicity endpoints, covering ~300 chemicals with > 40 to 60% in the top 1% of highest overall predicted toxicity. PODs were highest for several globally restricted pesticides, like diazinon, phorate, and ethoprop, and their metabolites, such as phorate oxon, with predicted POD below 0.3 mg/kg-d and POD below 0.1 mg/kg-d. These chemicals inhibit acetylcholinesterase and other enzymes and form reactive oxygen species, able to induce congenital malformations and fetal death26–28, as well as neurotoxicity and specific organ toxicities29,30. This chemical class was represented in the training data with > 20 reported PODs, and showed elevated prediction uncertainty with the median 95% CI spanning 3.3-4.3 log10-units [−1.5-2.0, +1.9-2.1]. Alkyl fluorides, including many PFAS, were ranked with high reproductive/developmental toxicity, covering 444 chemicals with 48.4% in the top 1% of highest predicted toxicity. With a POD around 0.15 mg/kg-d, predicted toxicity was highest for perfluorooctanesulfonic acid which has been restricted under the Stockholm Convention since 2009. PFAS have high thermal stability and repel water and dirt, making them attractive in various applications as surfactants, lubricants, and flame retardants. Recent years have, however, seen increasing public concern for the entire group of PFAS, due to their high persistence and bioaccumulation combined with critical health effects, including reduced fertility and offspring health, leading to a proposed EU-wide ban of all PFAS variants covering 10,000 substances31,32. This chemical class was represented in the training data with nine reported PODs and showed elevated prediction uncertainty with the median 95% CI spanning 3.2 log10-units [−1.3, +2.0].

The highest-ranking classes for general non-cancer effects were benzo-p-dioxins, covering 96 compounds with 60% in the top 1% of predicted toxicity, and the dioxin-like dibenzofurans, covering 185 compounds with 48% in the top 1% of predicted toxicity levels. Polychlorinated variants like 2,3,7,8-Tetrachlorodibenzodioxin (TCDD) had the highest predicted toxicity (POD < 10-5 mg/kg-d), while non-halogenated variants were predicted as less toxic (POD > 10 mg/kg-d). In particular, planar congeners of dioxins and dioxin-like compounds like 2,3,7,8-TCDD show high toxicity by binding to the aryl hydrocarbon receptor (AhR), causing cancer and various non-cancer effects including immunotoxicity, neurological effects, and cardiovascular disease33. Despite strict regulation and declining environmental levels, exposure through food from contaminated areas remains a concern34. Benzo-p-dioxins and dibenzofurans were represented by seven and ten reported PODs, respectively. Both classes exhibited substantial variations in predicted toxicity and associated uncertainty, with the 95% CI spanning from 2.4 to 7.7 log10-units. The high uncertainty likely stems from the wide range of toxicity values within the group, driven in part by activity cliffs, where minor structural changes result in significant shifts in toxicity. The highly ranked group of 43 Strychnos alkaloids with a median POD around 1 mg/kg-d shows that it is not only synthetic chemicals that are predicted with high toxicity. Strychnines are known neurotoxins that act primarily via inhibiting glycine or acetylcholine receptors, and have been used as pesticides, poisons and in traditional medicine. This chemical class was represented in the training data with only a single reported POD and shows high prediction uncertainty with the median 95% CI spanning 4.5 log10-units [−2.4, +2.0].

Our results showed that predictions of top-ranked chemical classes generally aligned well with expectations. However, we also observed that the prediction uncertainty tended to be higher for chemicals predicted with higher toxicity. This can be explained with the significantly lower relative representation of high toxicity chemicals in the training data (0.4% with POD < 0.01 mg/kg-d and 1.5% with POD < 0.01 mg/kg-d), making high toxicity predictions less likely from the model’s perspective and thus decreasing the model’s confidence in these predictions. Other factors that may drive uncertainty in individual predictions can be related to insufficient representation of the unique features characterizing a given chemical class or elevated data-related uncertainty in the underlying training dataset. In addition, activity cliffs, where small structural changes result in significant shifts in toxicity can cause very high uncertainty in model predictions - a phenomenon likely responsible for the high uncertainty observed in dioxin-like compounds. This study incorporated additional data on critical dioxin-like chemicals to enhance prediction accuracy; however, boosting model confidence for these chemicals remained challenging. These findings underscore the urgent need to prioritize efforts to predict chemicals with high toxicity and activity cliffs, for instance, by developing models focused on such critical chemical domains that capture subtle differences in molecular structures to enable more confident toxicity estimates. In addition, Fig. 4 shows that the chemical classification used here is not always useful to group chemicals in terms of their toxicity potency. While some classes show very similar behavior in terms of toxicity levels and associated uncertainty, some class definitions seemed to be either too broad or too refined for grouping chemicals in terms of their toxicity potency. For example, organothiophosphates and PFAS were dispersed across multiple chemical classes, while PCBs were part of the broad class of Biphenyls and derivatives containing more than 1500 chemicals. These findings highlight the need for suitable clustering approaches that group chemicals under consideration of their similarity in structure, use and toxic mode of action. While this would be highly useful to translate prediction results into tangible (regulatory) decision support, such data is currently not available for most marketed chemicals.

Methods
The methodological workflow of this study are outlined in Fig. 5, which provides a schematic overview of the data processing, modeling, and evaluation steps.

Dataset
The toxicity prediction models in this study were built using the extensive datasets of probabilistic PODs derived by Aurisano et al.5 based on curated in vivo effect test data from the US EPA Toxicity Value Database (ToxValDB). The dataset contained PODs for reproductive/developmental (rd) and general non-cancer effects (nc) via chronic oral exposure for more than 10,000 chemicals that can act as surrogates for PODs derived through authoritative assessment. The separate endpoints allow considering the differing severity between lifelong reproductive/developmental impairments and general non-cancer toxicity effects that typically manifest later in life, when for example used in comparative impact assessment23. We complemented the dataset with 26 POD for dioxin-like chemicals that were not yet part of the dataset by leveraging toxic equivalency factors (TEFs) reported by DeVito et al. (2024)51, thereby improving the representation of chemicals with high general non-cancer toxicity. For every chemical in the dataset, chemical structure information in the form of SMILES was collected from the US EPA CompTox Chemicals Dashboard52 and PubChem53, which yielded unique chemical structures for 4832 chemicals with POD and 5370 chemicals with POD . Subsequently, both datasets were filtered to keep PODs that were obtained from at least four underlying experimental data points to reduce aleatoric uncertainty as Aurisano et al. derived surrogate PODs using the 25th percentile. The resulting data sets consisted of 2873 chemicals with POD and 2225 chemicals with POD (expanded dataset). After standardizing chemical structures using the protocol by Mansouri et al. (2018)54 ‒ which includes the removal of inorganic and organometallic compounds, large molecules (> 1000 g/mol), and the stripping of salts and solvents ‒ 2357 unique chemicals with POD and 1,945 with POD were available for model training and validation (standardized dataset). In addition, the expanded dataset was used to assess the potential for achieving robust uncertainty quantification for a more diverse chemical space than applicable for conventional ML models including non-standardized chemicals. The subset of non-standardized chemicals was also used as a challenging external test set for the models trained on standardized chemicals to assess how uncertainty quantification behaves in entirely unfamiliar chemical domains. All PODs were converted from mg/kg-d to log10-transformed units of mol/kg-d prior to training, representing the rational link between molecules and their biological activity55. Prediction results were presented in mg/kg-d for easier interpretation.

Training features
We considered four types of molecular descriptors that numerically describe chemical structures to be used as training features for the ML models. We chose two types of binary fingerprints (MACCS, Morgan) and two types of continuous descriptors (RDKit, CDDD) (see Supplementary Table 1 for details). While MACCS and RDKit descriptors are knowledge-based descriptors that encode structural fragments or properties of chemical structures, Morgan fingerprints and CDDD embeddings provide an unweighted representation of a given chemical’s full structure. We removed highly correlated features (Pearson correlation coefficient > 0.9) from the knowledge-based descriptors but not from the structural representations as these provide a valid representation only when used together. In addition, we applied a Yeo-Johnson transformation on the RDKit descriptors to achieve Gaussian-liked distributions and scaled all RDKit descriptors to zero mean and unit variance. To avoid data leakage in cross-validation, all feature processing parameters were obtained from training sets and applied to respective test sets within each cross-validation fold.

Conventional ML models without uncertainty quantification
We performed a systematic evaluation of conventional ML algorithms without uncertainty quantification to benchmark performance of best-practice ML model development protocols training each algorithm with the four types of molecular descriptors. For our comparison, we selected six algorithms that cover common algorithm families following different functional principles for finding an optimal fit to a given training dataset: Multiple linear regression, k nearest neighbor, support vector machine, random forest, extreme gradient boosting machine and a simple feed forward neural network (see Supplementary section 2.1 for details). For every descriptor-algorithm combination, model performance was assessed through a nested cross-validation procedure with five inner folds to optimize relevant hyperparameters of the respective algorithm and ten outer folds to estimate the generalization error of the model. Chemicals were assigned to each fold using target value-based stratified sampling to compensate impacts from data imbalance. The nested cross-validation was repeated ten times to assess the variation in the generalization error. The generalization error was assessed using mean absolute error (MAE), median absolute error (MdAE), root mean squared error (RMSE) and the (adjusted) coefficient of determination (R2) as performance metrics (see Supplementary Table 4 for details). The assessment allowed us to identify which algorithms, hyperparameter settings and molecular features have high potential to predict PODs from chemical structures. To establish a single, high-performing benchmark for the uncertainty-aware models, we created a consensus model by taking the average across the subset of conventional ML models with the highest statistically significant prediction performances (p < 0.05) using pairwise correlated two-sided t-tests based on ten-times repeated cross-validation (degrees of freedom = 9) (see Supplementary section 2.3 for details).

Uncertainty-aware ML models
To develop uncertainty-aware ML models (UAMs) that provide comprehensive uncertainty quantification covering aleatoric and epistemic uncertainty, we compared two distinct paradigms within ML for quantifying uncertainty, conformal prediction and Bayesian neural networks.

Conformal prediction (CP) is rooted in hypothesis testing following classical, frequentist statistics19. It builds confidence intervals around point estimates based on a desired confidence level thereby controlling the fraction of erroneous predictions with a statistically valid probability. CP offers a model-agnostic and non-parametric approach that can be combined with any algorithm and provides confidence intervals without assuming a specific uncertainty distribution. However, not all CP-based methods provide comprehensive uncertainty quantification. We applied uncertainty-aware conformalized quantile regression (UACQR) developed by Rossellini et al. (2023), which addresses both aleatoric and epistemic uncertainty56. Rossellini et al. implemented their method with four core algorithms: a quantile linear regression, a light gradient boosting machine with quantile loss, a quantile regression forest (qRF) and a quantile regression neural network with two hidden layers. In this study, we used qRF as the core algorithm based on our conventional ML model evaluation that showed RF among the high-performing algorithms, and due to its more comprehensive implementation. The CP models were trained with a 95% confidence level.

Bayesian neural networks (BNNs) apply Bayesian principles to model uncertainty and are the main representative for probabilistic modeling approaches19. BNNs move beyond deterministic outputs by attaching probability distributions to each model parameter (weights and biases) and returning a probability distribution for the output prediction itself. This allow them to effectively capture both aleatoric and epistemic uncertainty. We defined our BNN with one hidden layer with ReLU (rectified linear unit) activation and an output layer with two nodes assigned to the mean and standard deviation of a Gaussian distribution. The prior distributions of the model parameters were defined as the mixture of two zero-mean normal distributions with different variances and the posterior distributions as multivariate normal distribution following Blundell et al. (2015)57. Prediction results were assessed based on random samples of 500 draws from the predicted output distributions.

We assessed the prediction performance and uncertainty calibration of the UAMs through ten-fold cross-validation. Chemicals were assigned to each fold using target value-based stratified sampling to compensate impacts from data imbalance. Due to their unique structures, UAMs were not compatible with conventional hyperparameter optimization tools; hence, reasonable settings were chosen based on experience with equivalent conventional ML models (see Supplementary section 3.2 for details). The overall prediction performance was assessed using the same performance metrics as for the conventional ML models. The quality of the uncertainty quantification was assessed considering three aspects of calibration: confidence-based, error-based, and distance-based calibration. We defined confidence- and error-based calibration following Yang & Li (2023)58. The confidence-based calibration compares the expected confidence level and the observed fraction of measured data points falling within the predicted confidence interval. It can be summarized by the expected calibration error (ECE) which describes the mean absolute deviation between expected and observed fractions (see Supplementary Table 6 for details). The error-based calibration curve compares the consistency between the mean observed prediction error and the mean prediction uncertainty across chemical batches of increasing prediction uncertainty. To form the batches, the n testing data points were sorted by the prediction uncertainty and divided into B batches containing n data points each. The number of batches was set to the entropy-based optimal number of batches obtained as n59. The error-based calibration can be summarized by the expected normalized calibration error (ENCE) defined by Levi et al. (2022)60 describing the mean absolute deviation between observed error and prediction uncertainty across batches (see Supplementary Table 6 for details). As ENCE has been shown to be sensitive towards the chosen number of batches59 we assessed the change in ENCE as a function of number of batches ranging from 5 to 200 (see Supplementary Fig. 6). Inspired by the out-of-distribution analysis by Yin et al. (2023)61, we defined a distance-based calibration to investigate the uncertainty calibration for chemicals that are structurally different from training chemicals. It assesses the consistency between mean structural dissimilarity and mean prediction uncertainty across chemical batches of increasing prediction uncertainty, applying the same batching approach as for error-based calibration. The structural dissimilarity was described by the average Jaccard distance of each test chemical from its five nearest neighbors among training chemicals based on Morgan fingerprints (1024 bits, radius 2). The strength of correlation in all calibration curves was indicated by Pearson and Spearman correlation coefficients (see Supplementary Table 6 for details). The performance of the UAMs was compared to the consensus of the conventional ML models, and the statistical significance of the difference (p < 0.05) was assessed using two-sided student’s t-tests based on tenfold cross-validation results from the available dataset of size n (degrees of freedom = n – 1).

Model application
We applied the best-calibrated models to predict a set of 134,114 marketed chemicals, which we derived in our previous study3 from lists of registered chemicals and specific applications provided through US EPA’s CompTox Chemicals Dashboard v2.1. Before model application, the models were retrained on all available training data using a 20% split for internal model calibration. Using the standardization protocol by Mansouri et al. (2018)54, we obtained standardized structures for 126,060 marketed chemicals, which were then predicted using our models trained on the standardized dataset. Preliminary predictions for the remaining 8054 non-standardized marketed chemicals were generated using models trained on the expanded dataset. We visualized the prediction results using t-distributed stochastic neighbor embedding (t-SNE)25 to map Morgan fingerprints (1024 bits, radius 2) into a two-dimensional chemical space map following von Borries et al. (2023)3. All chemicals were assigned to their chemical taxonomy using ClassyFire to enable chemical class-specific interpretation of results62. Information on chemical class-related main uses, reported effects, and related mechanisms was obtained by screening available publications on Web of Science between March 2024 and January 2025.

Inclusion and Ethics
This research did not directly target or involve participants from resource-poor settings or underrepresented groups. No specific inclusion or exclusion criteria were applicable.

Reporting summary
Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

Data availability
In this work, we relied on previously published and publicly available datasets. To train and validate out machine learning models we used toxicity data reported by ref. 5 and ref. 51. We collected SMILES representations from the US EPA CompTox Chemicals Dashboard (version 2.1, https://comptox.epa.gov/dashboard/, ref. 52, 63) and PubChem (https://pubchem.ncbi.nlm.nih.gov/, ref. 53). Chemical classifications were obtained with the web-based application ClassyFire (http://classyfire.wishartlab.com/, ref. 62). All data used and generated in this study are available in the GitHub repository PODUAM (v1.0.0) [https://github.com/kejbo/PODUAM] under accession code https://doi.org/10.5281/zenodo.1795118164. The predictions for the large set of marketed chemicals generated in this study are also provided as a structured Excel file in the Supplementary Information. Source data are provided with this paper.

Code availability
All modeling, data analysis and visualization were performed with custom code developed in Python 3.11 leveraging publicly available python packages. We used RDKit (version 2022.9.5, https://www.rdkit.org/, ref. 65) to calculate RDKit descriptor, MACCS keys and Morgan fingerprints, and the python package CDDD (version 1.0, https://github.com/jrwnter/cddd, ref. 38) to calculate CDDD embeddings. Conventional machine learning algorithms and training support functions were implemented with scikit-learn (version 1.2.2, https://scikit-learn.org/) and TensorFlow (version 2.13.0, https://www.tensorflow.org/). Uncertainty-aware machine learning algorithms were implemented with the python package UACQR (version 2023-06-08, https://github.com/rrross/uacqr, ref. 56) and TensorFlow Probability (version 0.21.0, https://www.tensorflow.org/probability, ref. 66). Numpy (v1.23.5), Pandas (v1.5.3) and Scipy (v1.10.1) were used for data processing and statistical analysis, and Matplotlib (v3.7.0) and Seaborn (0.13.0) for data visualization. All code generated in this study is available in the GitHub repository PODUAM (v1.0.0) [https://github.com/kejbo/PODUAM] under accession code https://doi.org/10.5281/zenodo.1795118164. The models can also be accessed and applied via a Shiny web app at https://dtu-quantitative-sustainability-assessment.shinyapps.io/poduam.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "The manuscript proceeds along previous studies by this team, addressing the prioritisation and screening of the substances and their impact. This topic is quite important. The attention of this manucript is on the uncertainty when many susbtances are addressed. Indeed, for the majority of them the experimental data are missing. The manuscript covers two main kinds of uncertainty.\nAuthors should address two aspects more deeply. (1) the accuracy of the predictions for the susbtances where data exist and how this can be used to assess uncertainty. (2) the use of different predictive systems, since the predictions can vary depending on them.", "reviewer_2": "The authors developed uncertainty aware models that predict reproductive and non-cancer human toxicity of more than 130K marketed chemicals. The major contributions of this work are presented as 1) reducing human health risks and 2) improving model performance. These contributions appear to be somewhat overstated, as the authors do not present evidence that this work has reduced human health risk (though application of the knowledge produced by the authors certainly has the potential to be applied in such a way that could augment standard practices for identifying potentially toxic chemicals and thereby reduce human health risk), and the generation of 95% confidence intervals in estimating aleatoric and epistemic uncertainty for conformal and Bayesian network-generated predictions lacks novelty. Readers may reasonably interpret results as contributing primarily to a relatively new prediction task that estimates the risk for reproductive and non-cancer human toxicity of chemicals, some of which may not have been previously assessed for toxicity by quantitative methods, which is a considerable contribution, but would seem more appropriate for a more targeted audience.", "reviewer_3": "The study addresses the significant lack of toxicity information for most marketed chemicals by developing uncertainty-aware machine (UAM) learning models. For over 130,000 chemicals, these models predict toxicological points of departure (PODs) for general non-cancer harm as well as reproductive/developmental impacts on a broad scale. The study also identifies chemical groups posing high toxicity risks and areas where prediction confidence is low. In addition, the manuscript is well-organized, and figure 5 provided a very clear overview of the methodological workflow. The statistical analysis is comprehensive and in-depth and the topic will have high impact. While acknowledging these important work, I have a few minor points for consideration,\n1) Although conformal prediction (CP) seems perform better than Bayesian neural networks (BNNs), the performance of BNN may be limited by the network structure, training strategy (the range of candidate hyper-parameters), or data volume.\n2) While developed uncertainty-aware models (UAMs) measure the uncertainty and quantify prediction confidence, the random forests or neural networks are still “black box” models. The predictions are based on correlations in the data and may not fully capture the underlying mechanisms of toxicity. Enhancing the interpretability of the model to help understand the driving factors behind the predictions is important for regulatory applications.\n3) The validation for \"non-standardized\" chemicals are based on training/testing within an expanded dataset. This might not fully represent performance on truly unseen, challenging chemical structures compared to using a dedicated external set of such compounds. If it is feasible, the validation on a truly external dataset would support claims about model generalizability and robustness.\n4) The generation of this extensive prediction dataset with uncertainty and PODs hold significant potential value. Are there any plans to make these predictions and their uncertainties known to the wider scientific community or fully public accessible?\n\nThe overall quality of the repository and its reproducibility is very good. Several cross-validation scripts have been tested and yielded similar results with the reported findings. Here are some suggestions for implementation and replication,\n- The Mac OS users need to install CMake to compile code for some dependencies (e.g. dm-tree). The The README should add instructions on install build tools.\n- The README doc lacks instructions and commands on how to run the experiments. Adding sample commands or scripts would improve readability. For example, adding the command $ python ./manuscript/2-1_performance/run_cv_BNN.py or $ python train_final_models.py would be helpful.\n- For all training files, the replication could be improved by avoiding hardcoding values for parameter settings. Authors could consider use package like argparse to allow for more convenient replication or experiments, or maybe consider use configuration file (e.g., JSON) for easily editing the key parameters.\n- The current scripts assume that hyper-parameters are already considered \"optimal\". The nested cross-validation approach with hyper-parameters optimization has been described in the supplementary section but not mentioned in the repository. The README could provide a navigation to that supplementary section for further implementation details."}, "author_response": "Nature communications manuscript NCOMMS-25-15028– revised version\nDetailed answers to reviewers' suggestions:\nReviewer #1\nGeneral comment: The manuscript proceeds along previous studies by this team, addressing\nthe prioritisation and screening of the substances and their impact. This topic is quite important.\nThe attention of this manuscript is on the uncertainty when many substances are addressed.\nIndeed, for the majority of them the experimental data are missing. The manuscript covers two\nmain kinds of uncertainty. Authors should address two aspects more deeply. (1) the accuracy of\nthe predictions for the substances where data exist and how this can be used to assess\nuncertainty. (2) the use of different predictive systems, since the predictions can vary depending\non them.\nAnswer and action: Thank you for the thoughtful review of our manuscript. The\naccuracy of predictions for substances with available data is shown in Figure 1 for both\nthe training and cross-validated test chemicals. In the manuscript, we have generally\nused the term “prediction error” rather than “prediction accuracy”, as the latter is more\ncommonly associated with classification models. However, we recognize that “accuracy”\nmay be more intuitive for some readers, so we have clarified this terminology in the\nrevised text. We also added an explanation of the relationship between prediction error\n(or “accuracy”) and prediction uncertainty. Finally, we included an additional analysis to\nexamine how strongly predictions and uncertainty estimates correlate across the\ndifferent predictive systems used in this study. The relevant changes are presented\nbelow:\nL136: “We found that the random forest-based CP models achieved higher prediction\nperformance on test chemicals with existing data and were better calibrated to quantify\nuncertainty for these chemicals in comparison to the BNN models based on 10-fold\ncross-validation (see Figure 1 for CP models trained with RDKit descriptors, results for\nother models are shown in SI, Figures S1-7 to S1-13). Overall, models with lower\nprediction errors (higher prediction accuracy) showed lower average uncertainty,\ndemonstrating the integral relationship between accurate predictions and confidence in\nthese predictions. This link is not always directly used to derive uncertainty estimates:\nWhile CP models construct uncertainty estimates directly from observed prediction\nerrors using a calibration set of chemicals with available data, BNN models' uncertainty\nestimates derive from uncertainty in model weights, with higher uncertainty making\nprediction errors more likely.”\nL146: “Despite using different methods and descriptors, predictions across models were\nhighly correlated, with Pearson2 values ranging from 0.61 to 0.92 for median predictions\nand 0.17 to 0.64 for uncertainty estimates (see SI, Figures S1-14 to S1-18). This shows\nthat while models generally produce similar median predictions, their uncertainty\nestimates can vary more substantially. Still, the mean prediction uncertainty across\nmodels was correlated with the variability in median predictions across models\n(Pearson2 =0.41 and Pearson2 =0.49, see SI, Figure S1-19). This suggests that the\nrd nc\nfactors contributing to uncertainty within individual models may also drive differences\nobserved across models.”\nThe following key figures have been added to the supplemental file S1:\n• Figure S1-14 presents heatmaps showing the correlation between prediction\nmedians and uncertainty estimates across prediction systems (detailed scatter\nplots underlying the correlation values are presented in Figures S1-15 to S1-18),\n• Figure S1-19 shows how the variability in median predictions across model\nsystems correlates with the mean prediction uncertainty.\nFigure S1-14. Correlation heatmaps of median predictions and uncertainty estimates (given by 95% CI\nwidth) for reproductive/developmental (left) and general non-cancer (right) PODs across different methods\nand descriptors given by Pearson2 values. Abbreviations: BNN = Bayesian neural network, CDDD =\nCDDD embedding, CI = Confidence interval, CP = Conformal prediction, MACCS = MACCS keys,\nMorgan = Morgan fingerprints, RDKit = RDKit descriptors\nFigure S1-19. Correlation scatter plots with Pearson2 for variability (given as 95% CI width) across median\npredictions by different models and the prediction uncertainty (given by 95% CI width) averaged across\ndifferent models for reproductive/developmental and general non-cancer PODs.\nReviewer #2\nComment: The authors developed uncertainty aware models that predict reproductive and non-\ncancer human toxicity of more than 130K marketed chemicals. The major contributions of this\nwork are presented as 1) reducing human health risks and 2) improving model performance.\nThese contributions appear to be somewhat overstated, as the authors do not present evidence\nthat this work has reduced human health risk (though application of the knowledge produced by\nthe authors certainly has the potential to be applied in such a way that could augment standard\npractices for identifying potentially toxic chemicals and thereby reduce human health risk), and\nthe generation of 95% confidence intervals in estimating aleatoric and epistemic uncertainty for\nconformal and Bayesian network-generated predictions lacks novelty. Readers may reasonably\ninterpret results as contributing primarily to a relatively new prediction task that estimates the\nrisk for reproductive and non-cancer human toxicity of chemicals, some of which may not have\nbeen previously assessed for toxicity by quantitative methods, which is a considerable\ncontribution, but would seem more appropriate for a more targeted audience.\nAnswer and action: Thank you for your thoughtful review and for highlighting your\nconcerns regarding our manuscript. We appreciate the opportunity to clarify our\ncontribution and have revised the text accordingly.\nOur main objective has been to help close the substantial data gap on non-cancer\nhuman toxicity effects by developing predictive models and generating data with\nquantified uncertainty for globally marketed chemicals. The purpose of developing\nuncertainty-aware ML models with quantified uncertainty was not to improve model\nperformance per se, but rather make prediction performance across chemicals more\ntransparent. While we agree that the application of Conformal Prediction and Bayesian\napproaches is not novel in itself, they are still not widely adopted in cheminformatics or\nmany other ML application areas. We believe that our application-focused evaluation of\nthese methods provides value both to developers of such algorithms and to domain\nexperts building or using chemical prediction models, which is one reason why we reach\nfor a broader audience.\nAs pointed out in your review, by identifying chemical classes with high predicted toxicity\nand/or uncertainty, our results can be applied to: a) inform decisions to limit the use of\npotentially harmful chemicals without toxicity estimates based on experimental data,\nand\nb) guide efforts to improve model predictions for chemical classes with high prediction\nuncertainty – an opportunity for which we discuss different targeted data generation and\nmodelling strategies.\nWe have revised the manuscript to further clarify this distinction between main\ncontributions and intended applications and to prevent any potential misunderstanding.\nAbstract\nL39: “These results can be applied to inform decisions aimed at reducing potential\nhuman health impacts and guide data generation and modelling efforts to reduce\nprediction uncertainty.”\nHighlights\nL63: “Identified hotspots of human toxicity and uncertainty can inform decisions to limit\nthe use of potentially hazardous chemicals as well as guide future targeted data\ngeneration and modelling efforts to reduce uncertainty in predicted toxicity data.”\nIntroduction\nL99: “To improve confidence in ML predictions, substantial research has focused on\nunderstanding and quantifying uncertainty in ML models. ML uncertainty is generally\ndivided into two principal types: aleatoric and epistemic uncertainty22. Aleatoric\nuncertainty refers to data-related uncertainty arising from the inherent randomness and\nvariability in the available training data, for example, the uncertainty in PODs derived\nfrom small samples of observed effects with substantial biological and experimental\nvariability23. Epistemic uncertainty refers to model-related uncertainty caused by a lack\nof knowledge, which introduces variability in the model-building process, e.g., due to\nincomplete training data, choice of training features, and uncertain model parameters22.\nA wide range of uncertainty estimation methods have been developed to date. Some\nfocus on aleatoric uncertainty, such as quantile regression, which provides prediction\nintervals from empirically estimated quantiles. Others target epistemic uncertainty, for\nexample, ensemble methods that estimate variance across point predictions from a large\nset of predictors. Individually, these uncertainty estimates tend to be overconfident as\nthey fail to adequately account for both types of uncertainty. In addition, despite these\nadvancements, uncertainty quantification remains underused in ML model development,\nparticularly outside healthcare and autonomous systems as critical applications19,20.\nL125: “In addition, we analyzed hotspots of high toxicity and related uncertainty to\nidentify trends across chemical classes and highlight priority classes associated with\nparticularly high non-cancer toxicity potency or low prediction confidence.”\nResults\nL375: “Organothiophosphates, PFAS, and steroids are thus priority classes both for\nconsideration in chemical assessments aimed at limiting potential harm to human health\ndue to their high predicted toxicity, and as targets for improving model confidence, given\ntheir high prediction uncertainty.”\nL383: “These polybrominated and polychlorinated chemical classes are thus also priority\nclasses for consideration in chemical assessment and as targets for improving model\nconfidence as they combine high toxicity potency with high prediction uncertainty.”\nL389: “While these natural product classes show only slightly elevated toxicity potency,\nthe substantial prediction uncertainty makes them a priority class for model development\nefforts aimed at improving model confidence to better assess toxicity potency,\nparticularly given the known biological activity of some chemicals in these clusters like\nmorphinan alkaloids.”\nDiscussion\nL517: “Our models and predicted POD data were developed for application in support of\nbroader chemical impact and risk assessments across marketed chemicals by\ncombining our non-cancer human toxicity predictions with exposure estimates from\nexperimental data or exposure models.”\nL547: “Though uncertainty quantification on its own does not reduce prediction\nuncertainty, it offers key insights for model developers, helping to identify the factors\ndriving uncertainty and enabling to improve a model’s generalizability and reduce\nprediction uncertainty.”\nReviewer #3\nGeneral comment: The study addresses the significant lack of toxicity information for most\nmarketed chemicals by developing uncertainty-aware machine (UAM) learning models. For over\n130,000 chemicals, these models predict toxicological points of departure (PODs) for general\nnon-cancer harm as well as reproductive/developmental impacts on a broad scale. The study\nalso identifies chemical groups posing high toxicity risks and areas where prediction confidence\nis low. In addition, the manuscript is well-organized, and figure 5 provided a very clear overview\nof the methodological workflow. The statistical analysis is comprehensive and in-depth and the\ntopic will have high impact. While acknowledging these important work, I have a few minor\npoints for consideration,\nAnswer: Thank you for the thoughtful review of our manuscript and the many valuable\nsuggestions. We have carefully considered each of your comments and provide detailed\npoint-by-point responses below.\nComment 1: Although conformal prediction (CP) seems [to] perform better than Bayesian\nneural networks (BNNs), the performance of BNN may be limited by the network structure,\ntraining strategy (the range of candidate hyper-parameters), or data volume.\nAnswer and action: We agree that BNNs are very powerful and could perform equally\nwell, if not better than, CP models. Since increasing network complexity combined with\nmore extensive hyperparameter tuning to conventional NNs did not substantially improve\nprediction performance on our data set, we concluded that the performance of both NNs\nand BNNs in this work is primarily constrained by data availability. We expanded this\nexplanation in the manuscript and included a supplementary figure (S1) illustrating\nprediction performance across wider and deeper architectures.\nL217: “This was likely not caused by a lower capacity of BNN's to fit the data, but rather\nattributable to their weak assumptions requiring larger amounts of data to find a good\napproximation, resulting in larger epistemic uncertainty. In addition, the BNN’s\nperformance may have been constrained by limited exploration of network architectures\nand hyperparameter tuning, settings to which NN performance is more sensitive than\nrandom-forest based models. However, increasing network complexity combined with\nhyperparameter tuning did not consistently improve prediction performance for\nconventional NNs, supporting the conclusion that data limitations, rather than model\ncapacity, constrain the BNNs ability to learn a better approximation on our datasets (see\nSI, Figure S1-5).”\nFigure S1-5. Changes in prediction error (given by RMSE) for different neural network architectures in\nterms of number and size of hidden layers. Hidden layer sizes are defined relative to the input size (p) as\nmultiples of p. The figure shows that changes towards deeper and wider architectures did not lead to\nsubstantial reductions in prediction error, in particular for cross-validated test set performance. Boxplots\nshow the interquartile range (boxes) with whiskers extending to the 2.5th and 97.5th percentiles.\nComment 2: While developed uncertainty-aware models (UAMs) measure the uncertainty and\nquantify prediction confidence, the random forests or neural networks are still “black box”\nmodels. The predictions are based on correlations in the data and may not fully capture the\nunderlying mechanisms of toxicity. Enhancing the interpretability of the model to help\nunderstand the driving factors behind the predictions is important for regulatory applications.\nAnswer and action: We fully agree that improving interpretability of ML-based\npredictions is a critical challenge, particularly for regulatory applications. This challenge\nbecomes even greater when developing models intended to cover a broad chemical\nspace, as toxicity mechanisms can differ substantially across domains. Similar to how\nuncertainty-aware ML shifts from defining a global applicability domain to estimating\nuncertainty for individual predictions, mechanistic interpretation will also need to occur at\nthe level of individual predictions, which can be addressed by explainable AI (XAI)\nmethods. Combining uncertainty-aware ML with XAI is an important future direction but\nfalls beyond the scope of this manuscript. However, we have included general feature\nimportance of our models in supplemental figure S1- 23 and expanded the discussion to\nacknowledge the need for future research in this area.\nL539: “In addition, enhancing the interpretability of UAMs will be essential for increasing\ntheir acceptance, particularly in regulatory contexts. While overall feature importance\ncan be easily obtained (see SI, Figure S1-23), it is insufficient to interpret predictions in\nterms of underlying toxicity mechanisms. Combining UAMs with explainable artificial\nintelligence (XAI) techniques, such as Shapley values, surrogate modelling or\ncounterfactuals61, could provide valuable insights into both toxicity mechanisms and\nuncertainty drivers at the level of individual predictions. However, this integration may\nface challenges due to the added complexity, computational demands, and technical\nhurdles of combining both approaches.”\nFigure S1-23. Feature importance distributions from 1,000 quantile regression forests in the CP models\ntrained with the standardized datasets, shown for the 50 most important features ranked by mean importance\nfor PODrd (left) and PODnc (right).\nComment 3: The validation for \"non-standardized\" chemicals are based on training/testing\nwithin an expanded dataset. This might not fully represent performance on truly unseen,\nchallenging chemical structures compared to using a dedicated external set of such\ncompounds. If it is feasible, the validation on a truly external dataset would support claims about\nmodel generalizability and robustness.\nAnswer and action: Thank you for this interesting suggestion. Our primary goal in\nassessing UAM performance on an expanded training dataset was to evaluate whether\nthese models can learn from a more diverse chemical space than conventional ML\nmodels, while accurately reflecting varying levels of uncertainty. This was of interest, as\nit could reduce the need to exclude non-standardized chemicals from training, enabling\nlarger and more diverse datasets, thereby further expanding prediction applicability.\nStress-testing the models by predicting non-standardized chemicals exclusively as an\nexternal test set indeed provides additional insight into how effectively uncertainty\nestimates can serve as a warning when predictions move far outside the trained domain.\nWe agree this analysis could strengthen the manuscript and have added clarifications on\nour initial approach and our additional findings as presented below.\nL266: “To investigate the limits of our UAMs’ potential for predicting a more diverse\nchemical space than applicable for conventional ML models while accurately reflecting\nvarying uncertainty levels, we assessed their performance on a dataset comprising\ninorganic, metal, and organometallic compounds ‒ chemical classes typically excluded\nduring pre-processing of training datasets due to their non-standard bonding and\nvalencies, which typical molecular features, designed for drug-like chemicals that rarely\ncontain metals, fail to encode. Predictions for these chemicals are thus expected to be\nhighly uncertain. However, if UAMs quantify this uncertainty appropriately, it may reduce\nthe need to exclude non-standardized chemicals during training. This would allow the\nuse of larger, more diverse datasets, further expanding applicability and potentially\nprediction performance, while also indicating limitations in current training features\nthrough uncertainty estimates.”\nL297: “In comparison, when predicting non-standardized chemicals as external test set\nusing the models trained only on standardized datasets, mean prediction uncertainty\nwas 3.9 log10-units as opposed to 3.7 log10-units, despite the models never having\nobserved their limited prediction performance for these chemicals (see SI Figure S1-\n21,22). In this case, higher prediction uncertainty can be attributed to lower structural\nsimilarity between non-standardized test set and standardized training set, with a mean\nJaccard distance of 0.81 compared to 0.57 among standardized chemicals. This further\ndemonstrates that UAMs can adjust their uncertainty estimates based on both training\nobservations and structural unfamiliarity. However, prediction errors for non-\nstandardized chemicals were substantially higher when predicted with models trained on\nstandardized datasets, leading to overconfident uncertainty estimates and weaker error-\nbased calibration, despite the models increasing their prediction uncertainty. This\nsuggests that uncertainty estimates farther from the training domain should be\ninterpreted as indicative, as the models are forced to extrapolate beyond what they can\nreliably quantify. It also shows that, although the training features were generally less\ninformative for non-standardized chemicals, they still captured enough structural\ninformation, such as atom counts, fractions, or basic connectivity indices, to enable\nsome degree of predictive learning for these chemicals with the expanded datasets.”\nL356: “The slight shift towards higher uncertainty can be explained with a shift towards\nlower chemical similarity relative to the cross-validated test data (see SI, Figure S1-26).\nHowever, similarity remained within a comparable range and did not approach the\nextreme unfamiliarity observed during external testing with non-standardized chemicals.\nTherefore, uncertainty estimates are generally expected to align with the calibration\nestablished during cross-validation.”\nFigure S1-21. External prediction performance and uncertainty calibration for the non-standardized test set\npredicted by CP models trained with the standardized dataset for POD . (a) Prediction performance on the\nrd\nchallenging, external test data set of non-standardized chemicals (b) The mean prediction uncertainty\n(RMU) and observed prediction errors (RMSE) for non-standardized chemicals compared to standardized\nchemicals (during cross-validation). (c) Distribution of prediction uncertainty (95% CI width) for non-\nstandardized chemicals compared to standardized chemicals (during cross-validation) (e) Confidence-based\ncalibration curve for non-standardized chemicals (f) Error-based calibration curve for non-standardized\nchemicals. Abbreviations: ENCE=expected normalized calibration error, CI=confidence interval,\nCV=cross-validation, Jaccard=mean Jaccard distances from 5 nearest neighbors, MAE=mean absolute\nerror, MdAE=median absolute error, n=number of chemicals, Pearson=Pearson correlation coefficient,\nPOD=point of departure, RMSE=root mean squared error, RMU=root mean uncertainty, R2=coefficient of\ndetermination, Spearman=Spearman correlation coefficient\nFigure S1-22. External prediction performance and uncertainty calibration for the non-standardized test set\npredicted by CP models trained with the standardized dataset for POD . (a) Prediction performance on the\nnc\nchallenging, external test data set of non-standardized chemicals (b) The mean prediction uncertainty\n(RMU) and observed prediction errors (RMSE) for non-standardized chemicals compared to standardized\nchemicals (during cross-validation). (c) Distribution of prediction uncertainty (95% CI width) for non-\nstandardized chemicals compared to standardized chemicals (during cross-validation) (e) Confidence-based\ncalibration curve for non-standardized chemicals (f) Error-based calibration curve for non-standardized\nchemicals. Abbreviations: ENCE=expected normalized calibration error, CI=confidence interval,\nCV=cross-validation, Jaccard=mean Jaccard distances from 5 nearest neighbors, MAE=mean absolute\nerror, MdAE=median absolute error, n=number of chemicals, Pearson=Pearson correlation coefficient,\nPOD=point of departure, RMSE=root mean squared error, RMU=root mean uncertainty, R2=coefficient of\ndetermination, Spearman=Spearman correlation coefficient\nComment 4: The generation of this extensive prediction dataset with uncertainty and PODs\nhold significant potential value. Are there any plans to make these predictions and their\nuncertainties known to the wider scientific community or fully public accessible?\nAnswer and action: Thank you for recognizing the potential value of this work. Indeed,\nthe predictions for the >100,000 marketed chemicals, along with the trained models, will\nbe made fully and openly available. We will provide complete code and documentation\non GitHub (https://github.com/kejbo/poduam) , as well as a Shiny web application\n(https://dtu-quantitative-sustainability-assessment.shinyapps.io/poduam) that allows\nusers to run the models and explore predictions directly in the browser without requiring\nlocal installation. Details are provided in the Data Availability and Code Availability\nsections.\nComments on Code availability: The overall quality of the repository and its reproducibility is\nvery good. Several cross-validation scripts have been tested and yielded similar results with the\nreported findings. Here are some suggestions for implementation and replication,\n- The Mac OS users need to install CMake to compile code for some dependencies (e.g. dm-\ntree). The README should add instructions on install build tools.\n- The README doc lacks instructions and commands on how to run the experiments. Adding\nsample commands or scripts would improve readability. For example, adding the command $\npython ./manuscript/2-1_performance/run_cv_BNN.py or $ python train_final_models.py would\nbe helpful.\n- For all training files, the replication could be improved by avoiding hardcoding values for\nparameter settings. Authors could consider use package like argparse to allow for more\nconvenient replication or experiments, or maybe consider use configuration file (e.g., JSON) for\neasily editing the key parameters.\n- The current scripts assume that hyper-parameters are already considered \"optimal\". The\nnested cross-validation approach with hyper-parameters optimization has been described in the\nsupplementary section but not mentioned in the repository. The README could provide a\nnavigation to that supplementary section for further implementation details.\nAnswer and action: Thank you very much for your detailed review of our code. Based\non your suggestions, we have improved the README file to include instructions for Mac\nusers and a clearer overview of the analysis structure, indicating where each part can be\nfound and how to execute the scripts in the correct order. The scripts were developed\nand tested in an IDE (VS Code), which facilitates adjusting settings at the beginning of\neach script. To improve the compatibility with command-line execution, we have\nimplemented argument parsing in all scripts where users need to select between\ndifferent dataset configurations."}, {"round": 2, "reviewer_comments": {"reviewer_1": "The revised manuscript clarified the opne points. The manuscript is well-structured and explicative.", "reviewer_2": "The revised manuscript now describes the potential to apply results from the manuscript to inform decisions regarding limitations on use of potentially harmful chemicals and the potential to guide future improvements in predictive performance for chemical classes that are difficult to predict due to high uncertainty. These revisions incompletely address concerns regarding relatively weak novelty and the important potentials described in the revision – though testable – remain theoretical at this stage of investigation. This is important work that when carried forward to realize the above potentials will represent a major scientific contribution.", "reviewer_3": "The authors have addressed all my concerns. I appreciate their thorough response and efforts to improve the manuscript. In addition, the code has been significantly improved in clarity and documentation. I recommend this manuscript for publication."}, "author_response": "Nature communications manuscript NCOMMS-25-15028– revised version\nDetailed answers to reviewers' comments:\nReviewer #1\nComment: The revised manuscript clarified the open points. The manuscript is well-structured\nand explicative.\nAnswer: Thank you for the positive evaluation of our revised work. We are pleased that\nwe successfully addressed all of your comments and concerns.\nReviewer #2\nComment: The revised manuscript now describes the potential to apply results from the\nmanuscript to inform decisions regarding limitations on use of potentially harmful chemicals and\nthe potential to guide future improvements in predictive performance for chemical classes that\nare difficult to predict due to high uncertainty. These revisions incompletely address concerns\nregarding relatively weak novelty and the important potentials described in the revision – though\ntestable – remain theoretical at this stage of investigation. This is important work that when\ncarried forward to realize the above potentials will represent a major scientific contribution.\nAnswer and action: Thank you for your thoughtful comments and for recognizing the\nprincipal scientific contribution of our work. We agree that further evaluation of the\npotential to guide reductions in human health impacts and to improve effect data\npredictions in high-uncertainty chemical domains would provide substantial additional\nvalue. However to arrive at risk or health impact level, extensive new data collection\nefforts would be required – such as chemical production data, data on chemical use in\nconsumer products and industrial processes, indoor/outdoor fate and distribution, and\nhuman exposure data to rank health risks from predicted chemicals – as well as\ngenerating new experimental data in chemical domains with high uncertainty to improve\nprediction accuracy in these domains. We consider these important aspects that require\nsubstantial additional research effort, while they can already build on and be combined\nwith our effect predictions as an essential component.\nTo address these concerns, we added a paragraph in the Introduction emphasizing the\nnovel aspects of our work. In the Discussion, we further clarified why demonstrating the\npotential applications for human health risk and impact assessments was beyond the\nscope of the present study.\nIntroduction\nL100-105: “This study provides the first comprehensive demonstration of uncertainty-\naware toxicity modeling at a chemical space of this scale, enabling transparent\npredictions that support confidence-building in machine learning–based chemical\nassessments. Unlike prior studies, which have focused mainly on predictive accuracy,\nwe quantify and validate prediction uncertainty, addressing a key limitation to the uptake\nof predicted data in decision-making tools.”\nDiscussion\nL424-447: “Our models and predicted POD data were developed for application in\nsupport of broader chemical impact and risk assessments across marketed chemicals.\nSuch assessments can build on our non-cancer human toxicity predictions as an\nessential component and combine them with additional information to derive risk or\nhealth impact levels, requiring data on chemical production, chemical use in consumer\nproducts and industrial processes, indoor and outdoor environmental fate, and human\nexposure, to quantify chemical-related health risks. For this application, the provided\nquantitative confidence intervals prevent overconfidence in prediction results and allow\nverifying if the associated uncertainty can alter or hinder robust conclusions from such\nchemical assessments. Depending on the decision context, assessments can leverage\nthe best estimates and associated confidence intervals for robust comparative analysis,\nas in life cycle assessments - or take a more protective approach by considering the\nupper toxicity bounds to establish safety margins, as is common when assessing human\nhealth risks. This is a substantial advancement in departing from the \"no data, no\nproblem\" mindset toward assessing chemical toxicity potency as needed based on the\nmost likely or conservative estimates while leveraging the same data source. In addition,\nthe uncertainty quantification provided can help prioritize experimental studies to reduce\nuncertainty in challenging chemical domains, iteratively improving the reliability of risk or\nhealth impact level assessments. This is an essential step toward harmonizing data\ngeneration for chemical risk and sustainability assessments, addressing potentially\nconflicting results between these two approaches36 – an increasingly important\nchallenge in the context of integrated frameworks such as the EU Commission’s Safe\nand Sustainable by Design.17 While implementing such integrated assessments is\nbeyond the scope of the current study, our well-calibrated predictions and confidence\nintervals provide a solid foundation for future studies to carry out comprehensive risk and\nimpact evaluations.”\nReviewer #3\nComment: The authors have addressed all my concerns. I appreciate their thorough response\nand efforts to improve the manuscript. In addition, the code has been significantly improved in\nclarity and documentation. I recommend this manuscript for publication.\nAnswer: Thank you for the positive evaluation of our revised work. We are pleased that\nwe successfully addressed all of your comments and concerns."}]

Decision letter