Active Learning for Optimal Minimization of Experimental Characterization Uncertainty

Authors: marcus  schwarting
nathan  seifert
logan  ward
ben  blaiszik
ian  foster
yuxin  chen
kirill  prozument
Venue: NeurIPS_cc_2024_Workshop_BDU
Published: 2024-08-29 15:45:19

Abstract: Collecting experimental measurements is rarely an end in itself; rather, measurements inform key outcome statistics. Standard active learning procedures can drive a cumulative decrease in measurement uncertainty, but do not account for the uncertainty of the outcome. Here we present an active learning framework that operates to minimize the uncertainty of the outcome, and demonstrate its applicability with imaging and spectroscopic tasks. We show how our framework can effectively select regions for measurement without iteratively retraining a model. We conclude with two instances where our framework has outperformed standard active learning procedures to accelerate the classification of unknown samples.

Main paper: 
 introduction: 
 Accurate characterization of an unknown chemical or material can be a time-and resource-intensive process, especially in a context-free setting. Characterizing a sample often requires obtaining multiple individual noisy measurements that, when combined, yield the desired identifying information. Effectively allocating resources to characterizing a sample is an important challenge with many applications. We present a novel active learning (AL) approach for negotiating the trade-off between time and precision that allows a researcher to characterize a sample reliably with fewer measurements. We start with a pre-trained supervised model with built-in uncertainty quantification (UQ), and iteratively select characterization routines to reduce model uncertainty optimally. While previous works select from a broad set of measurements to improve retrained model performance (and often do not repeat measurements), our approach accounts for even extremely noisy individual measurements to drive towards minimizing classification uncertainty with a fixed trained model.Spectroscopy is an important area of analytical chemistry where a typical workflow involves iteratively recording noisy measurements until a domain expert can draw conclusions about the contents of a sample. AL has been applied for certain spectroscopic sub-tasks [21] , but has never before been applied to rotational spectroscopy. Rotational spectroscopy uses microwave and millimeter-wave frequencies (ca. 2-800 GHz) to probe transitions between the quantized energy levels of molecules associated with their overall angular momentum [35] . The observed rotational spectrum is extremely sensitive to molecular structure and intramolecular interactions. Rotational spectroscopy is unique in both the structural insights its spectra provide and the rich data it carries for machine learning (ML) applications [38] . Rotational spectra can be acquired via broadband or narrowband measurements. Broadband rotational spectroscopy [3, 33] has enabled researchers to acquire thousands of rotational transitions in a single measurement, with simultaneous acquisition of ∼10 5 resolution elements at a typical signal-to-noise ratio (SNR) of ∼10 4 [25, 36] . In broadband rotational spectroscopy, transitions from an entire spectral band can be acquired almost instantaneously, with repeated measurement averaging eventually leading to an acceptable SNR [37] . Alternatively, a full spectrum can be obtained from multiple sequential narrowband measurements, where a decrease in bandwidth corresponds to an increase in SNR at a rate of (∆ν) -1 2 [3] . For a fixed measurement SNR, there is no statistical advantage to either averaging broadband measurements or sequential narrowband measurements [23] . However, when an outcome is based on decreasing molecule identification uncertainty, rather than decreasing measurement uncertainty, we find that our AL framework can efficiently select narrowband measurements to reduce classification uncertainty.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Our abstract outlines our active learning framework, which we claim outperforms other active learning approaches for active noise acquisition in an experimental setting.We substantiate these findings throughout the rest of the work. 
background: 
 Methods for UQ can use an ensemble (Monte Carlo)-based approach or an incorporated Bayesian approach. For a classification task, a model with UQ will generate a probability distribution for each class, centered at the predicted class probability. Classical ML models such as random forest can give both a prediction and UQ derived from the ensemble. A close deep learning equivalent can be naively achieved by training an ensemble of feed-forward architectures, or via a dropout regularization approach. For an overview of ensemble deep learning approaches for UQ, see Gawlikowski et al. [13] . Alternatively, UQ can be a baked-in part of model training and inference, as in Gaussian process regression (or kriging) [28] . After setting a statistical prior based on previously collected data, a Gaussian process regressor makes calibrated predictions from inputs, including both a prediction mean and a standard deviation [11] . For more information on models that incorporate UQ as part of inference, see Abdar et al. [1] .Data acquisition using AL can be divided into approaches that iteratively retrain the prediction model and those that keep the prediction model fixed [31] . The former is often referred to as active instance labeling (e.g., [34] ), and the latter as active feature evaluation (e.g., learning with feature cost [17] , interactive troubleshooting [7] and medical diagnosis [2, 6] ). When the model is iteratively updated, the objective is to select points to improve model performance [27] . When the model is fixed, active learning operates on a fixed pre-trained model where decisions must be made in serial to come to a conclusion with high confidence [5, 26] .Whether a UQ model is fixed or iteratively updated, selection of the acquisition function can significantly alter the progress of an AL approach. Consider a classification task with N classes, and where T samples are drawn iid from a Bayesian prior. For a fixed patch index i ∈ {1, ..., r}, we can establish probabilities P i,t = [P i,t 1 , ..., P i,t N ]; t ∈ {1, ..., T }. Furthermore, for class n, let P i n = 1 T T t=1 P i,t n . The acquisition functions can be broken into information theoretic approaches (based on entropy) and standard deviation approaches [12] . We focus on four acquisition functions:• Predicted information entropy [32] , defined as ENTR(P i ) = -N n=1 P i n log P i n . • Bayesian AL by disagreement, or BALD [15, 18] -akin to an expected entropy improvement metric. BALD(P i ) = ENTR(P i ) + 1 T N n=1 T t=1 P i,t n log P i,t n .• Mean standard deviation [16] , defined as MSTD(P i ) = 1 N N n=1 1 T T t=1 (P i,t n -P i n ) 2 .• Baseline uniform acquisition, where UNIF(P i ) is drawn i.i.d. from U (0, 1).Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Within the results section, we discuss how there is a window where the performance of our active learning framework excels when collecting measurements. Outside of this window, we discuss how other techniques may be better suited. 
methods: 
 We start with a demonstration using the MNIST dataset of 28×28-pixel gray-scale images of handwritten digits [8] . We train a feed-forward neural network on MNIST with a standard 80/20 train/test split, augmenting training images with added Gaussian noise [30] . We break a 28×28 image from the test set into 4×4 image patches. For a ground-truth patch x i ∈ R 16 , we can draw measurements from a "noisy oracle" as xi = x i + ε; ε ∼ N (0, Σ M ). We use a multivariate Gaussian prior π with mean µ and covariance Σ for each 4×4 patch from the MNIST training set (see Appendix B). Comparing central vs. perimeter patches suggests that patches with higher prior variance will be more useful to measure (satisfying a pure Bayesian exploration objective). But prior variance cannot be the only selection criterion, since measurements of patches with high prior variance need not reduce classification uncertainty.Next we show a similar demonstration with rotational spectroscopy. We select the molecule propene (CH 3 CHCH 2 ) as our ground truth class for subsequent tests [20, 24] . We then create similar classes of hypothetical molecules with spectra resembling propene. Additional spectroscopic details, along with how these similar spectral classes were created and augmented for training robust classifiers, is given in Appendix A. We choose to simulate spectra in the 65-85 GHz range, which corresponds to a common experimental apparatus range [37] . Also, we bin and renormalize all spectra into either b = 20 or b = 200 equally sized patches. While a variety of supervised classification models could be readily employed, we opted for a logit classifier with a one-versus-rest scheme. Regardless of the model choice, the result is expressed as M : R b → [P 1 , ..., P N ]. Finally, we use the mean vector and covariance matrix across all molecule spectra in the PC9 dataset of small organic molecules [14] .When querying a noisy oracle, we assume samples are drawn from a multivariate Gaussian likelihood with covariance Σ M . Therefore, we can use a conjugate prior/posterior that is also a multivariate Gaussian [10] . Algorithm 1 shows our acquisition workflow. For a given patch we sample from the prior, impute the sample, and run model inference. From the classification probabilities, we select a patch using the acquisition function f ∈ {BALD, MSTD, ENTR, UNIF}. After making a measurement, we update the prior. Appendix C shows how we can generalize to both a Poisson noise model or regression tasks. We benchmark acquisition function loss via a the remaining model probability entropy after a measurement is captured.Algorithm 1: Active learning workflow for experimental acquisition across patches.Initialize: Prior π with mean µ and covariance Σ, model M, measurement covariance Σ M . for q = 1, ..., Q do // Measurement Loop for i = 1, ..., r do // Patch Selection Loop for t = 1, ..., T do // Monte Carlo Sample Loop I i,t ∼ π i (•); µ * i = [..., µ i-1 , I i,t ,Îs = I s + ε; ε ∼ N (0, Σ M,s ) ; // Measurement Capture on Patch s µ s ← (Σ -1 s + Σ -1 M,s ) -1 (Σ -1 s µ s + Σ -1 M,s Îs ); Σ s ← (Σ -1 s + Σ -1 M,s ) -1 ; // Prior Update on Patch s return M(µ) = P final ;// Final Model ClassificationQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: This work does not contain any formal proofs or theorems. In the main body of the paper we highlight how our technique works in one of four cases (classification with a Gaussian noise model) but suggest how it could work for the other three cases (with either a regression objective or a Poisson noise model). We go into further details on this in an appendix, with some derivations. 
results: 
 Figure 1 (left) shows the outcome of 1000 guided measurements on patches of an MNIST test image, with error bounds constructed over 50 runs on the same test image. Insets show the posterior updates at intervals of 250 iterations for each acquisition function. For measurements using BALD and MSTD acquisition functions, a quantitative improvement in the loss function is reflected qualitatively when we see that the true digit (a "2") more discernible for BALD and MSTD than for UNIF or ENTR. The ENTR acquisition function quickly becomes stuck taking measurements at the periphery where prior measurement variances start off small and downstream classification uncertainties barely change, so there is virtually no change in the measurement posterior. The UNIF acquisition captures superfluous measurements that have little impact on either the posterior or model scoring.Figure 1 (right) shows the outcome of 1000 guided measurements on 20 patches of the rotational spectrum of propene with nine additional classes, with error bounds constructed over 50 runs on the same ground truth spectrum. While there is no statistical guarantee that either BALD or MSTD will always outperform UNIF, however using a one-sided t-test we can be confident that both BALD (t = -7.17, p < 10 -9 ) and MSTD (t = -4.16, p < 10 -5 ) are likely to outperform UNIF. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? 
conclusion: 
 In this work, we presented an active learning framework for measurement acquisition that works to decrease uncertainty in classification rather than measurement. We demonstrate how our framework can operate effectively in both an imaging and a spectroscopic context to drive towards accurate classification. We also demonstrate how our AL framework can effectively select from among many possible narrowband measurements to disambiguate between similar molecule species. In the future, we hope to generalize this approach to eventually characterize the sorts of complex mixtures present in an analytical chemistry context. We also hope to demonstrate the viability of our approach for regression and segmentation tasks, which may have applications in areas of microscopy and tomography.NeurIPS Paper ChecklistQuestion: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We present two examples. We first use the MNIST dataset, so all data and parameters are readily available. We then use a ground truth species (propene) for our rotational spectroscopy test, and the explanations on reproducing the results are found in the corresponding appendix. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: Both figures in the main body of the paper show an error bar corresponding to 1/2σ, or half of a standard deviation. We also present one-sided T-test statistics to demonstrate that our technique, while not guaranteed to outperform a baseline, does so at a statistically meaningful frequency. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We briefly mention the compute resources we used in an appendix, however it is worth noting that the approaches we describe did not require significant compute. In fact, in a separate test we were able to place our code base on a desktop connected to a rotational spectrometer and perform similar tests. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: As stated above, we cannot currently identify any potential negative societal impacts that could result from the work performed. However we are certainly open to adding a section describing such negative societal impacts, if reviewers or editors identify such concerns from our stated methods. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: As far as we are concerned, this paper poses no such risks. 
licenses for existing assets: 
 Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: For our MNIST example, we of course cite the relevant work and models. For our rotational spectroscopy example, we cite the experimenters and computational chemists who first identified and derived the spectra we are using. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? 
 
 Appendix: Justification: A repository containing the code and methods for both examples will be made public following the acceptance of this manuscript. Furthermore, our primary contribution (the active learning framework) is laid out in an algorithm. All data used for our tests is publicly available.Justification: A repository containing the code and methods for both examples will be made public following the acceptance of this manuscript.Justification: Our approach is intended to make noisy experimental sampling more efficient. Within our moral imagination, we cannot envision ways that our approach, scoped as it currently is, could have negative moral, cultural, social, or societal impact.A Details of Rotational SpectroscopyRotational spectroscopy is a powerful analytical technique, but faces hurdles to greater adoption in industry settings due to the expertise required to manually interpret spectra and the amount of time required to capture reliable spectra. To a first approximation, the rotational spectrum of a molecule (assuming a single low-energy structure) can be described with three inertial constants A ≥ B ≥ C, usually presented in units of MHz, that primarily control peak locations. There are also three corresponding dipole constants µ A , µ B , µ C , usually presented in units of Debye, that primarily control peak intensities. These six constants are a near-unique representation of the rotational spectrum of a low-energy molecular structure, at least to a first approximation [29] . There are three types of peaks present in a rotational spectrum: A-type peaks (which depend on B and C constants), B-type peaks (which depend on A and C constants) constants, and C-type peaks (which depend on A and B constants). The impressive SNR provided by rotational spectroscopy means that peak frequencies can be measured to <10 kHz precision (corresponding to around seven or eight significant figures). Peak intensities, on the other hand, can vary more widely due to several quantum and thermodynamic effects that are difficult to properly account for beforehand.We chose propene as a reference species because is easily measured using standard microwave instruments [20] , is limited to one low-energy conformer at low temperatures [9] , and is of particular astrochemical interest [4] . Based on density functional theory calculations (performed using ORCA [22] with a B3LYP functional and a 6-31g(d) basis set), we can estimate the rotational constants of propene to be A * = 46753.53 MHz, B * = 9239.72 MHz, C * = 8101.64 MHz. Likewise, we can estimate the dipole constants of propene to be µ * A = -0.343 Debye, µ * B = 0.085 Debye, µ * C = -0.001 Debye. We then create similar classes of hypothetical molecules with very similar spectra to propene. After propene, we create either 9 or 99 additional classes with constants obtained by sampling χ i ∼ U (χ * -δ, χ * + δ); χ ∈ {A, B, C}, setting δ to 100 MHz. This creates a set of class-specific spectra that are closer to the spectra of propene than those of other similar structures. For smaller δ, classes will more closely resemble one another and the classification model will have a harder time distinguishing between classes. We augment spectral intensities by adding Gaussian noise in the frequency domain with a standard deviation of 0.1, with a floor set at zero (since intensities cannot be negative). We augment peak locations by randomly adjusting rotational constants according to χ † i ∼ N (χ i , σ 2 ); χ ∈ {A, B, C}. This effectively adjusts A-type, B-type, and C-type peaks individually, and can help account for the known differences between simulated constants and constants identified experimentally [19] . We augment each class over 1000 random iterations, then train our logit classifier.We show the prior distributions for the MNIST training data in Figure 2 and for the rotational spectroscopy example in Figure 3 . We use a Gaussian noise model to express the prior of both the MNIST and the rotational spectroscopy training set, binned into 100 MHz regions. For rotational spectroscopy, the acquisition of FID measurements is expected to inherently follow a Gaussian noise model (although not necessarily with nonzero covariances between bins). In the main body of this work, we show how our active learning framework can operate given a Gaussian noise model and a classification objective. Here we suggest that our approach can also work with a Poisson noise model and for a regression objective. This requires adjusting two equations: the conjugate prior update (stated in Algorithm 1) and the acquisition function (stated in Section 2). Assuming we are sampling from a Poisson likelihood noise model, we rely on a conjugate prior and posterior that is Gamma-distributed. We must therefore construct an initial Gamma-distributed prior which, with respect to α s and β s , is updated with a new observation Îs as α s ← α s + Îs and β s ← β s + 1. Note that because there is no documented conjugate prior relationship for a multivariate expansion of a Poisson likelihood (that we know of), we must treat each index of our distributions as a distinct and independent Poisson random variable.Next, suppose we wish to optimally decrease measurement uncertainty for a regression task. While our conjugate priors remain the same for new observations, our acquisition functions must change. Suppose our UQ model (such as a Gaussian process regressor) operates as M : R b → (x t , σ 2 t ). Suppose for a set of Monte Carlo samples on a patch, we perform model inference to obtain {(x 1 , σ 2 1 ), ..., (x T , σ 2 T )}. First we could simply take MSTD(x, σ 2 ) = 1 T T t=1 (x t -x) where x = T t=1 x t , or the mean standard deviation of the output predictions (excluding predicted variances). We can also reformulate a predicted information entropy equation as ENTR(x, σ 2 ) = Note that these entropy-based formulations exclude predicted values. To get the best of both worlds, acquisition functions that bring together both predicted values and standard deviations could also be constructed.38th Conference on Neural Information Processing Systems (NeurIPS 2024). 



Reviews: review: title: title: Motivation is not clear, lacks details on method or key idea
main_review: main_review: The paper proposes an active learning method by combining multiple existing approaches. 

-- However, it lacks a clear discussion of the key innovations or any details that differentiate this method from others in the field. For example, it is hard to understand the effect of patch selection loop and how this lead to benefits. 

-- Additionally, the claim that standard active learning methods do not account for uncertainty in the outcome is unclear. Methods like Upper Confidence Bound (UCB) explicitly capture the objective value and incorporate uncertainty in their decision-making process. 

-- The authors fail to specify which model is used in the paper, which limits the reader's ability to fully understand the proposed approach.
title: title: Review of "Active Learning for Optimal Minimization of Experimental Characterization Uncertainty"
main_review: main_review: The paper presents an active learning framework to achieve a cumulative decrease in the uncertainty of key outcome statistics in experimental settings with noisy data. The novelty is that the target for the decrease in uncertainty is a key outcome statistic and not an overall measurement of uncertainty. The authors achieved active learning by selecting regions for measurement without iteratively retraining a model.

Other advantages:
1. The paper is well-written (I could not find typos), and the ideas are mostly clearly presented. However, I think some further discussion is needed regarding the practical benefits of the approach (see major comments)
2. Novel perspective:  The paper challenges the common assumption in active learning that minimizing measurement uncertainty is optimal. Instead, it focuses on minimizing outcome (classification) uncertainty, which is more relevant in many experimental settings.
3. Practical application: The framework is demonstrated on rotational spectroscopy, showing potential for accelerating chemical characterization tasks. This real-world application adds significant value to the paper.

Comments regarding cons and needed improvements:
1. Computational complexity:  The paper does not thoroughly discuss the computational requirements of the method, which could be a concern for real-time or large-scale applications. More discussion on the computational requirements and potential scalability issues would be valuable.
2. It is unclear to me whether the main practical value of the approach is either a) the bottleneck is collecting new data or b) the bottleneck is training the model. The approach requires a pre-trained model, which I presume needs to be trained on the same data (at least, this is the case on the MNIST application where all the images are used). Thus, obtaining a good model will require training on many data; thus, the value is not in point a). It seems, then, that the main value relies on point b), and in that case, I ask what improvements this approach gives compared to the final pre-trained model that could not be obtained by, for instance, training the model for more epochs or with more data. More discussion on this is needed in the paper to clarify the benefits. For instance, it will be nice to see the loss function curve of the training of the pre-trained model preceding the loss function of the AL to check the overall benefit. 
 3. Applicability: From my understanding, a dedicated workflow will need to be constructed for each type of task (regression, classification, etc.) and data type, which hinders the approach's broad applicability. The workflow for the two applications applies only to two specific tasks and data types. The approach's applicability would greatly increase by developing a more generic workflow that is more agnostic to the task and data type. 
4. The authors could consider some theoretical bounds or guarantees on the performance of their method, at least in a simpler setting and model, such as linear regression.



Conclusion:
The paper's approach of using a fixed pre-trained model and focusing on optimal measurement selection is interesting and well-motivated. This is particularly relevant in experimental settings where retraining models may be impractical or when collecting experimental data is expensive. The authors have demonstrated good scientific thinking by challenging common assumptions and providing a new perspective on active learning for experimental characterization. The discussion of how the method can be adapted for Poisson noise models and regression tasks (Appendix C) extends the applicability of the approach. However, these extensions were not experimentally validated.  My main concert relies on a better discussion and quantification of the benefits, as mentioned in the major comment number two. While there are some areas for improvement, such as broader experimental validation and more theoretical analysis, the paper's strengths outweigh its weaknesses. Overall, I recommend acceptance.