Active Learning with LLMs for Partially Observed and Cost-Aware Scenarios
Authors: Nicolás Astorga
Tennison Liu
Nabeel Seedat
Mihaela van der Schaar
Venue: NeurIPS 2024 poster (2024)
Keywords: Active Learning
Large Language Models
Experimental Design
Primary area: active_learning
TL;DR: Formulation of POCA and instantiation, a problem related to the efficient acquisition of features and/or labels under some acquisition constraints

Abstract: Conducting experiments and gathering data for machine learning models is a complex and expensive endeavor, particularly when confronted with limited information. Typically, extensive _experiments_ to obtain features and labels come with a significant acquisition cost, making it impractical to carry out all of them. Therefore, it becomes crucial to strategically determine what to acquire to maximize the predictive performance while minimizing costs. To perform this task, existing data acquisition methods assume the availability of an initial dataset that is both fully-observed and labeled, crucially overlooking the **partial observability** of features characteristic of many real-world scenarios. In response to this challenge, we present Partially Observable Cost-Aware Active-Learning (POCA), a new learning approach aimed at improving model generalization in data-scarce and data-costly scenarios through label and/or feature acquisition. Introducing $\mu$POCA as an instantiation, we maximise the uncertainty reduction in the predictive model when obtaining labels and features, considering associated costs. $\mu$POCA enhance traditional Active Learning metrics based solely on the observed features by generating the unobserved features through Generative Surrogate Models, particularly Large Language Models (LLMs). We empirically validate $\mu$POCA across diverse tabular datasets, varying data availability, acquisition costs, and LLMs.

Active Learning with LLMs for Partially Observed and Cost-Aware Scenarios
Nicolás Astorga, Tennison Liu, Nabeel Seedat & Mihaela van der Schaar DAMTP, University of Cambridge Cambridge, UK
nja460cam.c ac uk
Abstract
Conducting experiments and collecting data for machine learning models is a complex and expensive endeavor, particularly when confronted with limited in- formation. Typically, extensive experiments to obtain features and labels come with significant acquisition cost, making it impractical to carry out all of them. Therefore, becomes crucial to strategically determine what to acquire to max- imize the predictive performance while minimizing costs. To perform this task, existing data acquisition methods assume the availability of an initial dataset that is both oully-observed and labeled, crucially overlooking the partial observability features characteristic of many real-world scenarios. In response to this chal- lenge, we present Partially Observable Cost-Aware Active-Learning (POCA), new learning approach aimed at improving model generalization in data-scarce and data-costly scenarios through label and/or feature acquisition. Introducing μPOCA as an instantiation, we maximize the uncertainty reduction in the predictive model when obtaining labels and features, considering associated costs. μPOCA enhance traditional Active Learning metrics based solely on the observed features by gener- ating the unobserved features through Generative Surrogate Models, particularly Large Language Models (LLMs). We empirically validate μPOCA across diverse tabular datasets, varying data availability, acquisition costs, and LLMs.
Introduction
In real-world machine learning (ML) applications. applications,fllly-bsseeeeee pristine training data is an excep- tion rather than the norm. This challenge especially evident during the initial stages of model development when training data is limited and varies in its informativeness across samples [13] At this stage, obtaining additional data crucial for improving model generalization but is fraught with challenges 64 In particular, acquiring new data can be costly, often resulting in only essential fea- tures and labels being collected, leading to partially observed features in training data. Therefore, it's vital that acquisition is efficient, yet it remains unclear which features and labels from each instance will ultimately prove essential. Furthermore, data sources themselves can also be partially observed, with different features available across samples, further complicating the acquisition process. These challenges emphasize the importance of a new problem we call Partially Observable Cost-Aware Active- Learning (POCA) illustrated in Figure [ Before its formalization in Section 2 provide an intuitive overview:
"In situations with limited labeled data and partial feature observations, our objective to enhance the generalization capabilities of a predictive model by strategically collecting features and/or labels. This goal should account the cost associated with data collection, as well as the varying levels of informativeness of labels and features across different instances"
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

fields where missing features Addressing the POCA problem is vital when building systems with partial observation or relevant features are yet to be defined, particularly in fields like customer churn, monitoring, healtheare, and finance (see Appendix A). For example, developing churn customer prediction system might start with some basic client information, such as demographics and income. However, to build such system, additional features may be needed, which could be gathered through further customer interactions or surveys. At the outset, it's uncertain which specific features will prove essential, and acquiring additional information and relevant labels (e.g., churn events) necessary to refine the ML system involves costs related to money, time, or risks limiting the data acquisition in practice. From a practical perspective, we envision POCA to be useful in applications exist, and also data acquisition techniques like Active Learning (AL) are necessary. Applications from different fields dealing with missing features and/or applying AL can be found in Table 2
Related work. The most related data acquisi- tion technique is AL. (7100. AL centers around enhancing model generalization through the ac- quisition of only additional labels. It operates under the assumption of having access to an ini- tial small, fully observed training set (referred to as the historical labeled set) and seeks to acquire additional labels for samples from unlabeled dataset (referred to as the pool set). This set is assumed to be fully observed in features, miss- ing only labels. The distinctions between POCA and AL are illustrated in Figure Tangentially, Active Feature Acquisition (AFA) methods [T1- 13 have been proposed to enhance the predic- tion of individual samples at test time-whers the sample is partially observed. Like AL, it as- sumes a -bssered historical labeled set, on which a model has already been trained. Given the trained model, the task then becomes identifying the most relevant unobserved features to acquire for partially observed instances at test time. We emphasize that AFA's primary focus is on optimizing feature acquisition for individual test samples, differing from our broader goal of data collection to enhance model training.
Partially Observable Cost sesivve-eaaning (POCA)
Active Learning (AL)
medel tratning
movir training
(pertictig o
(fullyy ofs.)
Stbbaet of features any and/ /or label
All featurrss and label y
pool pot (partialig ohs.)
fesetion
pool art
Sunction
Fally ohs.) )
Figure 1: Overview of data acquisition meth- ods. POCA acquires features and/or labels from a partially-observed pool incorporating them into a partially-observed training set. In contrast, AL targets label acquisition assuming fully-observed pool set and training set.
Towards Instantiation of POCA. Given this problem definition, it is natural to wonder whether traditional AL metrics can be employed straightforwardly in the POCA setting. These metrics are usually derived from predictive model that typically operates with fixed-size inputs. Consequently, predictions on partially observed instances can adversely affect the accuracy of AL metric estimations, leading to acquiring poor quality samples [4]. To overcome this challenge, we incorporate Generative Surrogate Models (GSM) to impute missing features in partially observed inputs, facilitating more precise estimation of AL metries. The effectiveness of GSM hinges on its ability to discern feature interrelations from available but unlabeled data. This task is particularly challenging due to the varying degrees of missingness in the instances and the constraints of limited sample sizes. To address these complexities, we employ Large Language Models (LLMs) to instantiate GSMs, utilizing their generation ability based on arbitrary conditioning and strong sample efficiency, allowing robust imputations to support the estimation of AL metrics under partial observability 11517
Uncertainty POCA. We term this instantiation uncertainty POCA (μPOCA), due its connection with Bayesian Experimental Design (8422), and its application in Bayesian Active Learning (BAL) 23 and Bayesian Optimization (4) From a Bayesian perspective, μPOCA maximizes the expected information gain or also known as expected uncertainty reduction, in the model's 's hypothesis resulting from an experiment. More specifically, μPOCA extends the concepts of expected information gain in the model's parameters (EIG) and expected predictive information gain (EPIG) to partially observed scenarios, introducing PO-EIG and PO-EPIG, respectively 7 27 (8). Here, these methodologies maximize the expected uncertainty reduction when acquiring labels and a subset of features. Since the impact of unacquired features cannot be directly assessed, GSMs facilitate the computation of these metrics.
In summary, we make the following contributions:
2

---

! We address the unexplored challenge of costly data acquisition to enhance model generalization in partially observed scenarios. This leads introduce and formalize POCA, novel ML paradigm for the acquisition of features and/or labels in the partially observed setting.
② We propose μPOCA, cost-aware Bayesian instantiation of POCA that maximizes the un- certainty reduction when acquiring data. μPOCA extends traditional AL metrics by imputing partially observed instances using GSMs. We theoretically show that the uncertainty reduction is larger than using vanilla AL metrics.
3 We propose the use of LLMs as specific instance of GSMs, designed to address challenges in partially observed scenarios, including data efficiency, arbitrary information conditioning, and handling both categorical and numerical feature values.
+ We empirically demonstrate μPOCA outperforms standard active learning on a variety of partially observability scenarios spanning datasets, sample availability, and acquisition metrics- highlighting the usefulness and applicability of μPOCA.
2 POCA: Partially Observable Cost-Aware Active-Learning
Preliminaries. Partially Observable Cost-Aware Active-Leaming is data acquisition problem that focuses on improving the predictive performance of p& (u]z)) in the supervised setting, with & the models can employ. We denote x E X' and y/ E 3 instances of observed features and target, alongside the respective random variables (RV) X and Y. Bold variables, expressed as a = {x, }
represent a set of variables, in this case, features indexed by j E [J] = (1,.... J.. where the bold form of j indicates set of sub-indices j. The sample index i E [T] possible indexes in the pool set, is omitted when unnecessary, i.c., Tiij = Zz. We denote to as the observed features with o C [ In the general case, we assume that feature Fijj considered for acquisition and the output of interest y: have associated acquisition costs Cıj and G.,++1- Here, Cij represents the total cost of acquiring the variables indexed by j for instance i.
(1, (1....: T), representing
POCA
In the context of partially observed data, our focus is on efficiently gathering features and/or labels to optimize a utility function, U+(-) subject to an acquisition constraint r-) at iteration t. U+(-) quantifies the trade-off between the costs of data acquisition and the increased generalization
capabilities of the model Φ, estimated from the available information t, and the hypothetical
acquisition of specific set of features and/or labels. We formulate the optimization of this utility as follows:
(i,5)*
arg max U. (i, s.t. (i,
(1)
+E(1),J[(++]
U+(+) is broadly defined, potentially estimated as result of using Bayesian techniques [18 23 27
frequentist techniques 211 RL techniques [22 33 or can even be subjectively defined through human desires. Note, optimizing Uz(-) involves an iterative process of ! selecting the instance and variables (i to acquire (features and/or labels); @ adding these variables into the training set; O updating the model ó using the updated training set. In more general case, this could also encompass batch acquisition [3 35] by using instead of i. Note that Eq. (1) represents the most general form of POCA, supporting model generalization when only features are acquired, as in semi-supervised or self-supervised learning. Our specific jPOCA instantiation (Section 2.1 focuses the supervised case, where selected features and labels are acquired.
Common modalities for POCA. We anticipate that most applications of POCA will center on the tabular domain (see Table 2). However, it could also find valuable uses in fields like medical and satellite imaging, where noise-induced occlusion is common. In these cases, determining when a sample requires additional information (features) is essential for enhancing prediction accuracy and model training- Likewise, interactive robots that learn through vision may benefit from this approach, as they need to discern which scenarios (samples) merit interaction to effectively learn the relationship between features (objects) and labels (task to solve).
In contrast with AL, POCA assumes R. C instead of the fully observed assumption of AL x. In addition, POCA considers the acquisition of features and/or labels, in this case, represented j.
3

---

2.1 I.POCA: A Bayesian implementation of POCA
Although several techniques can be used to implement POCA, we opt for a Bayesian approach due to its widespread success in data acquisition literature. Building on the foundational principles of Bayesian Experimental Design [18]
23 27], which provides a comprehensive framework for integrat- ing various sources of information [36 37 we introduce an instantiation of POCA within Bayesian framework. This new approach, termed uncertainty POCA or μPOCA, leverages information theory [88] to recast Eq. ( as cost-aware uncertainty reduction problem [27 33]. The core of μPOCA centered on reducing uncertainty through a class of models that are exclusively trained using supervised learning, focusing feature and label acquisition in partially observed scenarios. Here, we denote Pi.j as the uncertainty reduction for acquiring the label and features j for sample i, which varies based on the approximation method used.
POOA
We reformulate the optimization problem ① by substituting U. with utility function Ú. This function Ü is designed to capture the offeeoof between uncertainty reduction, Rig. and the acquisition costs associated with features and labels, represented by C.g. The new objective can be expressed as:
(i,j)* arg max U(ing Gig5, s.t. r(i,j),
(2)
(E[[).C[M
here Cr = Cijj + C,,+11 our research, we explore one specific instantiation, among potentially infinite options, denoted by Üن = foi. and r(i.j) = Cjjj < c, respectively, with c indicating the iteration's budget
by acquiring data, decreasing the predictive uncertainty produced by the possible hypothesis explaining the data. We work within the supervised model framework, hence we represent hypotheses as distributions over parameters. Our approach assumes the predictive model po (u (x²) can be expressed as:
How to obtain this uncertainty? We aim to minimize epistemic uncertainty 40 41
(oly'==) = (w) PPo ,,)],)"',,]] ,w],
where wa E W is an instance of the parameter space and its associated RV. Here, ø specifies the model choice, defining the functional form of (w) = p( (D), the posterior given the observed training set D, and the posterior predictive distribution ps (y x'), marginalized over w. Here, x' rep- resents a partially observed input, estimating Pa x²) must be adaptable to varying lengths of x'. To achieve this flexibility, models capable of handling variable-length inputs (such as Transformers) or, more broadly, marginalization techniques introduced in Section 3.2 can be employed.
This formulation is general, encompassing Bayesian models, neural networks with certain stochastic parameters [42 (3), and ensemble models 44 45) It also applies to Gaussian processes [66 when the posterior (D) is interpreted as a distribution over functions.
3 Method: Optimizing ,POCA
The challenge in optimizing ,POCA is in developing an uncertainty reduction metric, Psjz that accurately represents the decrease in uncertainty when acquiring subset of features for instance i, which has not been thoroughly investigated in the Bayesian literature. To address this, let's 's first provide some key background information. For data acquisition in ML, the primary focus has been on maximizing the expected uncertainty reduction, also known as expected information gain, when acquiring data (27). This concept can be mathematically defined as:
[(A, B) = H(A) HA)
where H(A) quantifies the uncertainty (entropy) about A, and H(ABB) represents the uncertainty of A after observing B (in expectation). Existing AL approaches that utilize the expected reduction of uncertainty methods maximize the uncertainty reduction of I(g.Yl when Y is observed.
summarized in Table
1 These
Altermative utility functions may balance uncertainty against costs as Ü = -Bss/C.. Other constraints could consider c the overall experimental budget.

---

Here, we use G to represent any random variable aligned with the generalization capabilities of the model and any arbi- trary conditioning. the Appendix, for completeness, we derive the estimation for these acquisition metrics.
Table 1: AL metries with form of I( Yl-).
Method
objective
BALD 2 EPIG
Q Yenal
1e. D
min. parameter uncertainty min. predictive uncertuinty
28
Ra. D. Xevst
JEPIG
42
Y
Zo. D, X
min. predictive uncertainty
3.1 Metrics for uncertainty reduction in partially observed scenarios
Challenges in designing ر... In real-world scenarios, the challenge is estimating uncertainty reduction based solely on accessible data oo- Traditional AL acquisition metries, denoted as Hoo (2,). estimate uncertainty scores assuming T, = æ. However, in partially observed scenarios where only subset of imputs, 3% S æ, is available, the observed features may lack sufficient informativeness for precise y estimates and reliable uncertainty scores Má (Io)
Generative Surrogate Model (GSM) to estimate metrics. more accurate estimate of current met- rics can be achieved using the aforementioned AL metrics by imputing the potential missing features in expectation:
Train
unlabeled
Generative Madel
unlabeled hist. set eet-
po(z,, (a.) 3
unlabeled pool set
(20) == Eε, ( U ū2)]-
(5)
Aeg,
Here, the samples ū, are obtained with a GSM denoted as po po(my (m))))))) (.)). which sample possible un- observed features z. based on the observed a. It's worth noting that training pe (2+g (r.)) could be done leveraging unlabeled data. In Figure 2 illustrate the acquisition process of μPOCA using GSMs.
(i, selected
Peatures
Prod. Model Pa(siz)
Acq, function
(.5)
arguna Diigg Eig) 5)
Figure 2: μPOCA leverages GSMs trained on unlabeled data for imputing missing features. The imputed observations are used as an input for the predictive model, whose outputs are used to compute the acquisition metric.
Why generative imputation can help Active Learning? In Bayesian active learning, acquisi- tion is closely linked to the concept of uncertainty reduction. To identify which features need be acquired, it is essential to estimate the possible un- observed values. these values lie in areas of high uncertainty within the hypothesis space, acquiring these features is beneficial, as it will help reduce this uncertainty. Conversely, if the possible values for certain unobserved features show little or no impact on uncertainty, then acquiring these features may not be necessary. Notably, deterministic imputation cannot achieve this, the lack of variability prevents assessment of its effect on uncertainty within the hypothesis space. This concept is illustrated in Figure II from Appendix H
Are we doing better? We demonstrate the theoretical value of this approach for a family of acquisition metrics presented in Table [ delving into their impact on the optimization process. These propositions convey the intuitive idea that acquiring more information, this case, features, leads to higher reduction in uncertainty for the predictive model (proofs can be found in Appendix (B).
Let p(xx) be an acquisition metric that can be written as I(g (o)), with G and representing the same variables observed in traditional AL (Table 7V, and with Y, X3 as previously defined. I Xi)o, the following equality holds:
Proposition 1.
I(G, (Y, (Y)))).) Exy [(G.\=
Corollary 1. Under the assumptions of Proposition the subsequent inequality is established:
Ex,I(9 Y |æj o) 2
I(G,
Equality is attained when G X,
Q Proposition D states that the uncertainty reduction of G (c.g., the random variable of the parameters, 52) by knowing Y and X, is equivalent to the expected uncertainty reduction achieved by knowing Y while conditioning on unobserved variables æz. This is convenient as the conditioning on æy can be computed using Monte-Carlo approximation 48]
5

---

Q Corollary 1 implies that acquiring both labels and features results in greater uncertainty reduction compared to acquiring only labels, the objective maximized in traditional AL (Table D The uncertainty reduction is equivalent when, given and Y, the unobserved features X, don't have any impact in generalization g.
Note that the independence assumption of Proposition is valid in the supervised models we consider. In essence, this is because acquiring features without labels do not aid parameter updates and in consequence generalization improvements. The foundation of this assumption lies in the predictive mapping process from X + Y + Ω2 rather than in the data itself. Appendix B provides a more detailed explanation of this independence assumption's 's validity. Additionally, empirical evidence supporting the validity of Corollary I and, by extension, Proposition I is shown in Appendix K
Equations (6) and (7) always apply to the true random variable of unobserved features or any of its
approximations. However, the terms in Eq. (6) reflect the uncertainty reduction of obtaining the actual features when the approximated distribution of the GSM accurately reflects the distribution of the true random variable. We empirically investigate this approximation and its practical utility.
PO Active learning metrics. Building on Proposition [ and Corollary [ we ex- tend BALD and EPIG as > Partially Observable Expected Information Gain (PO-EIG): Ex, I(2, Y (ar,, Lo. D) and & Partially Observable Expected Predictive Information Gain (PO-EPIG): Egg I(Yeval, Y|æ,,, oio Xeval D). Corollary 1 states that these metrics provide higher uncertainty reduction than their vanilla counterparts. We use Monte- Carlo for estimation (see Appendix C
3.2 Predictive models in the PO setting
Our derivations are based on distribution perspective, considering different numbers of conditioned variables. For instance, when calculating PO-EIG, expressed as EZ [(2, Y|æj vo D), it is necessary to compute the distribution Po R. 2.)) Here, T, could vary in length from one instance to another and zj varies based on the number of features considered for computing the uncertainty reduction metric. In practical terms, this means that the predictive model, attempting to approximate this distribution, must effectively handle inputs with varying variables and lengths.
To address this challenge, we employ GSMs to impute the missing information to enable predictive models that expect fixed-size inputs. This imputation is separated in two different steps (1) condition- ing and (2) marginalization. Essentially, when evaluating the uncertainty reduction of an unobserved subset of features æj considered for acquisition, we condition this subset æj and zo (the observed features), marginalizing over the remaining subset of unobserved features Ty (where Ty U xy is the set of all unobserved features). This approximation process is mathematically formalized as follows, with supplementary visual aids provided in Figure & of Appendix C.3
(o xz) = Po (o: æy Z. poo (2 [To, Z.) Ep (mny (m [p)())
(8)
Here the predictive model simulates the behavior, wherein the predictive model only has access to v, and æj but computed using a model as it would have all the features. The marginalization step is essential for accurate metric estimation in the pool set and can also be applied during training. However, to reduce costs, we use GSM to impute features not acquired in the training set.
3.3 Efficient computation of utility function, Eq. 2
Our goal is to maximize (.)(.)) = Ü(P.. C..)). which incorporates the uncertainty reduction Ps. It crucial recognize that Aم. could encompass all possible combinations of unobserved features. How- ever, computing f for every possible combination of (i.j) is impractical, since it is of order ((2)). To overcome this challenge, we propose estimating the uncertainty reduction for all unobserved features
Algorithm 1 Acquisition process
1: P [].F=I] []
2:
for E
[ do
3:
j [J]
4:
while r(i,j") do
5:
arg max Uner sa. (())
6:
j"
7:
end while
8: Padd Fadd(j")
and subsequently excluding the less relevant ones, i.e. those contributing minimally to uncertainty reduction.
9: end for
10: s* arg en Pii,,"" - =F("
This ensures that we always retain the most relevant
11: Return: (f")""

---

features until the constraint in Eq. is satisfied, in order OJJ2). The acquisition process is summarized in Algorithm[[] with feature selection steps highlighted in teal. Appendix C.3 provides details on an efficient approach to computing the marginalization step necessary for estimating Pi.j This efficiency can be further improved by selecting the most informative samples, followed by the application of Algorithm | and details on GSM training and sampling, refer to Appendix D
(see Appendix[D) (D). For comprehensive overview, including cost analyses,
3.4 Large Language Models as Generative Surrogate Models
LLMs as GSMs. For the scenarios outlined in POCA, we specify the following desiderata for GSMs: (P1) generative capability, (P2) ability to learn from partially observed data, (P3) sample efficiency, and (P4) seamless integration of mixed-type variables. We argue that L.MMs are well-suited to meet these criteria due to their generative capabilities and flexibility in training under arbitrary conditioning contexts [49]5 Moreover, recent research highlights their exceptional performance in few-shot settings [52] 53] and their generative capabilities applied to tabular data comprising mixed-type attributes [S]] These strengths provide strong justification for focusing our research on LLMs as GSMs. However, any imputation method that fulfills these criteria may also serve as a suitable GSM, as further discussed in Appendix G
We use LLMs as GSMs leveraging the unlabelled information via Supervised Fine-Tuning (SFT). When working with tabular data, we serialize rows of the data, thereby converting it to natural language. For example, a set of features is serialized as "Age is 25, Gender is Female, ..., Blood pressure is 0.57". The LLM is then used to predict unobserved features based on available information. To achieve this, we utilize SFT on the LLM with the available observed features. The training data can encompass all unlabeled data, including historical and pool set data. The process entails generating random masks to form an input, m @ Vo and an output, (1 - m) @ Co for SFT across all available data. This empowers the LLM to predict missing information by leveraging various combinations of observed features For more details, refer to Appendix F22
Analysis of GSMs. The effectiveness of μPOCA in partially observed settings is closely tied to the GSM's s ability to approximate the distribution of unobserved features. Two primary factors influence the accuracy of this approximation: (1) the approximation capacity of the GSM and (2) intrinsic characteristics of the dataset. detailed examination of these factors is provided in Appendix]] []]
Experiments
We evaluate µPOCA across three dimensions First, in the case that all features acquired, we demonstrate that μPOCA acquisition metrics are more informative in selecting instances with informative features than AL metrics. Second, we present synthetic experiment accompanied by theoretical insights. Finally, we explore scenarios with budget constraints demonstrating that μPOCA on more challenging scenarios.
Comparing jPOCA with the current AL. models complex, as the latter are designed for fixed-size inputs. To address this challenge, we developed Scenario 1 (see visual aid in Appendix HH. This scenario involves dividing each instance in the pool set into the same observed and unobserved feature sets. We specifically select half of the features remain unobserved, chosen by their high relevance to the predictive task as identified by a preliminary RF. It is important to note that while μPOCA methods can handle any form of missing data, Scenario / ensures a fair comparison by allowing AL models to operate without any modification, which could bias our evaluation. This scenario presumes the availability of a historical unlabeled dataset for training the GSM, using instances that include data on unobserved features. In practical applications, the pool set can often serve as the training set itself, representing more realistic scenario may encounter. We refer this setting as Scenario 2. Results for this scenario are presented in Appendix [] where GSM is trained on partially observed data, while vanilla AL employs deterministic imputation to manage this case.
We select Magic, Adult, Housing, Cardio, and Banking tabular datasets based on their use in AL 28 tabular generation 49 51), LLM-based classification [54] and relation with potential roal-world
'Witoout loss of generality, in-context learning viable for an LLM- GSM 'Code can be found at:
or https: //githuu/ com/
vandersscheaJJab/POOa

---

13750 seagle
1390 Iveadee
Figure 3: PO-EIG and PO-EPIG computed across diverse datasets showing they either outperform or match their fully observed counterpart in terms of predictive performance
applications (Appendix Ap. These datasets have diverse characteristics: sample size, number of features, number of categorical, and numerical variables. We prioritize datasets with over 1000 samples guarantee sufficient samples for the pool set. We showcase results using RF trained with 100 estimators. We start training with two fully observed samples per class, conduct 150 acquisition cycles, repeat each experiment 60 seeds, and display 95% confidence interval. We train Mistral7B-Inntruuttvd)) using Monte-Carlo samples for generative imputation.
4.1 Need for POCA: Shortfalls of Active Learning
Objective. To assess the need for more generalized methodologies such as μPOCA, we analyze the performance of PO-EIG, partially observed extension of BALD (EIG)-the most widely used metric in active learning literature. Additionally, we incorporate EPIG into the study, a recently developed actíve learning metric within the family. According to our theoretical framework (see Corollary [ PO-metrics outperforms their vanilla counterparts in terms of uncertainty reduction. Our goal is examine whether this uncertainty reduction lcads to improved downstream performance when all features are acquired based on the same information, Co or, in other words, the selected instances possess features that are more relevant.
Setup. To ensure fair comparison, we evaluated PO-metrics and Kanilaa-etriie under Scenario 1, using Random and Oracle as reference baselines. Here, Oracle represents the Vanilla-metrics acquisition metric, but with access to features. Ideally, when GSM functions optimally, the performance of PO-EIG should align with that of Oracle.
Analysis. The first thing note is that EIG metrics computed with partially observed features can be significantly worse than simple baselines like random as shown in Magic dataset from Figure (top). Figure 3 (top) demonstrates that PO-EIG generally either outperforms or worst case matches their fully observable counterparts BALD across all datasets. A similar behavior is observed for PO-EPIG, which generally outperforms their vanilla metric counterpart. This suggests that an increase in uncertainty reduction translates into an increase in downstream performance. While PO-EIG and PO-EPIG metrics consistently outperform baselines, they occasionally fall short of oracle performance, notably in the Housing datasets. This may stem from two factors: Firstly, the GSM has poor prediction performance on the unobserved data due to insufficient data or model capacity. Secondly, even with adequate capacity and data, weak correlation between unobserved data and the target hinders the acquisition process. We study these factors in Appendix []] We note that it is non-trivial to quantify the GSM's capability correlations of unobserved data to the target. Thus, the practical implication is that both PO-metrics should be preferred in PO settings, providing performance boost at least matching their vanilla counterparts. Additionally, in Appendix [.I we include other relevant Active Learning metrics that, while not fitting into the family of studied metrics, also demonstrate performance gains with the proposed framework.
V First, AL metrics computed on partially observed features can dramatically fail for selecting relevant instances. Second, PO-EIG and PO-EPIG generally oulperform match fully observed counterparts.

---

4.2 Theoretical insights
Objective. We investigate the implications of our theoretical findings (Eq. 7 on the acquisition process; determining whether a weak correlation between unobserved features and the target, results in small gap between PO-EI EIG and BALD. We also explore how correlation affects performance.
Setup: We create intuitive synthetic 2D experimental setup (Figure 4 with a variable target. The target is determined linearly with varying slopes, leading to different correlations with the features. Our chosen features- 1 X2, and X data along the x-axis, y-axis, and Gaussian category, respectively. Introducing the Gaussian category injects stochasticity into the marginalization process, ensuring non-trivial solutions. The observed feature is X1, with possible acquisition of X2. X3. We examine three scenarios: 1) Low (o,r( where the class depends solely on X due to vertical slope; 2) High Corr(X2,,Y), where the class depends solely on X2, rendering Xi irrelevant; and 3) Mid. Corr(X2,X) where X1 has some impact. Note, we evaluate acquisition metrics and performance until the convergence of the oracle (BALD with all features)
Analysis. Figure 5J empirically validates that PO-BIG is always equal to greater than BALD, consistent with our theoretical insights (Eq. 7 Figure 5 illustrates the evolution of the metric gap between PO-EIG and BALD under varying correlations between X2.3 and Y. In low corre- lation scenarios (orange line), the gap diminishes towards the acquisition's 's end. aligning with Corol- lary D where both metrics should converge when X2,3|211 9, i.c., when the unobserved fea- tures don't impact generalization. Initially, the gap exists as the model learns from data the redun- dancy of unobserved features. The same figure shows larger correlation leads to a wider gap, observed most notably in the large correlation scenario (purple) and moderately in the medium correlation scenario (teal). Figure shows that, generally, the degree of problem correlation provides proxy correlation with acquisition performance. For example, the purple line exhibits the largest differ- ence between BALD and EIG. Particularly in low correlation scenarios, the performance difference between PO-EIG and BALD is negligible across the acquisition (orange line).
CAM
Figure 4: Synthetic dataset
Das
Loe Can X,
1dd Cir Xo
Had Ouer A
Figure 5: Comparing PO-EIG and BALD.
4.3 Cost-aware active learning
We evaluate the performance of μPOCA (specifically PO-EIG) under budget-constrained feature acquisition, aiming to determine if acquiring only a subset of features, denoted as j. offers an advantageous trade-off in performance. This selective acquisition approach enables acquiring larger number of instances within the same budget. Furthermore, we aim to show that imputation alone cannot fully replace the need for direct data acquisition.
Objective:
Setup. We use the Magic dataset as case study to examine the impact of cost constraints on predictive performance and the feature acquisition process. To facilitate this assessment, we introduce costs associated with both features and labels. For simplicity and visualization elarity, we assume the cost of instance to be 1, representing the sum of the costs for all features and the label, with each feature assigned an equal cost. This setup allows us to analyze four distinct approaches: (1) the Vanilla acquisition metric (EIG), (2) PO-EIG, (3) PO-EIG with maximum feature acquisition limit of 60%, and (4) PO-EIG with unrestricted feature acquisition. We evaluate the performance of these approaches in three ways: by accuracy based on acquired instances (Figure [6] left), by performance relative to the budget utilized (assuming no label costs) (Figure 6 middle), and by performance with varying label costs under a fixed total budget of 50 (Figure right).
Analysis. Figure 6 (left) illustrates that acquiring fewer features generally results in decreased perfor- mance; however, still outperforms the EIG baseline. While limited feature acquisition impacts per- formance, it allows for a more efficient budget allocation across instances, enabling the acquisition of a larger instance pool. This trend is visible in [ where performance is plotted against the total budget spent, assuming no label cost. Here, methods focusing on selective feature acquisition excel, they gather more overall information through increased instance count and key features.

---

Bedart sputt Bustro oxailind)
205 Eral, oagaiced (PO-ED0)
(45 eagabied (PO-EIG)
100% Eod. aoguind (PO-503)
1073 Sest. aegaire (EIG)
Figure 6: Left: Accuracy vs, number of instances acquired. Middle: Accuracy vs, budget without considering label costs. Right: Accuracy budget with varying label costs.
Figure (right) demonstrates that the optimal PO-EIG method de- pends on feature acquisition cost: when cost is heavily weighted toward feature acquisition (right histogram), the best method is PO- EIG with 20% of features acquired, whereas a 50% label cost favors PO-EIG with 60% feature acquisition. While these findings might suggest that acquiring fewer features and imputing the rest is optimal for maximizing instances, this approach may introduce noise into the training set, potentially biasing the model. To explore this, analyze model performance at different levels of feature acquisition in Figure 7 with varying levels of pool data, using the full pool set for training (excluding non-acquired features). As shown on the y-axis, acquiring more features enhances performance. When the budget is unlimited, acquiring all available data is preferable; however, in practice, this may not be feasible, making POCA approaches advantageous.
Ziro value Imputation
IIM Imputee
261
211 763
Murberd csa 4s-d
Figure 7: Acquiring vs imputing.
@ First, μPOCA metrics (PO-EIG) can be more cost-effective than common active learning metrics. Second, imputation is useful for missing data but shouldn' 't replace data acquisition.
5 Discussion
We introduce and formalize POCA data acquisition framework, addressing the vital but underex- plored challenge of partially observed settings. Through μPOCA, a practical implementation of this framework, demonstrate the feasibility of acquiring unobserved features and labels based those partially observed features, using more generalized AL utility metrics - computed by estimating fea- tures generated using an LLM-based GSM. Our results over various scenarios are substantially more effective than alternatives - of substantial value for data acquisition cost-restrictive environments. We hope the POCA framework and our subsequent findings will spur additional work advance data acquisition in partially observed settings.
L Limitations. Our work focuses on the values of features, providing general framework where restrictions are the main source of constraints terms acquisition. However, we do not assess how these restrictions are selected, which could be promising area for future research. We also note that we use LLMs in the context of data acquisition. Like any GSM, LLMs can indeed exhibit biases that affect the acquisition process. In this study, we did not consider this issue, and it represents an interesting avenue for future work. If necessary, current debiasing techniques can be applied.
Practical consideration and future work. (1) In the PO setting with data "missingness," GSM imputation is essential for acquisition. Future work could quantify uncertainty 443 555 to assess
GSM efficacy. (2) LLM capability also impacts acquisition; while we use a 7B parameter model, larger models could further enhance performance, though this is beyond our current scope.
Acknowledgments and Disclosure of Funding
We thank the anonymous NeurlPS reviewers, members of the van der Schaar lab, Andrew Rashbass for insightful comments and suggestions. NA thanks W.D. Armstrong Trust for sponsorship and support. TL thanks AstraZeneca for support. NS thanks the Cystic Fibrosis Trust. This work was supported by Microsoft's 's Accelerate Foundation Models Academic Research initiative.
10

---

References
[1] Leslic Pack Kaelbling Kenji Kawaguchi and Yoshua Bengio. Generalization in deep learning. In Mathematics of Deep Learning, Cambridge University Press, appear. Prepint avaliable Massachusetts Institute of Technology, 2018.
as: MIT-CSAIL-TR-20188.000
[2] Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li, Dataset pruning: Re- ducing training data by examining generalization influence. arXiv preprint arXiv:2205.09329, 2022.
[3] Nabeel Seedat, Fergus Imrie, and Mihaela van der Schaar. Navigating data-centric artificial intelligence with DC-Check. Advances, challenges, and opportunities. IEEE Transactions on Artificial Intelligence, 2023.
[4] Yuji Roh, Geon Heo, and Steven Euijong Whang. A survey on data collection for machine learning: big data-ai integration perspective. IEEE Transactions on Knowledge and Data Engineering, 33(4): 13(4):1328-1)) 1328-13477 2019.
[5] Steven Euijong Whang, Yuji Roh, Hwanjun Song, and Jae-Gil Lee. Data collection and quality challenges in deep learning: A data-centric ai perspective. The VLDB Journal, 32(():791-81) 2023.
[6] Lea Goetz, Nabeel Seedat, Robert Vandersluis, and Mihaela der Schaar. Generalization key challenge for responsible ai in patient-facing clinical applications. npj Digital Medicine, 7(1):126, 2024.
[7] Neil Houlsby, Ferene Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv: 2011.
[8] Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep Bayesian active learning with image data. Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 118--1192. PMLR, 06-11 Aug 2017.
[9] Yanyao Shen, Hyokun Yun, Zachary C Lipton, Yakov Kronrod, and Animashree Anandkumar. Deep active learning for named entity recognition. arXiv preprint arXiv:1707.05000 2017.
[10] Dana Angluin. Queries and concept learning. Machine learning, 2:319-342, 1988.
[11] Hajin Shim, Sung Ju Hwang, and Eunho Yang. Joint active feature acquisition and classification with variable-size set encoding. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[12] Yang Li and Junier Oliva. Active feature acquisition with generative surrogate models. In International Conference on Machine Learning, pages
PMLR, 2021.
[13] Chao Ma. Sebastian Tschiatschek, Konstantina Palla, José Miguel Hernández-Lobuld Sebas- tian Nowozin, and Cheng Zhang. Eddi: Efficient dynamic discovery of high-value information with partial vac. arXiv preprint arXiv:1809.1 11099.11144 11122, 2018.
[14] Siddharth Karamcheti, Ranjay Krishna, Li Fei-Fei, and Christopher D Manning. Mind your outliers! investigating the negative impact of outliers on active learning for visual question answering. arXiv preprint arXiv:2 (0l.:2/07.02333 07.30331,.1..1 2021.
[15] Robin Jaulmes, Joelle Pineau, and Doina Precup. Active learning in partially observable markov decision processes. In João Gama, Rui Camacho, Pavel B. Brazdil, Alípio Mário Jorge. and Luis Torgo, editors, Machine Learning: ECML 2005, pages 601-608, Berlin, Heidelberg. 2005. Springer Berlin Heidelberg.
[16] Massih R. Amini, Nicolas Usunier, and Cyril Goutte. Learning from multiple partially observed views an application to multilingual text categorization. In Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, and A. Culotta, editors, Advances in Neural Information Processing Systems, volume 22. Curran Associates, Inc., 2009.

---

[17] Merlijn Krale. Active Measuring in Uncertain Environments. PhD thesis, 2023.
[18] Kathryn Chaloner and Isabella Verdinelli. Bayesian experimental design: a review, Statistical Science, 1995.
[19] Tom Rainforth, Adam Foster, Desi R Ivanova, and Freddie Bickford Smith. Modern bayesian experimental design. arXiv preprint arXiv:2302. 2023.
[20] Adam Foster, Desi R Ivanova, Ilyas Malik, and Tom Rainforth. Deep adaptive design: Amortizing sequential bayesian experimental design. In International Conference on Machine Learning, pages
PMLR, 2021.
[21] Anthony Atkinson, Alexander Donev, and Ray Tobias. Optimum Experimental Designs, with SAS. Oxford University Press, 2007.
[22] P. Sebastiani and H. P. Wynn. Maximum entropy sampling and optimal Bayesian experimental design. Journal of the Royal Statistical Society Series B, 62(1):145-157, 2000.
[23] David JC MacKay. Information-basee objective functions for active data selection. Neural computation, 4(4):590-600) 604, 1992.
[24] Tennison Liu, Nicolás Astorga, Nabeel Seedat, and Mihaela van der Schaar. Large language models to enhance bayesian optimization. In The Twelfth International Conference on Learning Representationss, 2024.
[25] Philipp Hennig and Christian J Schuler. Entropy search for information-efficcent global optimization. Journal of Machine Learning Research, 13(6), 2012.
[26] José Miguel Bernández-Lobatt Matthew W Hoffman, and Zoubin Ghahramani. Predictive entropy search for efficient global optimization of black-box functions. Advances in neural information processing systems, 27, 2014.
[27] Dennis V Lindley. On measure of the information provided by experiment. Annals of Mathematical Statistics, 1956.
[28] Freddie Bickford Smith, Andreas Kirsch, Sebastian Farquhar, Yarin Gal, Adam Foster, and Tom Rainforth. Prediction-oriented Bayesian active learning. International Conference on Artificial Intelligence and Statistics, 2023.
[29] Anthony Atkinson, Alexander Donev, and Randall Tobias. Optimum Experimental Designs, with SAS. 05 2007.
[30] A. Fisher. Design of experiments. British Medical Journal, 1(3923):554, Mar 1936. Letter.
[31] Thomas P. Ryan and J. P. Morgan. Modern experimental design. Journal of Statistical Theory and Practice, 1(3-4):501-506, 2007.
[32] Meng Fang, Yuan Li, and Trevor Cohn. Learning how to active learn: A deep reinforcement learning approach. arXiv preprint arXiv:1708.023. 02383, 2017.
[33] Arkady Epshteyn, Adam Vogel, and Gerald DeJong. Active reinforcement learning. In Proceedings of the 25th international conference on Machine learning, pages 296-303, 2008.
[34] Andreas Kirsch, Joost van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning, 2019.
[35] Andreas Kirsch, Sebastian Farquhar, Parmida Atighehchian, Andrew Jesson, Frederic Branchaud-Charron, and Yarin Gal. Stochastic batch acquisition for deep active learning. arXiv preprint 210iv:2106.10099 2021.
[36] Andreas Kirsch and Yarin Gal. Unifying approaches in active learning and active sampling via fisher information and information-theoreee quantities. Transactions on Machine Learning Research, 2022. Expert Certification.
12

---

[37] Andreas Kirsch and Yarin Gal. A practical & unified notation for information-theowm quantities ml. arXiv preprint arXiv:2106.1206, 2021.
[38] A mathematical theory of communication. The Bell system technical journal,, 27(3):379-4233
1948.
[39] Jose M. Bernardo. Expected Information as Expected Utility. The Annals of Statistics, 7(3):686 690, 1979.
[40] Lewis Smith and Yarin Gal. Understanding measures of uncertainty for adversarial example detection. arXiv preprint arXiv: wwt.ivv:803.... 1803.885333 2018.
[41] Vu-Linh Nguyen, Mohammad Hossein Shaker, and Eyke Hüllermeier. How to measure uncertainty in uncertainty sampling for active learning. Machine Learning, 111(1):89-12. 2022.
[42] Mrinank Sharma, Sebastian Farquhar, Eric Nalisnick, and Tom Rainforth. Do bayesian neural networks need to be fully stochastic? In Francisco Ruiz, Jennifer Dy, and Jan- Willem van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pages 7694-7722. PMLR, 25-27 Apr 2023.
[43] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050-1059, PMLR, 2016.
[44] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
[45] Thomas G. Dietterich. Ensemble methods in machine learning. In International Workshop on Multiple Classifier Systems, 2000.
[46] Carl Edward Rasmussen al. Gaussian processes for machine learning, volume 1. Springer.
[47] Andreas Kirsch, Tom Rainforth, and Yarin Gal. Test distribution-awart active learning: A principled approach against distribution shift and outliers, 2021.
[48] Tom Rainforth, Rob Cornish, Hongseok Yang, Andrew Warrington, and Frank Wood. On nesting Monte Carlo estimators. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4267-4276. PMLR, 10-15 Jul 2018.
[49] Vadim Borisov, Kathrin Sessler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. Lan- guage models are realistic tabular data generators. In The Eleventh International Conference on Learning Representations, 2022.
[50] Aivin V Solatorio and Olivier Dupriez. Realtabformer: Generating realistic relational and tabular data using transformers. arXiv preprint arXiv:2302.02041 2023.
[51] Nabeel Seedat, Nicolas Huynh, Boris van Breugel, and Mihaela van der Schaar. Curated IIm: Synergy of Ilms and data curation for tabular augmentation in low-data regimes. In Forty-first International Conference on Machine Leurning, 2024.
[52] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint
2022.
[53] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, al. Large language models encode clinical knowledge. Nature, pages 1-9, 2023.
13

---

[54] Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. Tabllm: Few-shot classification of tabular data with large language models. In International Conference on Artificial Intelligence and Statistics, pages 5549-5581. PMLR, 2023.
[55] Boris van Breugel, Zhaozhi Qian, and Mihaela van der Schaar. Synthetic data, real errors: how (not) publish and use synthetic data, arXiv preprini arXiv:2305.0922: 2023.
[56] Bing Zhu, Yin Pan, and Zihan Gao. Application of active learning for churn prediction with class imbalance. ICMLT "1, page 89-93, New York, NY, USA, 2018. Association for Computing Machinery.
[57] M. A. R. Khalid, M. A. H. Farquad, and V. Kamakshi Prasad. Data classification using active learning based data modification: An application to churn prediction. 2017 International Conference on Current Trends in Computer, Electrical, Electronics and Communication (CTCEEC), pages 529-533, 2017.
[58] Youngjung Suh. Machine learning based customer churn prediction in home appliance rental business. Journal of Big Data, 10(1):41, 2023.
[59] Elizabeth Barnes, Richard Meyer, Bob McClelland, Hildegard Wieseholfer, and Mike Worsam. Murketing: An Active Learning Approuch. Wiley-Blackwer paperback edition, April 1997.
[60] Toshiki Ueda and Hiroshi Ban. Active learning on digital marketing for advertising university museum exhibition. Procedia Computer Science, 126:2097-210 2018. Knowledge-Base and Intelligent Information & Engineering Systems: Proceedings of the 22nd International Conference, KES-2015 Belgrade, Serbia.
[61] V. Anand and Varsha Mamidi. Multiple imputation of missing data in marketing. In 2020 International Conference on Data Analytics for Business and Industry: Way Towards a Sustainable Economy (ICDABI), pages 1-6, 2020.
[62] Mariusz Grabowski. Handling missing values in marketing research using som. In Daniel Baier and Klaus-Dieter Wernecke, editors, Innovations in Classification, Data Science, and Information Systems, pages 322-329, Berlin, Heidelberg, 2005. Springer Berlin Heidelberg.
[63] J. Wu and X. Zhang. Credit risk assessment: An active learning approach. 01 2010.
[64] Yue Zhao, Yong C. Cao, Xiu Q Pan, Yong Lu, and Xiao N. Xu. A telecom clients' credit risk rating model based on active learning. In 2008 IEEE International Conference on Automation and Logistics, pages 2590-2200 2503,25532220 2008
[65] Jill Hussey. Understanding Business and Finance: An Active Learning Approach. Letts Educational, 6 1991. Paperback edition.
[66] Ronghao Tong. An active learning application loan default prediction: based forest classifier model. In Xiaoli Li, editor, Third International Conference on Artificial Intelli- gence and Computer Engineering (ICAICE 2022), volume 12610 of Society of Choto-Optical Instrumentation Engineers (SPIE) Conference Series, page 126104W, April 2023.
[67] Feng Zhao, Yan Lu, Xinning Li, Lina Wang, Yingjie Song, Deming Fan, Caiming Zhang, and Xiaobo Chen. Multiple imputation method of missing credit risk assessment data based on generative adversarial networks. Applied Soft Computing, 126: 109273, 2022.
[68] Jomark Noriega, Luis Rivera, and Jose Herrera. Machine learning for credit risk prediction: A systematic literature review, 08 2023.
[69] R. Florez-Lopez. Effects of missing data in credit risk scoring. comparative analysis of methods to achieve robustness in the absence of sufficient data. The Journal of the Operational Research Society, 61844486-- 2010.
[70] I. El-Hasnony, O. M. Elzeki, A. Alshehri, and H. Salem. Multi-label active learning-based machine learning model for heart discase prediction. Sensors, 22(3):1184 2022.
14

---

[71] Zoe Fowler, Kiran Premdat Kokilepersaud, Mohit Prabhushankar, and Ghassan Alregib. Clinical trial active learning. In Proceedings of the 14th ACM International Conference on Bioinformaties, Computational Biology, and Health Informatics, BCB '23, New York, NY, USA, 2023. Association for Computing Machinery.
[72] JJ Teijema, L Hofstee, M Brouwer, J de Bruin, G Ferdinands, J de Boer, P Vizan, S van den Brand, C Bockting, R van de Schoot, and A Bagheri. Active learning-based systematic reviewing using switching classification models: the case of the onset, maintenance, and relapse of depressive disorders. Frontiers in Research Metrics and Analytics, 8:1 8:1178181, 2023.
[73] Angona Biswas, Md Abdullah Nasim, Md Ali, Ismail Hossain, Md Azim Ullah, and Sajedul Talukder. Active learning on medical image, 06 2023.
[74] Samuel Budd, Emma C. Robinson, and Bernhard Kainz. A survey on active learning and human-in-the-loop deep learning for medical image analysis. Medical Image Analysis, 71:102062, 2021.
[75] Xing Wu, Cheng Chen, Mingyu Zhong, Jianjia Wang, and Jun Shi. Covid-al: The diagnosis of covid-19 19 with deep active learning. Medical Image Analysis, 68: 101913, 2021.
[76] A. Fong, J.L. Howe, K.T. Adams, and R.M. Ratwani. Using active learning identify health information technology related patient safety events. Applied Clinical Informatics, 8(1):35-46, 2017.
[77] S. Nijman, A.M. Leeuwenberg, I. Beekers, I. Verkouter, J. Jacobs, M.L. Bots, F.W. Asselbergs, K. Moons, and T. Debray. Missing data is poorly handled and reported in prediction model studies using machine learning: a literature review. Journal of Clinical Epidemiology, 142:218 1- 229, February 2022. Epub 2021 Nov 16.
[78] Yuan Luo. Evaluating the state of the in missing data imputation for clinical data. Briefings in Bioinformatics, 23(1 bbbab489, 12 2021.
[79] Sebastien Haneuse, David Arterburn, and Michael J. Daniels. Assessing Missing Data As- sumptions EHR-Based Studies: A Complex and Underappreciated Task. JAMA Network Open, 4(2)x e210184, 02 2021.
[80] Rolf H. H. Groenwold. Informative missingness in electronic health record systems: the curse of knowing, Diagnostic and Prognostic Research, 4(1):8, 2020.
[81] Catarina Pinto, Juliana Faria, and Luis Macedo. An active learning-based medical diagnosis system. In Goreti Marreiros, Bruno Martins, Ana Paiva, Bernardete Ribeiro, and Alberto Sardinha, editors, Progress in Artificial Intelligence, pages 207-218, Cham, 2022. Springer International Publishing.
[82] A Kulkarni, J Terpenny, and V Prabhu. Leveraging active learning for failure mode acquisition. Sensors, 2315:888 2023.
[83] Jonathan Sadeghi, Romain Mueller, and John Redford. An active learning reliability method for systems with partially defined performance functions. 12 2022.
[84] Cao Tong, Jian Wang. Jinguo Liu, and M. Bastos Pereira. A kriging- based active learning algorithm for mechanical reliability analysis with tinecconuuiing and nonlinear response. Mathematical Problems in Engineering, 2019:7672623, 2019.
[85] Jože M. Rožanec, Elena Trajkova, Paulien Dam, Blaž Fortuna, and Dunja Mladenić. Streaming machine learning and online active learning for automated visual inspection **this work was supported by the slovenian research agency and the european union's 's horizon 2020 program project star under grant agreement number h2020-956573. FAAC-PapesSOnI.ir 55(2):277- 282, 2022. 14th IFAC Workshop on Intelligent Manufacturing Systems IMS 2022.
[86] Rong Zhu, Yuan Chen, Weiwen Peng, and Zhi-Sheng Ye. Bayesian deep-learning for rul pre- diction: An active learning perspective. Reliability Engineering & System Safety, 228: 108758, 2022.
15

---

[87] Long Wang, Chen Wenbai, Liu Chang, Chen Weizhao, Liu Huixiang, Chen Qili, and Wu Peil- iang. A prediction method for the rul of equipment for missing data. Complexity, 2021:2122655, 2021.
[88] Kai Zhang and Ruonan Liu. Lstm-based multi-task method for remaining useful life prediction under corrupted sensor data. Machines, 1 1(3), 2023.
[89] Shengfei Zhang, Tianmei Li, Xiaosheng Si, Changhua Hu, Hao Zhang, and Yuzhe Ma. A new missing data generation method based on an improved degan with application to rul prediction. In 2021 CAA Symposium Fault Detection. Supervision, and Safety for Technical Processes (SAFEPROCESS), pages 1-6, 2021.
[90] Mehdi Elahi, Francesco Ricci, and Neil Rubens. Active learning in collaborative filtering recommender systems. In Martin Hepp and Yigal Hoffner, editors, E- E-Commerce and Web Technologies, pages 113-124, Cham, 2014. Springer International Publishing.
[91] Mehdi Elahi, Francesco Ricci, and Neil Rubens. A survey of active learning collaborative filtering recommender systems. Computer Science Review, 20:29-50, 50, 2016.
[92] Neil Rubens, Mehdi Elahi, Masashi Sugiyama, and Dain Kaplan, Active Learning in Recom- mender Systems, pages 809-846. 02 2016.
[93] Xiangyu Zhao, Zhendong Niu, Kaiyi Wang, Ke Niu, Zhongqiang Liu, and Jean-Charles Beugnot. Improving top-n recommendation performance using missing data. Mathematical Problems in Engineering, 2015:380472, 2015.
[94] Benjamin Marlin, Richard Zemel, Sam Roweis, and Malcolm Slaney. Recommender systems, missing data and statistical model estimation. pages 2686-2691, 01 2011.
[95] H Chen and Wang. Active learning for efficient soil monitoring in large terrain with heterogeneous sensor network. Sensors, 23(5):2365, 2023.
[96] Yifan Zhang and Peter J. Thorburn. Handling missing data in near real-time environmental monitoring: A system and a review of selected methods. Future Generation Computer Systems, 128:63 72, 2022.
[97] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Under- standing deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-155, 2021.
[98] John R Zech, Marcus A Badgeley, Manway Liu, Anthony B Costa, Joseph Titano, and Eric Karl Oermann. Variable generalization performance deep learning model detect pneumonia in chest radiographs: cross-sectional study. PLoS medicine, 15(1 :) 1026833, 2018.
[99] R. Bock. MAGIC Gamma Telescope. http://0b..rrg11..2433//22222
UCI Machine Learning Repository, 2007. DOI:
[100] Barry Becker and Ronny Kohavi. Adult. UCI Machine Learning Repository, 1996. DOI:
http:///oo.or//134443//XXX
[101] Cardiovascular-disesss dataset. Kaggle: http://wwwaaagccooouutteteeeteee disease, September 2023.
[102] Rita Moro, S. and Cortez. Bank Marketing. UCI Machine Learning Repository, 2012. DOI: https://doi.org/ http.o///0002442225553336
[103] Jinsung Yoon, James Jordon, and Mihaela Schaar. Gain: Missing data imputation using generative adversarial nets. In International conference on machine learning, pages 5689- 5698. PMLR, 2018.
[104] Steven Cheng-Xian Li, Bo Jiang, and Benjamin Marlin. Misgan: Learning from incomplete data with generative adversarial networks. arXiv preprint
2019.
16

---

[105] Seongwook Yoon and Sanghoon Sull. Gamin: Generative adversarial multiple imputation network for highly missing data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1466-8464, 2020.
[106] Alfredo Nazabal, Pablo M Olmos, Zoubin Ghahramani, and Isabel Valera. Handling incomplete heterogeneous data using vaes. Pattern Recognition, 107:107501, 2020.
[107] Oleg Ivanov, Michael Figurnov, and Dmitry Vetrov. Variational autoencoder with arbitrary conditioning. arXiv preprint arXiv: 02380, 2018.
[108] Trevor W Richardson, Wencheng Wu, Lei Lin, Beilei Xu, and Edgar A Bernal. Mcflow: Monte carlo flow models for data imputation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14205-14214, 2020.
[109] Pierre-Alexandre Mattei and Jes Frellsen. Miwae: Deep generative modelling and imputation of incomplete data sets. In International Conference Machine Learning, pages 4413 4423. PMLR, 2019.
[110] S. Jäger, A. Allhorn, and F. Bießmann. A benchmark for data imputation methods. Frontiers Big Data, 4:693674, 2021.
[111] Daniel J Stekhoven and Peter Bühlmann. Missforest non--oon---nnnneee non-paramctric missing value imputa- tion for mixed-type data. Bioinformatics, 28(1):112-113 2012.
[112] Esther-Lydia Silva-Ramirez, Rafael Pino-Mejías, Manuel López-Coello, and María-Dolores Cubiles-de-la Vega. Missing value imputation on missing completely at random data using multilayer perceptrons. Neural Networks, 24(1):121-12) 2011.
[113] Euredit. Interim report on evaluation criteria for statistical editing and imputation, 2005.
[114] Jaap Brand. Development, implementation and evaluation of multiple imputation strategies for the statistical analysis of incomplete data sets. PhD thesis, 1999.
[115] David Heckerman, David Maxwell Chickering, Christopher Meck, Robert Rounthwaite, and Carl Kadie. Dependency networks for inference, collaborative filtering, and data visualization. Journal of Machine Learning Research, 1(0ct):49-755 2000.
[116] Trivellore E Raghunathan, James M Lepkowski, John Van Hoewyk, Peter Solenberger, et al. A multivariate technique for multiply imputing missing values using sequence of regression models. Survey Methodology, 27(1):85-96, 2001.
[117] Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. arXiv preprint arXiv:2309.03409 2023.
[118] Nicolás Astorga, Tennison Liu, Yuanzhang Xiao, and Mihaela van der Schaar. Autoformulation of mathematical optimization models using llms. arXiv preprint urXiv:2411.01679. 2024.
[119] Tom B Brown. Language models are few-shot learners. arXiv preprint arXhv:2005.14166 2020.
[120] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, al. -haon-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
[121] Wenlong Huang. Pieter Abbeel. Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International conference machine learning, pages 91 18 9147. PMLR, 2022.
[122] Zirui Zhao, Wee Sun Lee, and David Hsu. Large language models as commonsense knowledge for large-scale task planning. Advances in Neural Information Processing Systems, 36, 2024.
[123] Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. The 2023 Conference Empirical Methods in Natural Language Processing, 2023.
17

---

[124] Max Ruiz Luyten and Mihaela van der Schaar. A theoretical design of concept sets: improving the predictability of concept bottleneck models. The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[125] Aditya Taparia, Som Sagar, and Ransalu Senanayake. Explainable concept generation through vision-language preference learning. arXiv preprint arXiv:2408. 13438, 2024.
[126] Boyi Li, Yue Wang, Jiageng Mao, Boris Ivanovic, Sushant Veer, Karen Leung, and Marco Pavone. Driving everywhere with large language model policy adaptation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 14948-14957, 2024.
[127] Tsung-Han Wu, Long Lian, Joseph E Gonzalez, Boyi Li, and Trevor Darrell. Self-correcting llm-controlle diffusion models. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pages 6327 6336, 2024.
[128] Paulius Rauba, Nabeel Seedat, Krzysztof Kacprzyk, and Mihaela van der Schaar. Self-healing machine learning: A framework for autonomous adaptation in real-world environments. In The Tiirty-Eighth Annual Conference on Neural Information Processing Systems, 2024.
[129] Samuel Holt, Tennison Liu, and Mihaela van der Schaar. Automatically learning hy digital twins of dynamical systems. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
[130] Hanqing Yang, Marie Siew, and Carlee Joe- Wong. An Ilm-based digital twin for optimizing human-in-the loop systems. arXiv preprint arXiv:2403. 2024.
[131] Nathan Zixia Hu, Eric Mitchell, Christopher Manning, and Chelsea Finn. Meta-learnine online adaptation of language models. In The 2023 Conference on Empirical Methods in Natural Langugge Processing, 2023.
[132] Katarzyna Kobalczyk and Mihaela van der Schaar. Informed meta-learning. 2024.
[133] Zhiyuan Hu, Chumin Liu, Xidong Feng, Yilun Zhao, See-Kiong Ng. Anh Tuan Luu, Junxian He, Pang Wei Koh, and Bryan Hooi. Uncertainty of thoughts: Uncertainty- aware planning enhances information seeking in LLMs. The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[134] Anonymous. Active task disambiguation with LLMs. In Submitted to The Thirteenth Interna- tional Conference on Learning Representations, 2024. under review.
18

---

Appendix: POCA: Partially Observable Cost-Aware Active-Learning with Large Language Models.
A Appendix A: Real-World Use Cases
Table 2: Real-world use-cases of POCA. We outline real-world scenarios where the POCA frame- work can have an impact. For each problem domain, we describe partially observable features, labels, and the underlying predictive task. We categorize references into three types: A) where active learn- ing is employed, B) where predictive modelling is performed in the presence of partially observed features, and C) active learning is applied to partially observed settings (with data pre-processing to handle missing features). The symbol stands for acquisition costs.
Setting
Observed Features
Acquirable
Task
Oudumer Chum
cssumer (danugnphis.
custemer inturaction salidiation Clum arulyes cudumer stalos
plin opee. usuge panenns).
veys * collection and operational
over
Madeeting
demographics basic pur-
Cansurer preferences
perception changes.
deployment
procesing.
Busi financial information
Ciedit Listrr
invesneeut poctialiss
defeshing,
lovd, emplosment statie, adtad markel tnands.
stimal cres, dats acquiction from catemal
roquinal for ostermes momieedt ihe noddell)
and pivazy
Healticore Dag-
Basic potent infoemation (deme-
Resuls from specitti medical tests (biood 1e9t, MKI
Diagnoss specitic diseases Clinical evalusoon
NU Ki
tonlis (Medicine)
graphics, medical histery, basic vi- bis)
scass, et.). & Medical test consooeeation costs
Expert anilyus or medicll vest ther could more exgersise mhnn seqpining fiseurre
XM
20
X11
Predictive Maine-
Rezulax operation data (machite
Desailed inspcctios idvanced senvor data (acoustic
Failure eveats or incimmaace needs ie Risk not
sance Manulia-
nutine, semperature, vibeston lev. dlc)
eniisioss cltrsonic testlng). a operational costs
doing marranence. Inspection or equipeent Callure osds.
uuingg
Oueaniaal
User actvily (pag views, clicka),
Detailed puncuase hisury, product revicu icsl,
Reseemendssion Rick wonn moummnnee
X0:2
Boummene
bais demograpiic
cotauner peeferences survey, medla
tims
Survey deployment
(rmpprston,
specific
spe-
Povinomental
Monioning
midiy. pescipttation) pellage Openttioal messuring way. atalysis samplex
Table 2 illustrates that active learning is extensively utilized in a variety of real-world application scenanos. Furthermore, it is not uncommon in these contexts to encounter situations with incomplete data, which can harm generalization [2 6 97 988 capabilities of downstream models. The breadth of related work covers diverse sectors including customer churn prediction, marketing research, healthcare diagnostics, and predictive maintenance. While active learning is adept at selectively querying labels in scenarios where data is fully observed, its application in the context of missing data is less clear. The challenge is compounded by the fact that, in similar problem settings, it is not always guaranteed that features will be fully observed. This reality underscores the need for alternative machine learning techniques to address such challenges. Our proposed approach, POCA, offers novel solution for applying active learning in scenarios with partially observed data, taking into account realistic cost constraints.
19

---

B Appendix B: Acquisition metrics for partially observed scenarios.
Proposition 1. Let p( be an acquisition metric that can be written as 1(g, [l)), with 9 and representing the same variables observed in traditional AL (Table 7V, and with Y, X; as previously defined. lfg 1 Xj) the following equality holds:
T(g (Y, x;) Bay 1(9, Y(g. :)
Proof: We can decompose the left part of Eq.
as: I(G, (Y, x3) G(
(10)
Using G I X2, 1(G X, 0, the first term the right of Eq. (10 cancels, obtaining 1(9, - E,, I(G Y |ay. o) concluding the proof.
Corollary Under the assumptions of Proposition 7 the subsequent equivalent inequality is established:
E,, +) (())
(11)
Equality is attained when G X,
Proof: Symmetrically before decompose the left part of Eq.
9 as:
I(G, (Y, x3)1,) = IG + [( x)
(12)
Using proposition we obtain:
Ez, I(9, Y (zj o) Y|) + 1() X, Y)
(13)
Clearly the equality is obtained when G 1 YY since the mutual information zero. When taking T(g, X5 Y, 2 0 we obtain:
Ex, ((G. 2 1(G, Ylo),
(14)
concluding the proof.
Observation: Note that, the independence assumption G .L. Xí1 always hold for the class of supervised learning models we consider. G is a random variable representing the generalization capabilities of the model w E W, with random variable Ω. This random variable is subject to model training A, which can be written as the mapping between the training set D and hyperparameters h E H to the output w, i.e., A: : D x H + W. Additionally, the model prediction is a mapping P between the model w and the input æ e X to the output Y. i.c., P W x X' + y.
It is crucial to acknowledge that the "world generator" influences D. X. and Y, but does not directly affect S2. Given that observable, any connection through this path is cut. The sole connection of Ω2 to X and Y is through the mapping PP which establishes a causal structure: Ω + Y +- X. According to this structure, S and X are generally independent unless Y observed, leading to a dependence due to P creating configuration known an "immorality" among these variables. This explains why G 1 Xil holds; however, this independence may not persist in scenarios where G IL X,lo,Y.
In the context of work the input X be decomposed in X -ooo X, the observed part of the random variable and the unobserved part of the random variable. Consequently, X, follows the same independency assumptions of X.
20

---

C Appendix C: Monte-Carlo Estimates
In study, we evaluate BALD and EPIG, along with their partially observed counterparts, PO- EIG and PO-EPIG. For completness, we show estimatation the vanilla metrics, and later their corresponding partially observed extensions.
C.1 PO-EIG and BALD
We follow a similar notation to 28 For categorical variables, BALD can be decomposed as:
I(2, Y 1o D) = Hy D) H(y xo. 22, D)
(15)
E pp((w) [ pogg + poollle,,) (ogpoo/ww]
(16)
po(((oo) log + K
Po Po
log (ol wa),
k= yey
(17)
where K represent the total number of parameter samples W/ wk given D. Here,
(w(w) from the posterior distribution
po(((x,)=
K Σ (o(
(18)
PO-EIG extends this formulation to include missing features as conditioning samples ãj from the GSM, accommodating partially observed settings. To illustrate, let's first consider the case where the metric is conditioned over all unobserved features, i..., j" = 0, which correspond to results of Figure Following similar dccomposition, we can approximate Ee, I(2, Y 2oo D, ..))
Ea, 1(2, Y| D,ū;) 2
Σ log (oll)=') yEY
(19)
Poly le wa) log P polle wa)
BEY
here L represent the total number of Monte-Carlo samples from the GSM, with øtj one possible sample. Here,
(o(()=0') po Vo 5
(20)
1 K
Ps (,)(²")
Polle ww).
(21)
What happen when j doesn't 't consider unobserved features?. This scenario useful when we want to asses the impact acquiring of subset of the unobserved features. Utilizing smart notation, this approximation can be stated identically as Equation (19, but the estimation of the predictive distribution changes slightly:
(yla wx)
Σ p To. at 27 wk),
(22)
here, P are a total new MC samples from the GSM. This marginalization trick is necessary to deal with model that expect fixed size inputs.
21

---

PO-EPIG and EPIG C.2
For PO-EPIG and EPIG, we replace the sub-index eval by Similar before, for categorical variables, EPIG can be decomposed as:
I(Y,, Ma,, x, D) Ez, [[,,, FI D)]
(23)
Ez, KL vo 2. po
(24)
1
(u, ya ( Po (val
Vo To x:
ΣEΣ yEy N.EV..
x.Th xTt) log
(25)
here,
k
(V. yo 20: x²") = Σ
Po(y To. wk wx)p wk).
(26)
K
k=1
Do(y)oo))
Σ Po(yl wi),
(27)
k=1
po(u.l=?)
Σ Po ww))
(28)
Similarly as before, PO-EPIG extends this formulation include missing features when conditioning on samples ē, from the GSM. We now derive the estimation of this metric when estimating PO-EPIG when considering all the unobserved features j. We can decompose Ex, Ee,I( Y ( Xx, D, .2) as:
E IY., Y
Co: X., D,
ΣΣ
Σ
Po Po | la',a"n)
M m= VEV E.
Po (y,, Uo 1 x x
x log
Do (3) x²)) po (y. x223
(29)
1 K
Do(y,,ala "a",,"""" ,,
Σ
( To te ww)poly (
(30)
K
Poly)=') Σ
(vl To कy wx),
(31)
k=1
K
H
Po (x
(y. 1xt arj wx),
(32)
k=1
Note that the marginalization step of the "evaluation set" is also necessary since we also assume to be partially observed. Here, ær) are Monte-Carla samples necessary make the predictive model receive fixed inputs. While expensive, PO-EPIG can also compute metrics when only a subset of features is estimated to acquired. The trick similar to PO-EIG, being necessary a marginalization across the j' for the the predictive mode receive fixed-size inputs (as )2
22

---

C.3 Illustration of efficient MC estimation
As outlined in the earlier sections, we approximate the PO-metrics through the techniques of condi- tioning and marginalization. Figure g provides a visual representation of this approximation.
In Eq. (8). for efficient computation in the marginalization of j', substitute po ( 2ÿ) with po(a23' (. using the same samples used to compute the uncertainty reduction across all unobserved features. This approximation does not significantly compromise precision, we exclude the least relevant feature sequentially. Consequently, the samples used to marginalize an "irrelevant" feature remain minimally impacted by the overall sampling strategy.
2 (shacevol)
3
# (marciualass))
(couditiooe)
Rj
myltr U (8s))
(as((a))
polao L (aga))
Beylnv(t u u.))
Figure 8: Illustrative diagram demonstrating the application of conditioning and marginalization techniques in the estimation of P0-metrics for an arbitrary instance.
23

---

D Pseudo-code of the whole acquisition process
Algorithm 2 μPOCA Algorithm
Initial pool indexed initially [7]. initial partially observed unlabeled dataset Du. initial partially observed Require: laheled dataset D:, identify costs for every element in the pool set, identify number of features J, select downstream model o, select Generative Surrogate Model parameterized by 0. select number of Monte-Carlo samples S. If using heuristic select the number of instances to analize for subset of feature sclection.
1: 0+ maximize 1itehioe__keeood(( Dz)
2: Ps +- generative wttpeemmptaaatiiionooo P.S)
3: while stap_condition() == False
4: D) f- impute
ittpute_msseinng___ccpp missing (D 0) fixed-size inpul
# Imputing when using a predictive model that receive
5:
á maximize_liie 1it.noood Di)
6:
if sase_heuristcc then
7:
4- compute_ top uncertain instances(o, (4.())
8:
end if
9:
P= =]]
10:
for E [1] do
11:
use wwwrietiiiiiio and I" then P.add(-inf), inf). ([a
12:
13:
continue
14:
end if
15:
j = [J]
16:
while (f.j"") do
17:
===1
18:
for r E do
19:
(t.jj" then
20:
compute_eueeeeeeeeeee cutpuuut_uncertittty..r Ps, Ps.i,j"(o))
21:
( G.a))
22:
else
23:
ULadi(-inn
24:
end if end for
25:
26:
0¹ arg max
27:
j = = while (.add(pe.s>) ), Faad(j") end for
28:
29:
30:
31:
arg m p' F(F"
32:
D: Di u P[""
P
34:
Ps Ps Po[i*]
35:
I6I
36:
end while
We assume that GSMs will be trained using available unlabeled data, which may be either fully observed (if a bank of fully observed unlabeled data is available) or partially observed (using the pool set itself), respectively. This assumption enables us to train the GSM and generate imputed values for the pool before the acquisition process begins. The complete acquisition is detailed in Algorithm
We define the generation cost Cg and the downstream cost Ca. Algorithm 2 indicates that the sampling cost for GSMs is O J J.S Cg), where / is the number of instances, is the number of features, and S represents the number of Monte Carlo samples. The inference cost of the downstream model is OTT J. Ca).
The cost of acquiring subset of features depends the restriction ((.j). which is bounded by J (the case where J features are discarded for acquisition). The cost for acquiring a subset of features for single instance is O(J 5 Ca) using Algorithm [ To reduce this overhead, we first select the most informative instance (assuming all features are acquired) in O J S Ca) (L7 in Algorithm 2), and then select the subset of features to acquire using Algorithm I (10 in our case) in order O R- S Ca). This analysis shows that the most critical factor is the number of features J, as affects both sampling and the downstream model (quadratically in this case).
Another consideration is when the available data is insufficient for reliable GSM. In this case, any additional features acquired during the acquisition process can be used to update the GSM weights periodically, which increases the costs.
24

---

E Datasets
Datasets were constructed such the number of pool samples were numerous enough to determine the impact of the acquisition performance any confounding effect. The distribution of pool set were selected maintaining the distribution the original dataset similar to previous work [28]. The "evaluation distribution" in EPIG is follows the same distribution of pool set. As mentioned in the main text, we maintain a small unlabeled historical set for GSM training allowing fair comparison with Active Learning. For large datasets like Adult and Housing limit their maximum original data size avoiding unnecessary costs in Monte-Carlo sample generation. The test set is defined as the 30 % of the intial dataset.
Magic 999 Original data size of 19020 samples. Historical of 1000 samples. Pool set
distribution; ClassO: 4980 samples. Classl: 2700
Adult [100] Original data size of 19020 samples (after cut). Historical set of 1000 samples. Pool set distribution; ClassO: 5760 samples. Class 1920.
Housing. Original data size of 19020 samples (after cut). Historical dataset of 1000 samples. Classl: 3840.
Pool set distribution; ClassO: 3840,
Cardio [00]. Original data size of 100k samples. Historical dataset of 1000 samples. Pool set distribution; ClassO: 3000,
Class 3000.
Banking [102] Original data size of 45211 I samples. Historical dataset of 400 samples. Pool set distribution; ClassO: 2000, Classl: 500.
We did slight modification in datasets allowing LLM better comprehension. For Magic, we didn't include fCConcl since its name similarity with fConc. For CMC dataset, we change categorical values O and to the categories that were represented on metadata. For example, in column "Standard-of- "ving_index" instead of using 0,1, 2, 3 we use Low, Medium-Low, Medium-High, and High.
With this adjustment the final columns of these datasets are:
Magic. 9 numerical columns: fLength, fWidth, fSize, fCone, fAsym, fM3Long', fM3Trans, fAlpha, fDist.
Adult. 8 categorical columns: workclass, education, marital-status, occupation, relationship,
race, sex, native-country. numerical columns: age, fnlwet, education-num, capital-gain, capital-loss, hours-per-week.
Housing. 8 numerical columns: MedIne, HouseAge, AveRooms, AveBedrms, Population, AveOceup, Latitude, Longitude.
Cardio. 10 numerical columns: ID, age, age_years, height, weight, ap_hi, ap_lo, lo, bmi categorical variables: gender, cholesterol, gluc, smoke, alco, active, pp__categor_eencone
Bank. 2 numerical columns: age, balance. categorical variables: job, marital, education, contact, day_of_ sta__of_eeeee week, default.
For model training, we use one hot encoding for categorical variables.
F Large Language Models, training and evaluation
We analyzed primarily one open-source findings show that instruct LLMs generally outperformed their non-instruct counterparts. In what follows, we give component used and general overview of the prompts used:
with more in Section J Preliminary
Input: The input is the tabular information provided to the LLM. Here, it is in the form of "[feature A: value A, feature B: value B. feature C: value C)", for possible features and values.
Query: The query represents the information requested by the LLM to generate response. It should only include the attribute names, formatted follows: "[feature [Mask], feature E: [Mask], feature F: [Mask]".
25

---

Response: The input is the tabular information provided to the LLM. Here, it is in the form only values: "(feature value feature E: value E. feature F: value F)
The specific prompts were adjusted to specific LLM format.
E.1 Mistral77B-Instruct-xv..
Training time: approximately 1-2 hours
(**Input, **Query) #iiitRespone:: (**Input, **Response]
Test time: The total number of generated samples varies depending on the dataset, with processing times ranging from hours.
[**Input, **Query) } ##sposse
Here as some samples and the respones On Magic:
[fAsym: -90.6431, fM3Trans: 169, fDist: 250.51, fConc: 0.1637, fAlpha: [Mask], fM3Long:
[Mask], ISize: [Mask], fWidth: [Mask], fLenggh: [Mask]] ###Response: {fAsym: -90.6431,
IM3Trans: -10. IDist: 250.51, fConc: 0.1637, fAlpha: 1.719, fM3Long: -33.7346, ISize:
3.3316, (Width: 28.4232,
fLength: 69.8763)
[fConc: 0.3174, fM3Trans: -4.5924, fAsym: -28.5264, fDist: 150.686, fSize: [Mask], fM3Long:
[Mask], fWidth: [Mask], fAlpha: [Mask], fLength: [Mask]] ###Response: (fCone: 0.3174,
fM3Trans: -4.5924, fAsym: -28.5264, fDist: 150.686,
fSize: 2.6819,
fM3Long: -21.4579, fWidth:
16.6296, fAlpha: 18.3037,
fLength: 27.3688)
[[M3Trans: 19.0026,, IDist: 294.682, fCone: 0.1546, fAsym: -13,,729, fAlpha: [Mask], fLength:
[Mask], IWidth: [Mask], IM3Long: [Mask], fSize: [Mask]] ###Response: [fM3Trans: 1990026,,
[Dist: 294.682, iCone: 0.1546, [Asym: -137.729,
fAlpha: 0.294, fLength: 124.816, IWidth:
38.3973, IM3Long: -71.1274,
(Size: 3.3655]
IIM3Trans -1 -1.0288, iDist: 85.2971. fConc: 0.1765, fAsym: 32.2464. fWidth: [Mask], fM3Long:
[Mask], fAlpha: [Mask], fLength: [Mask], fSize: [Mask]] ###Response: [fM3Trans: 11.0233,
fDist: 85.2971, fConc: 0.1765, fAsym: 32.2464, fWidth: 27.559, fM3Long: 551,3547, fAlpha:
21.297, fl.enghh: 56.6845, fSize: 3.3454)
F.2
Masking
We tested two masking strategies. First, when we had access to all the historical data, particularly when comparing against traditional AL in the main text. Second, when certain features might be missing, as encountered in Appendix[] In the first scenario, we randomly masked some observed information, always ensuring at least two features input to prevent overly complex tasks for the LLM. the second scenario, due to the partial observability of data, some features could be less observed than others, leading to varying degrees of missingness.
F.3 Training specification
We train the models using QLoRA using bit quantization, r 32, lora lora_alpha= 64, for 10000 steps, with 2 samples batch size, and a learning rate of 7.5e-5.
26

---

G Generative Surrogate Models
In this section, we outline the core desiderata for Generative Surrogate Models to be used within the μPOCA framework and compares related imputation methods against the desiderata.
[P1] Generative capability: The model must model a non-deterministic distribution over the unobserved features effectively identify and prioritize the most relevant features. To exemplify, we consider z as the subset of current features under review for acquisition, and z as the features already excluded from acquisition. PO-EIG's acquisition metric strategically selects the feature, that maximizes uncertainty reduction among all possible features, v, considered for exclusion from the set j". This effectively minimizes information loss. Formally, this is expressed as:
arg max En Bo 12, Ye, Lo, D).
(33)
"Ej"
In contrast, employing a deterministic GSM modifies the acquisition strategy for imputed features ã, simplifying to:
arg max (S2, Ya Ti D).
eEj"
arg max I(22, Y| To. D).
(34)
"Ef"
The second equality arises from the fact that we are limited to a single value due to estimated conditioning and marginalization. Consequently, any permutation of conditioning sets j and marginalization sets j' will lead to the same metric for a deterministic GSM. (34) explicitly demonstrates that employing deterministic GSM results in acquisition metric independent of v, the feature under consideration for exclusion through a greedy approach. In conclusion, the GSM should model stochastic distribution for acquisition purposes.
[P2] Learning partially observed data: The GSM must learn from partially observed training data, where different instances will have varying observed features.
[P3] Sample-efficiency: Given the potential scareity and variability of feature observability of training data, the GSM must efficiently learn from limited samples with different observed features.
[P4] Supports mixed-type variables: To support broad range of data types, the GSM should enable generative modelling across both continuous and discrete variables.
Table 3:
Overview of imputation methods. Comparison based on key desiderata of GSM.
Desiderata
LLM
Generative imputation
Discriminative imputation
Sample statisties
References
103 109
110 16
[P]] [P2]
[P3]
[P4]
Flexibility of Large Language Models. We believe that Large Language Models hold significant potential as sample-efficient methods for feature estimation. Although LLMs not specifically pretrained for tabular data estimation, their general capabilities have demonstrated effectiveness across various domains. These general abilities have been successfully applied in optimization [2 117
118).
reasoning 119 120), planning 121-1233,
concepts [24 125). autonomous adaptation 126 128
digital twin construction 119
130
meta-learinng 131 132 uncertainty reduction 133 134
They have also shown promise as few-shot tabular learners The application of LLMs in the intersection of tabular data tasks, particularly in the last two contexts, provides direct evidence of their viability in this area and to be used GSMs in the context of data acquisition.
and as tabular data generators
49
in supervised settings [52
53
27

---

H Comparing with Active Learning
Serting
Unceriainty reductionn motrio
Angrieed dazs
Tee perfermtore
Rasteical labelled
(all : Es.lae(a,
Sd. fasi si Lobel
mondiisnig allisaiio und mergiialiittion
Cuepprta metti co. semplo
suisle)
fenturea
Halte)
Compute metric with
Label
noquized
Pescurs soquiedd
Figure 9: Scenario 1.
Sctting
Ulocrluinly redlaction motriia
Acqured data
Tea perfermacc
Hiskorieal Jabelled
(aa( im (1
Sel. fratumne Fi and
Cheornte sumple coditiining marphalltation
Chequaie mutile canslle
Label
#o : varies depending instance
(a(S.)
Ingesteettbl
Label
All festioros Sag
soquired
copuira
Penturra imputed
Figure 10:
Scenario 2.
We used Scenario 1 as indicated in the main text. Another scenario where traditional active learning metrics can be applied is Scenario 2, which utilizes conventional imputation methods. The results for this scenario are shown Figure [2 when all features are acquired for the Active Learning metric. As discussed in the main paper determmistic imputation does not allowed for the acquisition of subset of features, which is illustrated in Figure
PPCA in feature selection.
Uncerteinty
μPOCA
27
0.0
Uncertainty!
-1.
Insuted
Clas
-20
5 Na
s4mte perdictiona
suaue
Cuullicedd prediction Hypsthenis
Figure 11: Figure (a) shows the distribution of X2 and X3 conditioned on With estimates of X2 and X3, μPOCA can identify the relevant feature (X2) and the relevant region. In contrast, AL metrics might use deterministic imputation (green), which does not reveal feature relevance or area of importance under partial observability. This because a point estimate can not explore the X2. X3 and how their variability affects the outcome.
28

---

I Additional Results
Mitral vB3 seggl
Miannd v0.3 sdelt
Stana v5 luriddas
SEarsd v0 cedio
Nistrd vil Lendaa
Sonaker
rotirr (7005 (xon)) Ouado
Figure 12: Pool set.
Figure 12 illustratee that Mistral Instrll-tnttutt.... trained LLMs solely on the Pool dataset, introducing a different scenario than the discussed in the main text. In this setup, only three features are consistently observed in the pool while others may be absent with uniform of probability. While the effectiveness of LLMs would depend on case-by case scenarios. In this demonstration, we underscore their viability and potential Generative Surrogate Models (GSMs).
I.1 Varying uncertainty metrics
2012 roigle
BBG bakking
BBG cardd
EIC Iweaien
Wande AUSCAOO
Staree
Stader
Saado
EPIG widd
02PC cardo
EPP
Stabec
Stader Bataacs
Shabk
Figure 13: Partially observed active learning metrics and their fully observed counterparts.
29

---

J Analysis of GSM
The performance of μPOCA in partially observed settings fundamentally depends on how well the GSM approximates the distribution of the unobserved features. There two key factors that affect the quality of this estimation:
GSM's approximation power: Referring to the model's capacity to accurately model the unobserved features.
Intrinsic characteristics of the dataset: Referring inherent correlations between ob- served and unobserved features. Indeed, lower correlations are more challenging.
In what follows, we investigate each factor in turn, [GI] studying the impact of different GSMs and [G2] investigating different dataset characteristics. This approach aims to delineate the conditions under which μPOCA is expected to excel.
[G1] GSM impact acquisition performance
Setup. We evaluated various LLMs (including Mistral-7B Mistral-7B-Instrucee Instruct-v0.33 Gemma2, and Llama-3.1) as GSMs to assess how model quality affects acquisition performance.
Analysis. Figure 14 demonstrates that GSM quality significantly influences acquisition results. Notably, Mistral-7B consistently outperforms the alternative GSMs, with one exception in the housing dataset. Interestingly, Gemma 2 performs well on this benchmark, highlighting this inter- model variability. The Cardio dataset further highlights these differences, with Mistral-7B performing significantly better than the other models.
Takeaway. These findings underscore the critical role of GSM quality in acquisition performance.
Mirrd LB3 wook
Xiann 20.3
Matta v1.3 bucidda
Aiaaed vt.
Lvndaa
Laae Wi
Dian banking
cards
130
2280
Puskes
BALD
120-500
Ocadk
Figure 14: Varying LLMs with MistrallB-IIntrra v0.3 based on EIG acquisition metric.
30

---

[G2] Performance across varying data characteristies
Setup. Next, we turn our attention to investigating how the data distribution affects acquisition performance. We are particularly interested in analyzing the effect of the correlation between unobserved features unoos and observed features Xobs on acquisition performance.
To demonstrate this, we examine a scenario where (1) Xumots correlates with the outcome, while (2) observed features do not. In this context, the GSM becomes crucial for downstream performance, as Xob. alone provides insufficient information to predict outcomes accurately. As such, the GSM must effectively model the relationship between Xobs and X unobs acquire missing features critical for predicting the outcome.
We model both Xobs and X unobe two-dimensional random Gaussian variables centered at and establish a specific controllable correlation between them through p.
12
px Nunobs 12
Σ
(unhhXunoha
1g
The label is then constructed to be independent Xoba using the orthogonalization
Xorthggonal
X
uunhe
Xobs (X obs Xobs)
X
obs Xunbe
which we use to construct the label using
logits
logits>=0
1 e Σ Xiehasraa
Analysis. Figure 15 illustrates how varying between Xobs and Xunobs empirically affects variable correlation, c, validating synthetic experiment design. Figure 15p analyzes EIG (traditional active learning without GSM) and PO-EIG with varying p. We note that when the correlation between Xoos and X wunhh is low, GSMs provide no performance benefits. However, correlation increases, the performance gains of PO-EIG over EIG expand significantly, confirming our hypothesis.
(a) Correlations between Xats and Y. and between Xunohs and Y is roughly unaffected across different of p.
(b) Performance EIG Across Datasets with Varying Correla- tions Between Observed Features (Xot.) and Unobserved Features (Xunohx). Observation: As correlation increases, the performance gains of PO-EIG (using GSM) over BALD (traditional AL) becomes more notable.
Figure 15: Synthetic experiments. Figure (a) visualizes the characteristics of the synthetic data with different values of p. while Figure (b) demonstrates the performance EIG (BALD) and PO-EIG as the degree of correlation between observed and unobserved features varies.
31

---

K PO-EIG VS BALD
Figure T6 Tolllustrates the comparison of uncertainty reduction between PO-EIG and BALD (EIG) at iteration 50 of training with seed zero. It is evident that PO-EIG consistently achieves equal or greater uncertainty reduction than EIG. This empirical observation supports the validity of Corollary and consequently, substantiates the assumption made in Proposition
60
Esiie delataae
Figure 16: PO-EIG BALD metrics various scenarios iteration 50.
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately reflects the claims made in the paper. We formalize the POCA problem and provide an instantiation of POCA which we validate empirically.
Guidelines:
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section discusses possible limitations with mitigation strategies.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
33

---

We provide all assumptions for proofs in Sections and 3. We also provide additional details in Appendix B and C.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Experimental details are provided in Section 4. with further details in Appendix D and E. The implementation of our instantiation closely follows Section com/
Code can be found at: https:://// http//ggtttb.../.. com/ ADDUSER/POCA, or wanderschaarlab/POG.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
34

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [No]
Justification: Besides the descriptions in Sec 4, we also provide details about the algorithms and data in Appendix D and E.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https::/nnip..cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nips. wwtpcc/pccppcpcccceooooooop httpc//gguu.eeeeeeeuuuubiiiiiiii wwwpanssssonppoo swiniisssiiiiiiyyy Policyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the details on data, training, prompts etc for the experiments are provided in Appendix D and E.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: Error bars (95% confidence interval) are included as relevant over 15 seeds for the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
35

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The compute details on the experiments are provided in Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: :tttaps://neuriiii newrips....../. cc/ppblicc/.hiccccudeeee
Answer: [Yes]
Justification:
We have read the code of ethics do not violate any of the dimensions.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We highlight broader impacts in Section and of the paper, as well as, Appendix A - discussing both positive impacts and possible negative impacts (e.g. biases).
Guidelines:
The answer NA means that there is no societal impact of the work performed.
36

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Not applicable - our paper presents process for data acquisition - which does not fall into this category.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Appendix D provides details and/or citations for all the open source data assets used in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
37

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: datasets.
The paper does not produce new assets such as datasets, but uses existing
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not have crowdsourcing experiments research with humans.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We do not have crowdsourcing experiments or research with humans that would need an IRB.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
38

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
39


Reviews:
[{"review_id": "FJYv5gopt8", "reviewer": "Reviewer_tNwd", "summary": "This paper introduces a novel active learning framework for optimizing data acquisition in scenarios with partially observed features and high costs. The proposed method, POCA utilizes GSM, specifically LLMs, to impute missing features and improve active learning metrics. By integrating Bayesian methods to maximize uncertainty reduction, the proposed active learning method can select features and labels to enhance predictive performance while minimizing acquisition costs.", "questions": "There are many typos, the author should carefully check the full manuscript. For example, Line 721 \"?.\" and line 723 \"Figure ??\".", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The task itself is novel, optimizing the acquisition of data (features and labels) in scenarios where information is partially observed and the cost of acquiring additional data is high. The author defines the cost of active learning from a new aspect, it is interesting.\n\nAdditionally, the combination of GSMs (especially LLMs) to impute missing features is also a novel aspect.", "weaknesses": "1. The Monte-Carlo-based approach in POCA would be slow and with high computational cost due to its reliance on repeated random sampling to estimate quantities of interest.\n2. This model is limited to tabular data.\n3. Sometimes the performance of POCA is not as good as random sampling, POCA may vary depending on the specific characteristics of the dataset and the underlying correlations​.\n4. Please consider more active learning methods as baselines except for BALD and random sampling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We are grateful to the reviewer for their insightful feedback that has improved the paper.*\n\n---\n## [P1] Computational costs of MC sampling\n\nThank you for highlighting this concern. We agree that it is important to discuss $\\mu$POCA's computational costs. We approach this discussion by **(1)** providing a detailed algorithm for complete acquisition in `Algorithm 1` (in the attached PDF), **(2)** detailing the complexity below, and **(3)** discussing how this cost can be alleviated in future works.\n\n**Training process.** Before we discuss inference complexity, we note that GSMs will be trained using available unlabeled data, which may be either fully observed (if a bank of fully observed unlabeled data is available) or partially observed (using the pool set itself). After the GSM is trained, it is used for inference during the acquisition process.\n\n**Inference complexity.** Using notation consistent with the paper, $I$ is the number of instances, $J$ is the number of features, and $S$ is the number of Monte-Carlo samples. A direct acquisition would consider selecting subsets of features over all instances, resulting in a complexity of $\\mathcal{O}(I\\*J^2\\*S)$. To reduce this overhead, we introduce an approximate algorithm (detailed in S4.3), where we **(a)** first select the most informative instance (assuming all features are acquired), then **(b)** select the subset of features of that instance to acquire:\n* **(a)** As we now make more explicit in `Alg 1` (in attached PDF), the first step involves identifying the instance to acquire, which has a sampling cost of $\\mathcal{O}(I\\*S)$ (note, all unobserved features are sampled jointly).\n* **(b)** Once the instance is selected, the cost of acquiring a subset of features is bounded by $\\mathcal{O}(J^2\\*S)$.\n\nOur approximate procedure thus has a total complexity of $\\mathcal{O}(I\\*S + J^2 \\*S)$, compared to  $\\mathcal{O}(I\\*J^2\\*S)$ of the exact procedure. This analysis indicates that the dominant factor is the number of features $J$ as it affects sampling costs quadratically.\n\n**Future work.** In our work, we estimate uncertainty using Monte Carlo samples. However, a promising area for further research is improving efficiency by avoiding sampling with tractable analytical expressions. For example, assuming that features and labels are jointly Gaussian allows us to compute Eq. 3 in closed form. Other conjugate distributions can also be explored. Another approach to enhancing the efficiency of sampling-based approaches is in-context learning, as generative imputation can be done in parallel for multiple instances.\n\n**Actions taken:** The above analysis and `Alg 1` (attached PDF) have been included in App J, with appropriate references in the main text.\n\n---\n## [P2] Focus on tabular data\n\nWe agree that the primary applications of POCA are in tabular data, and we focused our empirical analysis on this domain. However, we emphasize that tabular data is ubiquitous in the real world, particularly in fields such as finance, healthcare, industrial engineering, and retail, *where the POCA problem is most likely to be applicable.*\n \nHowever, our formalism and method can be readily extended to other modalities, such as images, by treating pixels or pixel patches as features. Although this is less common, practical applications could exist in fields like medical and satellite imaging, where occlusion caused by noise is prevalent. In these cases, it's crucial to determine when a sample requires additional information (features) for accurate prediction and model training. Similarly, interactive robots that learn through vision can also benefit from this approach.\n\n**Actions taken:** We have **(1)** added discussions on extensions to other modalities in the Future Works section (S5) and **(2)** extended Table 1 to include example applications involving other modalities, demonstrating the versatility of POCA.\n\n---\n## [P3] Conditions that affect POCA performance\n\nThank you for highlighting this concern. We acknowledge that the performance of $\\mu$POCA-based methods can vary in certain scenarios. The key question we want to answer is: in what settings do $\\mu$POCA methods perform poorly? We hypothesize that this can occur when the GSM does not accurately learn the underlying data distribution, which is affected by (1) the approximation power of the GSM, and (2) the intrinsic properties of the partial observability. \n\nTo investigate this question and our hypothesis, we conducted a comprehensive investigation in **[G1, G2]** of the global response, highlighting how GSM approximation performance and intrinsic correlations in the dataset affect $\\mu$POCA performance.\n\n**Actions taken:** Thank you for this suggestion, which has improved our analysis of $\\mu$POCA performance in different settings. We have introduced a dedicated section in the appendix for **[G1, G2]**.\n\n---\n## [P4] Additional baselines\n\nThank you for highlighting this concern. We direct you to Fig 13 (App I) where we have included a subset of uncertainty-based AL methods. These results feature EPIG (from the information family in Table 1) and other uncertainty-based metrics (marginal entropy, mean-std) **[C2]**.\n\n**Actions taken:** We have made the links to these additional results more prominent in the main paper.\n\n---\n\n*We hope that most of the reviewer’s concerns have been addressed. We’d be happy to engage in further discussions.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JQ2fheJ0Gh", "reviewer": "Reviewer_ci71", "summary": "This paper  addresses the challenge of efficiently gathering features and labels in partially observed settings to enhance model generalization while considering data acquisition costs. It introduces POCA and its Bayesian instantiation, leveraging Generative Surrogate Models (GSMs) to impute missing features and compute uncertainty-based active learning metrics. The paper demonstrates the practical usage of µPOCA through empirical validation across different datasets.", "questions": "In Equation 1, it appears that the annotation strategy does not require that data samples' labels be annotated. If a sample only reveals a few features, how does such annotation benefit supervised learning? Is the primary benefit for the generative model? If so, could you provide a clearer explanation of this benefit?", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.The paper effectively formalizes the problem of active learning for partially observed data, presenting a practical approach that leverages GSMs for imputing missing features. This makes the method applicable to real-world scenarios where data acquisition is costly and incomplete.\n2.Utilizing LLMs as GSMs to impute missing features is a novel approach. This leverages the generative capabilities of LLMs to enhance the estimation of uncertainty reduction metrics, which is crucial for the proposed method's success.", "weaknesses": "1.While the practical approach is well-formalized, the theoretical contributions and novelty are somewhat limited. The problem addressed by POCA has similarities with experimental design, and the extension of existing active learning methods to partially observed data is a relatively straightforward adaptation.\n2.The paper does not adequately address how partial observability of features impacts the overall performance and reliability of the predictive model. There is a need for a deeper exploration of the theoretical implications and limitations of using GSMs for feature imputation.\n3.In Equation 1, it does not seem to require that the data samples' labels must be annotated. If a sample only reveals a few features, how does such annotation help supervised learning? The only benefit I can think of is for the generative model, but the authors did not explain this clearly.\n4.The description of the predictive model in Equation 3 is inaccurate. According to the authors' definition and the footnote on page 3, bold bold x represents fully observed features, but the predictive model should generally accept partially observed data.\n5.The meaning of p_{phi} is unclear to me. In line 92, the authors mention that phi is a model we can employ. Does this mean that p_{phi}() represents a specific density function (e.g., Gaussian)? If so, how can we ensure that both the likelihood and posterior follow the same distribution, whose density is denoted by p_{phi}? The authors did not impose any conjugate restrictions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We are grateful to the reviewer for their insightful feedback that has improved the paper.*\n\n---\n## [P1] Emphasizing novelty\nThank you for highlighting this concern. Allow us to further clarify our novelty and theoretical contributions for partially observable data acquisition.\n\n**Theoretical analysis.** We start by focusing on $\\mu$POCA, which we theoretically analyze in Proposition 1, applying to a family of Bayesian methods. Although the theorem may initially seem intuitive, it crucially implies that the new acquisition term requires estimating the *distribution* of the unobserved features. This requirement led us to develop *GSMs* to estimate this distribution. In other words, this theoretical rationale, which begins by analyzing uncertainty reduction in a partially observable setting, highlights the necessity of GSMs in this context, representing a novel contribution.\n\n**Methodological novelty.** To demonstrate the methodological novelty of $\\mu$POCA, we compare it with **Vanilla Active Learning (VAL)** and **Active Learning with Imputation (ALI)** using EIG as an illustrative example:\n* **PO-EIG.** The acquisition metric is represented by $I(\\Omega, Y|x_{\\boldsymbol{o}}, \\mathcal{D}, X_j)$, which measures predictive uncertainty considering the variability of $X_j$. This helps identify which values of $X_j$ are worth acquiring, as its *variability* allows measuring the impact of uncertainty reduction on predictive performance. `Fig 3a` in the attached PDF illustrates this, showing that exploring the unobserved regions $X_2$ and $X_3$ helps understand areas of uncertainty in the predictive model. This capability enables us to measure *what features are relevant* and in *what regions*.\n* **VAL**, **ALI**. We use $\\tilde{x}$ to denote an imputation, the acquisition metrics for VAL and ALI are then represented by $I(\\Omega, Y|x_{\\boldsymbol{o}}, \\mathcal{D})$ and $I(\\Omega, Y|x_{\\boldsymbol{o}}, \\mathcal{D},\\bar{x})$, respectively. *These metrics cannot determine which features to acquire*: VAL does not consider unobserved features and their impact on predictive performance; ALI provides only a point estimate in the unobserved region (as shown in `Fig 3a` from the PDF), which is insufficient for understanding uncertainty in the unobserved space. \n\n**Actions taken:** This discussion and the illustrative example in `Fig 3` have been added to App B.\n\n---\n## [P2] Effect of partial observability on performance\nWe appreciate your feedback and agree that partial observability (PO) is a critical factor that can influence the ability of GSMs to accurately approximate the underlying data distribution. This approximation becomes vital as it directly impacts both predictive accuracy and acquisition performance in downstream tasks. \n\n**Actions taken:** We hypothesize that PO can affect acquisition performance when the GSM does not accurately learn the underlying data distribution, which is affected by (1) the approximation power of the GSM, and (2) the intrinsic properties of the partial observability. To investigate our hypothesis, we conducted a comprehensive investigation in **[G1, G2]** of the global response, highlighting that GSM approximation performance and intrinsic correlations in the dataset affect $\\mu$POCA performance.\n\n---\n## [P3] Acquisition of only features in POCA\nThank you for raising this point. As you correctly identified, Eq 1 describes that acquiring labels or features can improve generalization. We intended for this formalism to be as general as possible, as our work is the first to formalize the POCA problem.\n\nWe clarify that in a purely *supervised learning* setting, acquiring only features does not improve generalization performance. However, this does not preclude this application for other learning problems. Two prominent examples are *semi-* and *self-supervised* learning, where models use additional unlabeled data to enhance generalization (by learning underlying structures).\n\n**Actions taken:** We have clarified our statement in L100 and explicitly referenced applications of Eq 1 in semi and self-supervised learning.\n\n---\n## [P4] Clarifying Eq 3\nThank you for spotting this. We agree with the reviewer that in Eq 3, the input of the predictive model can be partially observed data. \n\n**Actions taken.** We have updated Eq 3 to describe the predictive model as $p_{\\phi}(y|\\mathbf{x}')$, where $\\mathbf{x}'$ represents partially observed data. We further clarify that while models that accept variable-length inputs (eg Transformer, random forests) naturally handle this, models that expect fixed-size inputs would require imputation.\n\n---\n## [P5] Clarifying $\\phi$\nEffectively, $p_{\\phi}(y|\\boldsymbol{x})$ represents a likelihood over $Y$ when conditioned on the input $\\boldsymbol{x}$. $\\phi$ represents the model, which is specified by the functional form, the parameters $\\omega$, and its distributions. For example, in a Bayesian neural network:\n\n$$p_{\\phi}(y|\\mathbf{x})=\\int_{\\Omega}p(\\omega|\\mathcal{D})p_{\\phi}(y|\\mathbf{x}, \\omega) \\, d\\omega$$\n\nWhere $p(\\omega|\\mathcal{D})$ is the posterior distribution over weights, $p(\\omega|\\mathcal{D}) \\propto p(\\mathcal{D}|\\omega)p(\\omega)$. This model is specified with an architecture and priors over weights, $p(\\omega)$. As such, $p_\\phi(y|\\mathbf{x})$ is the posterior predictive distribution marginalized over $\\omega$. \n\nWe note that the posterior $p(\\omega|\\mathcal{D})$ does not necessarily have the same distribution as the likelihood $p_\\phi(y|\\mathbf{x})$ (eg in BNNs, $p(\\omega|\\mathcal{D})$ are typically Gaussian, but the likelihood function could be categorical for classification). As the reviewer has pointed out, obtaining an analytical expression might require the use of conjugate distributions. However, in many practical cases, this is not necessary (eg by using variational inference).\n\n**Actions taken:** We have clarified the meaning of $\\phi$ in Eq 3 using a paraphrased version of this discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PT0HIA1PbS", "reviewer": "Reviewer_CwS1", "summary": "This paper introduces a novel problem setting in Active Learning (AL) called Partially Observable Cost-Aware Active-Learning (POCA). POCA deals with situations where acquiring both features and labels comes with a cost and where datasets are partially observed, meaning some features might be missing for certain data points. The authors propose µPOCA, an instantiation of POCA that uses Generative Surrogate Models (GSMs), specifically Large Language Models (LLMs), to impute missing features and estimate the uncertainty reduction from acquiring both features and labels. This uncertainty reduction then guides the acquisition process, aiming to maximize the model's generalization capability within a given budget. The authors theoretically show that acquiring both labels and features leads to greater uncertainty reduction compared to acquiring only labels (the traditional AL approach) and provide empirical validation of µPOCA's effectiveness on various synthetic tabular datasets.", "questions": "- Have you considered evaluating µPOCA on real-world datasets with varying degrees and patterns of missingness? This would provide a more comprehensive understanding of its practical applicability and limitations.\n- Are there specific situations where µPOCA significantly outperforms traditional AL, even when the GSM imputation accuracy is not perfect?", "limitations": "The authors briefly address the limitations of µPOCA.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Originality: The paper tackles a practical problem in AL that has been largely overlooked - the case of partially observed data with associated feature and label acquisition costs. Formalizing POCA problem setting is on its own a valuable contribution.\n- Significance: The paper provides theoretical justification for µPOCA, demonstrating its potential to outperform traditional AL methods in specific scenarios. The empirical results support the effectiveness of the proposed approach.\n- Addressing imputation vs. acquisition: The authors address the important question of whether imputation can replace data acquisition, showing that while LLM imputation is beneficial, it doesn't reach the performance of acquiring all features (Figure 7).", "weaknesses": "- Clarity and flow: The paper's clarity and flow could be improved. For example, the use of GSMs for both training and metric estimation is not immediately clear and could be better explained upfront. The notation is sometimes confusing, for example, the use of 'j' to represent both a single feature index and a set of feature indices, as in 'ci,j' and 'xj', and J+1 being used for the label can be easily missed. Additionally, some crucial concepts, like the need for a stochastic GSM (P1 in Appendix F), are introduced late in the paper, making it difficult for the reader to grasp their importance. Finally, the difference between Scenario 1 and 2 could be better explained as it's not entirely clear what the different implications of the two are and how the unnatural pattern of Scenario 1 impacts the performance of µPOCA.\n- Potentially strong assumption of independence: The theoretical justification seems to rely on the assumption of independence between generalization capability (G) and unobserved features (Xj) given observed data (•).  Intuitively it seems like this assumption would not hold in many real-world scenarios. The authors should discuss the practicality of the assumption, the implications of violating this assumption, and potential mitigation strategies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Thank you for your feedback. We have addressed your individual points in this response and in the revised manuscript*\n\n---\n## [P1.1] GSM for metric estimation and training\n---\nThank you for your suggestion; we acknowledge that this important detail should be clear.\n\n**Update.** In response, before line 165 (after the new inclusion of [P1.3]), we added: *\"For models with fixed-size input, like Random Forest, GSMs can be used to impute missing features (see Section [3.2] for details).\"*\n\n\n## [P1.2] Clarity on indexes\n---\n**Index omissions** Bold variables are defined as a set of variables in line 94, which we agree can be easily overlooked. To clarify, we have now highlighted \"**Bold variables**\" for emphasis. \n\nAdditionally, we corrected line 149 by bolding j, and we found no other omissions. The omission of $i$ is noted in line 96; to make this clearer, we state \".... is omitted when unnecessary, i.e., $x_{i,j}\\equiv x_{j}$\".\n\n**Index J+1** is an important detail. We thank the reviewer for spotting it. Consequently, for improved clarity, now line 99 states: \"...indexed by $\\boldsymbol{j}$, for instance, $i$, with index $j = J+1$ indicating the label acquisition\".\n\n## [P1.3] Stochastic GSMs\n---\nAs the reviewer correctly mentions, the stochasticity of the GSM is a key in uPOCA. To be more direct about this, we made the following updates:\n\n- **Update 1**, after the presenting the functionality of GSMs, we include in line 165 the following: *\"**Why generative imputation can help Active Learning?** Note, that various (unobserved) features can be compatible with the observed features $x_{\\boldsymbol{o}}$, leading to potentially different outcomes. In this case, a traditional AL metric would select this instance because $x_{\\boldsymbol{o}}$ alone is insufficient to predict $Y$. However, generative imputation incorporates information about unobserved features into the predictive model, allowing it to select instances that remain uncertain even after this information is considered. Additionally, generative imputation can identify features that are not worth acquiring, as changes to these features do not impact $Y$.*\n\n- **Update 2**, We include the canonical example (see attached PDF) complementing this explanation in Appendix K.\n\n## [P2] Independence assumption \n---\nThank you for highlighting this concern, which has helped us to be more explicit in illustrating why this is **not a strong assumption**. \n\nLines L176-181 state that the assumption is valid for the supervised learning models we consider. Appendix B explains this further, and Appendix H provides empirical results confirming the validity of this assumption through Eq. (7). We made two modifications to clarify this:\n\n- **Update 1.** Now sentence L176 reads: *\"Note that the independence assumption of Proposition 1 **is not a strong assumption and is valid** for the supervised models we consider.\"*. Also, in L181, we corrected the link to Appendix H, which was previously incorrect.\n- **Update 2.** Now sentence L180 reads: \"... detailed explanation of this independence assumption's validity. In the same Appendix, we include a more informal but intuitive and practical explanation\". The intuitive and practical explanation  is the following:\n***Intuition.** Let the RV of observed and unobserved features be $X_1$ and $X_2$, respectively. Consider $\\mathcal{G} = \\Omega$, representing the EIG case, where the model parameter is related to generalization. Our downstream model is a Bayesian neural network (BNN) that samples $w_1$ half the time and $w_2$ the other half. This BNN is trained on dataset $\\mathcal{D}$, using samples from $[X_1, X_2]$ as input. Given an instance with observation $x_1$, we use GSM to sample $x^\\*_2$. The prediction $y$ is obtained by sampling $w^\\*$ from the BNN and $x^\\*_2$ from GSM, resulting in $y^\\* = w^\\*([x_1, x^\\*_2])$. It is clear to notice that knowing $w^\\*$ gives **no information** about the possible values of $x^\\*_2$; this is because how we sample $w^\\*$ depends on $\\mathcal{D}$ and how the model is trained. This is the **independence assumption**. It is only possible to know something about $x^\\*_2$ when we know $y^\\*$, thanks to $y^\\* = w^\\*([x_1, x^\\*_2])$. In the **observation**, we refer to this as the immorality.*\n\n## [Q1] Study on different missingness patterns ##\n---\n\nWe believe that researching pattern missingness and its impact on GSM and acquisition performance is an important area of study that deserves a comprehensive study on its own. That said, we agree that a better understanding of how partial observability affects the acquisition process is necessary. In our study, we already consider real-world datasets that were engineered to construct partial observability. However, to be able to fully control the level of correlation of unobserved and observed feature, we extend this study using synthetic datasets in **[G2]**.\n\n## [Q2] when $\\mu$POCA outperform traditional AL under limited GSMs\n---\n\nActive Learning's main limitation is that its acquisition metrics do not consider uncertainty reduction in feature acquisition. Our work addresses this by using GSMs to estimate the distribution of unobserved features and samples from this distribution to estimate the uncertainty reduction, though, as noted by reviewers, the GSM can be limited in some applications. uPOCA-based methods offer the advantage of approximating unobserved features, and even if imperfect, it can help identify non-useful features. To determine which unobserved features are relevant, we only need to observe some effect in the outcome when varying these features. In contrast, irrelevant features will have minimal impact on the outcome despite variability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QqAQGkzXrS", "reviewer": "Reviewer_A8wS", "summary": "This paper explores the problem of active learning where we can choose not just which data point to annotate, but also which features to obtain, when given an unlabeled pool of data points for which only some features are known. The paper proposes using generative models to sample the missing features and apply bayesian active learning techniques to estimate the expected information gain from annotating the data points.", "questions": "- What is the motivation for having the label cost be separate in problem (1)? If this problem is not really studied and if you are focusing only on (2) anyways where the label cost is baked in, then this seems unnecessary generalization.\n- If you have only some features for some data, what is the training process that you use?\n- How does the quality of the generative model affect the downstream performance of POCA? Have you tried different models, or at the minimum, injected artificial noise to estimates?\n- The numerical analysis primarily focuses on comparisons against BALD. While this makes sense given that the method is an extension of BALD, it would be interesting to see how other benchmark active learning techniques fare on this problem. Specifically, could it be that other uncertainty or diversity-based active learning methods, beyond Bayesian ones, also perform well for example if the number of missing features is small?", "limitations": "Limitations are discussed in the paper", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is overall strong, addresses an interesting problem and proposes a simple and intuitive methodology that extends standard techniques. The implications of the theory is discussed and validated with empirical analysis.", "weaknesses": "There are some points of clarification in writing and the numerical analysis is missing some potential interesting ablations and comparisons with different baselines. See questions below", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We are grateful to the reviewer for their insightful feedback and constructive comments that have improved the paper.*\n\n---\n## [P1] Inclusion of label cost in Equation 1\n\nThank you for your concerns. We appreciate the opportunity to clarify the inclusion of label cost in Equation 1.\n\nIn Equation 1, we include label costs because this expression allows for the acquisition of a subset of features without the label. This separation of label cost is intentional. \n\nNote that one of our primary goals is to formalize the POCA problem, and for this, we opted for the most generalized problem definition. Notably, the POCA problem, i.e., acquisition of a subset of features and/or labels, along with its associated cost, is not explicitly addressed in the existing experimental design literature **[C3]**, which is a contribution in itself as remarked by other reviewers (**CwS1**).\n\nThe utility function in Equation 1 measures the level of generalization based on acquisition. Semi-supervised and self-supervised learning are examples of how generalization can be achieved using only unlabeled data.\n\n---\n## [P2] Training when some features are available\n\nThank you for highlighting this concern. We can interpret this issue in two ways:\n\n- **Training a downstream model with only a subset of features.** In the context of $\\mu$POCA, we work with predictive models that require fixed-size inputs. However, not all features may be observed due to the acquisition of a subset of features. To handle this situation, we employ a GSM to impute any missing features. This approach ensures that the downstream model receives the fixed-size inputs it requires. This imputation is indicated in L208.\n\n- **Training a GSM with only a subset of features.** This is a common case when the GSM is trained on the pool set (results in Figure 11). Note, that our training of the GSM for any available unlabeled data. As described in L246, the process involves generating random masks to create an input $m \\odot \\boldsymbol{x}_o$, and an output, $(1 - m) \\odot \\boldsymbol{x}_o$. However, we ensure the input always has at least two features and the output at least one feature. Therefore, we only consider training samples with at least three observed features. This detail is now included in Appendix E.2.\n\nAs illustrated in the $\\mu$POCA Algorithm (see attached PDF), the predictive model is trained after each data acquisition. In contrast, the GSM is trained with the available unlabeled data before the acquisition process begins.\n\n---\n## [P3] GSM quality\n\nWe wish to clarify the impact of GSM quality on downstream performance. The ability of GSMs to accurately approximate the underlying data distribution is crucial for effective generative feature imputation, which in turn affects both predictive accuracy and acquisition performance in downstream tasks. In **[G1, G2]**, we identify the specific conditions under which GSMs can successfully approximate the data distribution. We will flag this more clearly in the revised manuscript.\n\n---\n## [P4] Additional baselines\n\nThank you for highlighting this concern. We direct you to Fig 13 (App I) where we have included a subset of uncertainty-based AL methods. These results feature EPIG (from the information family in Table 1) and other uncertainty-based metrics (marginal entropy, mean-std) **[C2]**.\n\n**Actions taken:** We have made the links to these additional results more prominent in the main paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
---

*We are grateful to the reviewers for their insightful feedback.*

We are happy to hear that reviewers found our work as a "novel task, optimizing the acquisition of data (features and labels) in scenarios where information is partially observed and the cost of acquiring additional data" (**tNwd**), which is noted as "a practical problem in AL that has been largely overlooked" (**CwS1**). Furthermore, we are also grateful for the acknowledgment of the formalization of POCA (**ci71**), noted as "on its own a valuable contribution" (**CwS1**).

We appreciate the recognition of the innovative combination of GSMs (particularly LLMs) to impute missing features, which was highlighted as a novel aspect (**ci71, tNwd**), and provides "theoretical justification for $\mu$POCA, demonstrating its potential to outperform traditional AL methods" (**CwS1**). The reviewers also highlighted that our work "addresses an interesting problem and proposes a simple and intuitive methodology" and is supported by "empirical analysis" (**A8wS**).

We have also taken the reviewers’ feedback into account and made the following key changes to improve the paper:
* **[G1] GSM impact on acquisition performance:** Demonstrating how underlying GSM performance affects acquisition results (see `Fig 1` in attached PDF), observing that GSM quality has a notable impact on acquisition performance.
* **[G2] Performance across varying data characteristics.** Analyzing $\mu$POCA's performance using synthetic datasets with varying correlations (see `Fig 2`), confirming our hypothesis that in settings with increased correlations between observed and unobserved features, the performance gains of POCA are more notable.

We hope these updates address the reviewers' concerns. We welcome any further feedback.

With thanks,

The Authors of #15686

---
## Overview
The performance of $\mu$POCA in partially observed settings fundamentally depends on how well the GSM approximates the distribution of the unobserved features. There are two key factors that affect the quality of this estimation:

1. **GSM's approximation power:** referring to the model's capacity to accurately model the unobserved features.
2. **Intrinsic characteristics of the dataset:** referring to inherent correlations between observed and unobserved features. Indeed, lower correlations is more challenging.

In what follows, we investigate each factor in turn, **[G1]** studying the impact of different GSMs and **[G2]** investigating different dataset characteristics. This approach aims to delineate the conditions under which $\mu$POCA is expected to excel.

---
## [G1] GSM impact on acquisition performance
**Setup.** We evaluated various LLMs (including `Mistral-7B-Instruct-v0.3`, `Phi-3`, and `Llama-3`) as GSMs to assess how model quality affects acquisition performance.

**Analysis.** `Fig 1` demonstrates that GSM quality significantly influences acquisition results. Notably, Mistral-7B consistently outperforms the alternative GSMs, with one exception in the housing dataset. Interestingly, Llama-3 performs well on this benchmark, highlighting this inter-model variability. The Cardio dataset further highlights these differences, with Mistral-7B performing significantly better than the other models.

**Takeaway.** These findings underscore the critical role of GSM quality in acquisition performance.

---
## [G2] Performance across varying data characteristics

**Setup.** Next, we turn our attention to investigating how the the data distribution affect acquisition performance. We are particularly interested in analyzing the effect of the correlation between unobserved features $X_{unobs}$ and observed features $X_{obs}$ on acquisition performance.

To demonstrate this, we examine a scenario where (1) $X_{unobs}$ correlate with the outcome, while (2) observed features do not. In this context, the GSM becomes crucial for downstream performance, as $X_{obs}$ alone provide insufficient information to predict outcomes accurately. As such, the GSM must effectively model the relationship between $X_{obs}$ and $X_{unobs}$ to acquire missing features critical for predicting the outcome.

We model both  $X_{obs}$ and $X_{unobs}$ as two-dimensional random Gaussian variables centered at zero and establish a specific **controllable** correlation between them through $\rho$:

$$\Sigma = [I\_2  ~~~~~~ \rho\_{X\_{obs}X\_{unobs}} I\_2 $$

$$~~~~~~ \rho\_{X\_{obs}X\_{unobs}}I\_2~~~~~~~~  I\_2 ] $$

The label $Y$ is then constructed to be independent of $X_{obs}$ using the orthogonalization:

$$X_{\text{orthogonal}} = X_{unobs} - X_{obs} (X_{obs}^T X_{obs})^{-1} X_{obs}^T X_{unobs}$$

which we use to construct the label using $\text{logits} = \frac{1}{1 + e^{-\sum B_{\text{orthogonal}}}}, \quad C = \mathbf{1}_{\text{logits} > \text{0}}$

**Analysis.** `Fig 2a` illustrates how varying $\rho$ between  $X_{obs}$ and $X_{unobs}$ *empirically* affects variable correlation, c, validating our synthetic experiment design. `Fig 2b` analyzes EIG (traditional active learning without GSM) and PO-EIG with varying $\rho$. We note that when the correlation between $X_{obs}$ and $X_{unobs}$ is low, GSMs provide no performance benefits. However, as correlation increases, the performance gains of PO-EIG over EIG expand significantly, confirming our hypothesis.
 
---
### References

**[C1]** Borisov, V., Leemann, T., Seßler, K., Haug, J., Pawelczyk, M., & Kasneci, G. (2021). Deep Neural Networks and Tabular Data: A Survey. IEEE Transactions on Neural Networks and Learning Systems, 35, 7499-7519.

**[C2]** Tharwat, A., & Schenck, W. (2023). A survey on active learning: State-of-the-art, practical challenges, and research directions. Mathematics, 11(4), Article 820.

**[C3]** Tom Rainforth. Adam Foster. Desi R. Ivanova. Freddie Bickford Smith. "Modern Bayesian Experimental Design." Statist. Sci. 39 (1) 100 - 114, February 2024. <!--https://doi.org/10.1214/23-STS915 -->

Meta-review: The paper introduces 𝜇POCA, a novel framework for optimizing data acquisition in scenarios with partial observability and high acquisition costs. 𝜇POCA, the proposed instantiation, leverages Generative Surrogate Models (GSMs), particularly Large Language Models (LLMs), to impute missing features and maximize uncertainty reduction during acquisition. The reviewers agree that the problem addressed is timely and that integrating GSMs for feature imputation is a novel contribution with practical relevance.

However, the reviewers also highlight several areas for improvement. The reliance on Monte-Carlo sampling introduces significant computational costs, and the paper's clarity could be enhanced. Additionally, the empirical validation is primarily focused on tabular data (in the original submission), limiting the generalizability of the findings. The authors have provided a rebuttal addressing most of these concerns. If accepted, the authors are encouraged to refine the theoretical analysis and broaden the scope of their experiments.
Decision: Accept (poster)