Active Feature Acquisition with Generative Surrogate Models
Authors: Yang Li
Junier Oliva
Venue:  (2021)
Keywords: Reinforcement Learning
Active Feature Acquisition
Feature Selection

Abstract: Many real-world situations allow for the acquisition of additional relevant information when making an assessment with limited or uncertain data. However, traditional ML approaches either require all features to be acquired beforehand or regard part of them as missing data that cannot be acquired. In this work, we propose models that perform active feature acquisition (AFA) to improve the prediction assessments at evaluation time. We formulate the AFA problem as a Markov decision process (MDP) and resolve it using reinforcement learning (RL). The AFA problem yields sparse rewards and contains a high-dimensional complicated action space. Thus, we propose learning a generative surrogate model that captures the complicated dependencies among input features to assess potential information gain from acquisitions. We also leverage the generative surrogate model to provide intermediate rewards and auxiliary information to the agent. Furthermore, we extend AFA in a task we coin active instance recognition (AIR) for the unsupervised case where the target variables are the unobserved features themselves and the goal is to collect information for a particular instance in a cost-efficient way. Empirical results demonstrate that our approach achieves considerably better performance than previous state of the art methods on both supervised and unsupervised tasks.

Under review as a conference paper at ICLR 2021
ACTIVE FEATURE ACQUISITION WITH GENERATIVE SURROGATE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Many real-world situations allow for the acquisition of additional relevant infor- mation when making an assessment with limited or uncertain data. However, traditional ML approaches either require all features to be acquired beforehand or regard part of them as missing data that cannot be acquired. In this work, we pro- pose models that perform active feature acquisition (AFA) to improve the predic- tion assessments evaluation time. We formulate the AFA problem as a Markov decision process (MDP) and resolve it using reinforcement learning (RL). The AFA problem yields sparse rewards and contains a high-dimensional complicated action space. Thus, we propose learning generative surrogate model that cap- tures the complicated dependencies among input features to assess potential infor- mation gain from acquisitions. We also leverage the generative surrogate model to provide intermediate rewards and auxiliary information to the agent. Furthermore, we extend AFA in task we coin active instance recognition (AIR) for the unsu- pervised case where the target variables are the unobserved features themselves and the goal is collect information for particular instance in cost-efficient way. Empirical results demonstrate that our approach achieves considerably bet- ter performance than previous state of the art methods on both supervised and unsupervised tasks.
INTRODUCTION
A typical machine learning paradigm for discriminative tasks is learn the distribution of an out- put, given a complete set of features, æ E RR: p(y z). Although this paradigm is successful in a multitude of domains, it incongruous with the expectations of many real-world intelligent systems in two key ways: first, it assumes that a complete set of features has been observed; second, as a consequence, it also assumes that no additional information (features) of an instance may be obtained at evaluation time. These assumptions often do not hold; human agents routinely reason over instances with incomplete data and decide when and what additional information to obtain. For example, consider doctor diagnosing patient. The doctor usually has not observed all possible measurements (such as blood samples, x-rays, etc.) for the patient. He/she is not forced to make diagnosis based on the observed measurements; instead, he/she may dynamically decide to take more measurements to help determine the diagnosis. Of course, the next measurement to make (fea- ture observe), if any, will depend on the values of the already observed features; thus, the doctor may determine a different set of features to observe from patient to patient (instance to instance) depending on the values of the features that were observed. Hence, not each patient will have the same subset of features selected (as would be the case with typical feature selection). Furthermore, acquiring features typically involves some cost (in time, money and risk), and intelligent systems are expected to automatically balance the cost and the return improvement of the task performance. In order to more closely match the needs of many real-world applications, we propose an active feature acquisition (AFA) model that not only makes predictions with incompletc/missing features, but also determines what next feature would the most valuable to obtain for particular instance.
In this work, we formulate the active feature acquisition problem as Markov decision process (MDP), where the state is the set of currently observed features and the action is the next feature to acquire We also introduce special action indicate whether to stop the acquisition process and make a final prediction. Reinforcement learning then utilized to optimize the MDP, and the agent learns policy for selecting which next feature acquire based the current state.

---

Under review as a conference paper at ICLR 2021
step 1
step 5
step 10
stop 15
stop 20
step 1
step
step 10
stop 15
step 20
5
7
1.0
gos
averaged samples
0.0g
5
9.0 5
Figure 1: Active feature acquisition on MNIST. Example of the acquisition process and the corresponding prediction probabilities.
Active instance recognition on MNIST. Example of the acquisition process and the averaged inpaintings.
Figure 2:
After acquiring its value and paying the acquisition cost, the newly acquired feature is added to the observed subset and the agent proceeds to the next acquisition step. Once the agent decides to terminate the acquisition, it makes a final prediction based on the features acquired thus far. For example, in image classification task (Fig. 1), the agent would dynamically acquire pixels until it is certain of the image class. The goal of the agent is to maximize the prediction performance while minimizing the acquisition cost.
In the aforementioned MDP, the agent pays the acquisition cost at each acquisition step but only re- ceives a reward about the prediction after completing the acquisition process. To reduce the sparsity of the rewards and simplify the credit assignment problem for potentially long episodes (Minsky, 1961; Sutton, 1988), we leverage a surrogate model to provide intermediate rewards. The surrogate model captures the arbitrary conditional distribution p(y, Ro), where is the target variable and u, o C .l.....d] d} are arbitrary subsets of all d-dimensional features. Note that the surrogate model must be able to capture arbitrary conditionals (for subsets o) since the acquired features will vary from instance to instance. We propose using the surrogate model to calculate intermediate rewards by assessing the information gain of the newly acquired feature, which quantifies how much our confidence about the prediction improves by acquiring this feature.
In addition to producing intermediate rewards, also propose using the surrogate model to pro- vide side information that assists the agent. First, in order to inform the agent of the current infor- mation held in observed features, we pass uncertainty on the target through p(y xo). Second, to inform the agent about potential values for unobserved features, we pass imputed values by sam- pling S. p( To). Lastly, to inform the agent about the expected utility of acquisitions, we pass an estimate of the expected information gain of acquisitions i for the target variable, i.e., H xo) I.. (y 24, We note that the expected information gain can be used to directly build greedy policy, where the next feature to acquire is the one maximizes the expected information gain (Ma et al., 2018; Gong et al., 2019). contrast, our agent learns a non-greedy policy to maximize the long-term returns and use the greedy approach as a 'prior' policy to guide our agent.
In summary, our agent actively acquires new feature and pays the acquisition cost until it decides to terminate the acquisition process and make a final prediction. Meanwhile, the surrogate model calculates the information gain of the acquired feature as intermediate reward and provides side information to assist the agent in assessing its current uncertainty and help it 'look ahead' to expected outcomes from future acquisitions. When the acquisition process is completed, the environment provides a final reward based on the agent's prediction. Note that the environment does have access the ground-truth target y to evaluate the reward, but cannot reveal the agent. Equipped with the surrogate model, our method, denoted as GSMRL, essentially combines model-free and model- based RL into a holistic framework.
Above we discussed AFA for supervised tasks, where the goal is to acquire new features to predict a target variable y. some cases, however, there may not be a single target variable, but instead the target of interest may be the remaining unobserved features themselves. That is, rather than reduce the uncertainty with respect to some desired output response (that cannot be directly queried and must be predicted), we now propose active instance recognition (AIR), where the task is to query as few features as possible that allows the agent to correctly uncover the remaining unobserved fea- tures. For example, in image data AIR, an agent queries new pixels until it can reliably uncover the remaining pixels (see Fig. 2). AIR is especially relevant survey tasks, which are broadly applica- ble across various domains and applications. Most surveys aim to discover broad set of underlying characteristics of instances (c.g.. citizens in a census) using a limited number of queries (questions in the census form), which is the core of AIR. Policies for AIR would build personalized subset
2

---

(I) where L((xx), y) represents the loss function between the prediction ((z.) and the target y. Note that the prediction is made with the acquired feature subset zo. {1...., d). Therefore the agent should be able to predict with arbitrary subset as inputs. C(o) represents the acquisition cost of Under review as a conference paper at ICLR 2021
Algorithm 1: Active Feature Acquisition with GSMRL
load pretrained surrogate model M, agent agent and prediction model fo(:);
2. instantiate an environment with data D and surrogate model M: env = Environment(DA.. MM)
Wa. o, done, reward = env.reset(; // 0, done=False, reward=0
while done do
aux M query(zt.. o): // query M for auxiliary information
// aux contains the prediction j~p(y ra) and output likelihoods,
// the imputed values & ~ ~p(x 2z) and their uncertainties,
// and estimated utilities U, for each iew (equation 4)
action agent.act(ro. o, aux); // act based on the state and auxiliary info
Zo. o, done, r= enw.step(action); // take step based on the acticn done=True, ==-L(((z)))))
// if action indicates termination:
// else: dcne=False, r=rm(s, action) aC (acation)) o = U action
reward reward + accumulate rewards
end
prediction = M predict((2,. aux) or fo(2.. o, aux): // make final prediction
of survey questions (for individual instances) that quickly uncovered the likely answers to all re- maining questions. To adapt our GSMRL framework to AIR, we the target variable y equal and modify the surrogate model accordingly.
Our contributions are as follows: 1) We propose a way of building surrogate models for AFA prob- lem that captures the state transitions with arbitrary conditional distributions. 2) We leverage the surrogate model to provide intermediate rewards as training signals and to provide auxiliary infor- mation that assists the agent. Our framework represents novel combination of model-free and model-based RL. 3) We extend the active feature acquisition problem to an unsupervised case where the target variables are the unobserved features themselves. Our RL agent can be adapted to this problem with simple modifications. 4) We achieve saate-of-the-ze performance on both supervised and unsupervised tasks. 5) We open-source standardized environment inheriting the OpenAl gym interfaces (Brockman et al., 2016) to assist future research on active feature acquisition. Code will be released upon publication.
METHODS
In this section, we first describe our GSMRL framework for both active feature acquisition (AFA) and active instance recognition (AIR) problems. We then develop our RL algorithm and the cor- responding surrogate models for different settings. We also introduce a special application that acquires features for time series data.
2.1
AFA AND AIR WITH GSMRI.
Consider discriminative task with features Rdd and target y. Instead of predicting the target by first collecting all the features, we perform sequential feature acquisition process in which we start from an empty set features and actively acquire more features. There typically a cost associated with features and the goal is to maximize the task performance while minimizing the acquisition cost, i.c.,
minimize C(j(xo), v) + aC(o),
the acquired features o. The hyperparameter a controls the trade-off between prediction loss and acquisition cost. For unsupervised tasks, the target variable y is equal to x; that is, we acquire features actively represent the instance with selected subset.
In order to solve the optimization problem in equation 1, we formulate it as Markov decision process as done in (Shim al., 2018):
|o, Ns), a E uu U r(s, a) = -L(j y) = の) - aC(a)T(a + Φ).
The state s is the current acquired feature subset o C {1,..., d} and their values T., The action space contains the remaining candidate features (1,...,d} and special action o that in-
3

---

Under review as a conference paper at ICLR 2021
dicates the termination of the acquisition process. To optimize the MDP, reinforcement learning agent acts based on the observed state and receives rewards from the environment. When the agent
acquires new feature i, the current state transits a new state following i o 4 Io U Ji, and the reward is the negative acquisition cost of this feature. Note I is obtained from the envi- romment (i.e. we observe the true &ch. feature value for the instance). When the agent terminates the acquisition and makes a prediction, the reward equals to the negative prediction loss using current acquired features. Since the prediction made at the end of the acquisition, the reward of the pre- diction is received only when the agent decide to terminate the acquisition process. This is a typical temporal credit assignment problem for RL algorithms, which could affect the learning of the agent (Minsky, 1961; Sutton, 1988). In order to remedy this issue, we propose to leverage generative sur- rogate model to provide intermediate rewards for each acquisition. The surrogate model estimates the state transitions with arbitrary conditional distributions p(y, Fu 20)) for arbitrary subsets u and o. We propose using the surrogate model to assess the intermediate reward Im for a newly acquired feature i. The intermediate rewards are inspired by the information gain to the target variable
I'm (s, Hy (o) 7 (H Too Wi),
(3)
where 7 is discount factor for the MDP. In appendix A, we show that our intermediate rewards will not change the optimal policy.
In addition intermediate rewards, we propose using the surrogate model to also provide side infor- mation to assist the agent, which includes the current prediction and output likelihood, the possible values and corresponding uncertainties of the unobserved features, and the estimated utilities of the candidate acquisitions. The current prediction and likelihood p(y xo) inform the agent about its confidence, which can help the agent determine whether to stop the acquisition. The imputed values and uncertainties of the unobserved features give the agent the ability to look ahead into and future and guide its exploration. For example, if the surrogate model very confident about the value of currently unobserved feature, then acquiring it would be redundant. The utility of feature i is estimated by its expected information gain to the target variable:
(4) where the surrogate model is used to estimate the entropies. The utility essentially quantifies the conditional mutual information [(x:: x²)) between each candidate feature and the target variable. A greedy policy can be easily built based on the utilities where the next feature to acquire is the one with maximum utility (Ma al., 2018; Gong al., 2019). Here, our agent takes the utilities as side information to help balance exploration and exploitation, and eventually learns a non-greedy policy.
U4 = (y xx) - Ep(x,I..) ( 2,, = (x; xx) - Ep(yl..) (I; y, xo).
When the agent deems that acquisition is complete, it makes a final prediction based on the acquired feaures thus far. The final prediction may be made using the surrogate model, i.c., p(y I.), but it might be beneficial to train predictions specif- ically based on the agent's own distribution of acquired fea- tures o, since the surrogate model is agnostic to the feature acquisition policy of the agent. Therefore, we optionally build
fg
prediction
state & auxiliary Info
Agent
intermediate rewards
auxliary information
a prediction model (-).) that takes both the current state xo and the side information as inputs (i.e. the same inputs as the policy). The prediction model can be trained simultancously with the policy as an auxiliary task; weight sharing between the policy and prediction function helps facilitate the learning of more meaningful representations. Now we have two pre- dictions, from the surrogate model and the prediction model respectively. The final reward -L(ů. y) during training is the maximum one using either predictions. During test time, we choose one prediction based on validation performance. An il- lustration of our framework is presented in Fig. 3. Please refer to Algorithm for the pseudo-code of the acquisition process with our GSMRI. framework, We will expound on the surrogate models for different settings below.
action
sta
Surrogate Model
Environment
Illustration of our GSMRL framework with an op- tional prediction model. Dashed arrows connect optional prediction model fa.
Figure 3:
2.1.1 SURROGATE MODEL FOR AFA
As we mentioned above, the surrogate model learns the conditional distributions p(y. Iy | xx)). Note that both x. and F. arbitrary subset of the features since the surrogate model must be able
4

---

Under review as a conference paper at ICLR 2021
to assist arbitrary policies. Thus, there are d! different conditionals that the surrogate model must estimate for a d-dimensional feature space. Therefore, learning a separate model for each different conditional is intractable. Fortunately, Ivanov et al. (2018) and Li et al. (2019) have proposed models to learn arbitrary conditional distributions p(z, (o)) They regard different conditionals as different tasks and train VAE and normalizing flow based generative models, respectively, in multi-task fashion to capture the arbitrary conditionals with a unified model. In this work, we leverage the ACFlow model (Li et al., 2019) and extend it model the target variable y as well. For continuous target variables, we concatenate them with the features, thus p(y. Tu 2o) can be directly modeled with ACFlow. For discrete target variables, we use Bayes' rule
ply, Fu xo)) p(a, y. (IFO Pv))
(5)
Ey p 11r)///)))))))) P(y')
We employ a variant of the ACFlow model that conditions on the target y to obtain the arbitrary conditional likelihoods p(a.. v.) and p(æ. y) in equation
Given a trained surrogate model, the prediction pyy xo), the information gain in equation 3, and the utilities in equation 4 can all be estimated using the arbitrary conditionals. For continuous target variables, the prediction can be estimated by drawing samples from p(y Io),, and we express their uncertainties using sample variances. We calculate the entropy terms in equation 3 with Monte Carlo estimations. The utility in equation 4 can be further simplified as
p( y xo)
p(y Ti, vo) p(u Io)
U4, = p(p.. log
ply Zo)p((; xo) = Bp(u.r, log
(6)
We then perform a Monte Carlo estimation by sampling from p(r,, y is evaluated on sampled z; rather than the exact value, since have not acquired its value yet.
Io). Note that p(y 2s 2o)
For discrete target variables, we employ Bayes' rule to make a prediction
p( P()) P(y xx) = Ey p( y P(y')
(logg v') + log P(y)))
and the uncertainty is expressed as the prediction probability. The information gain in equation 3 can be estimated analytically, since the entropy for categorical distribution is analytically available. To estimate the utility, we further simplify equation 6 to
P(y (i, Ply xa)
Ui = P(olle)))))))ll log
[ (.)) xx)),
(8)
where the KL divergence between two discrete distributions can be analytically computed. Note T; is sampled from p(z; xx)) as before. We again use Monte Carlo estimation for the expectation.
Although the utility can be estimated accurately by equation 6 and equation 8, it involves some overhead especially for long episodes, since need calculate them for each candidate feature at each acquisition step. Moreover, each Monte Carlo estimation may require multiple samples. To re- duce the computation overhead, we utilize equation 4 and estimate the entropy terms with Gaussian approximations. That is, we approximate p(r; x.)) and p(æ; y, Xa)) as Gaussian distributions and entropies reduce to a function of the variance. We use sample variance as an approximation. We found that this Gaussian entropy approximation performs comparably while being much faster.
2.1.2 SURROGATE MODEL FOR AIR
For unsupervised tasks, our goal is represent the full set of features with an actively selected subset. Since the target is also x, we modify our surrogate model to capture arbitrary conditional distributions p( xo)), which again be learned using ACFlow model. Note that by plug- ging in = x to equation 4, the utility simplifies to the entropy of unobserved features, which is essentially their uncertainties.
U, (2; Fo) Eppa)e,) (2, 21, = H(r: xo)) Ep(aa H(s, x)) H(x, x.). (9)
The last equality is due the fact that H(x; x)) 0. We again use Gaussian approximation to estimate the entropy. Therefore, the side information for AIR only contains imputed values and their variances of the unobserved features. Similar to the supervised case, we leverage the surrogate
5

---

Under review as a conference paper at ICLR 2021
model to provide the intermediate rewards. Instead of using the information gain in equation 3, use the reduction of negative log likelihood per dimension, i.e,
I'm (8, i) = loloogppps p( xo) log p( Zo: xia)
(10)
[
since equation involves estimating the entropy for potentially high dimensional distributions, which itself is an open problem (Kybic, 2007). We show in appendix A that the optimal policy invariant under this form of intermediate rewards. The final reward -L( x) is calculated as the
negative MSE of unobserved features - ((f, = - - zul12.
2.2 AFA FOR TIME SERIES
In this section, we apply our GSMRL framework on time series data. For example, consider scenario where sensors are deployed in the field with very limited power. We would like the sensors decide when to put themselves online collect data. The goal is to make few acquisitions as possible while still making an accurate prediction. In contrast to ordinary vector data, the acquired features must follow a chronological order, i.e., the newly acquired feature i must occur after all elements of o (since we may not go back in time to turn on sensors). In this case, it is detrimental to acquire a feature that occurs very late an early acquisition step, since we will lose the opportunity to observe features ahead ot. The chronological constraint in action space removes all the features behind the acquired features from the candidate set. For example, after acquiring feature t, features (1....,() () no longer considered as candidates for the next acquisition.
2.3 IMPLEMENTATION
We implement our GSMRL framework using the Proximal Policy Optimization (PPO) algorithm (Schulman et al., 2017). The policy network takes in a set of observed features and set of auxiliary information from the surrogate model, extracts a set embedding from them using the set transformer (Lee et al., 2019), and outputs the actions. The critic network that estimates the value function shares the same set embedding as the policy network. To help learn useful representations, we also use the same set embedding as inputs for the prediction model ʃe-
To reflect the fact that acquiring the same feature repeatedly redundant, we manually remove those acquired features from the candidate set. For time-series data, the acquired features must follow the chronological order since we cannot go back in time to acquire another feature, therefore need to remove all the features behind the acquired features from the candidate set. Similar spatial constraints can also be applied for spatial data. To satisfy those constraints, we manually set the probabilities of the invalid actions to zeros.
3 RELATED WORKS
Active Learning Active learning (Fu al., 2013: Konyushkova al., 2017: Yoo & Kweon, 2019) is related approach in MI. to gather more information when a learner can query oracle for the true label, y. of complete feature vector x E IRd to build a better estimator. However, our methods consider queries to the environment for the feature value corresponding to an unobserved feature dimension, i, in order to provide a better prediction on the current instance. Thus, while the active learning paradigm queries oracle during training to build classifier with complete features, our paradigm queries the environment at evaluation to obtain missing features of a current instance to help its current assessment.
Feature Selection Feature selection (Miao & Niu, 2016; Li al., 2017; Cai al., 2018), ascer- tains static subset of important features to climinate redundancies, which can help reduce com- putation and improve generalization. Feature selection methods choose a fixed subset of features C (I,..., d). and always predict y using this same subset of feature values, xg. In contrast, our model considers dynamic subset of features that is sequentially chosen and personalized on an instance- beeeeeby---itttee basis to increase useful information. It is worth noting that our method may be applied after an initial feature selection preprocessing step to reduce the search space.
Active Feature Acquisition Instead of predicting the target passively using collected features, pre- vious works have explored actively acquiring features in the cost-sensitive setting, Ling et al. (2004), Chai et al. (2004) and Nan et al. (2014) propose decision tree, naive Bayes and maximum margin based classifiers respectively to jointly minimize the misclassification cost and feature acquisition

---

Under review as a conference paper at ICLR 2021
/
2
3
4
6
MNI5Tid=256,.m100
90
Accuracy
00
70
GSHRL
JAFA
60
GSM44reedy
60.3
50
LDD!
16
0.00
Acquired
Figure 4: Example of acquired features and prediction.
Figure 5: Test accuracy
100
Gacid=10.co kw33)
95
-1k5id-12k-21
0.40
Parinnooric-
covRE
0.45
89
70
JARA
COMAL
80
6.33
0.34
0:30
JAFA
Coot-yyddd
ED01
0.35
CSM Greody
6:30
90
EDG
0:20
0.00
2.0
25
3.5 4.0 of Acquiee Fest
3.3
55 6.0 65. 75
7.5
0.0 8.5
2.5
30 35 CD 5. 6.0
of Acquired Features
Acquired Features
Acqureed Features
Figure 6: Test accuracy on UCI datasets.
Figure 7: Test RMSE on UCI datasets.
cost. Ma et al. (2018) and Gong et al. (2019) acquire features greedily using mutual information as the estimated utilityy Zubek al. (2004) formulate the AFA problem as a MDP and fit transition model using complete data, then they use the AO* heuristic search algorithm to find an optimal pol- icy. Rückstieß et al. (2011) formulate the problem as partially observable MDP and solve it using Fitted Q-Iteration. He et al. (2012) and He et al. (2016) instead employ the imitation learning ap- proach guided by a greedy reference policy. Shim et al. (2018) utilize Deep Q-Learning and jointly learn policy and classifier. The classifier is treated as environment that calculates the classi- fication loss as the reward. Similar to ours, ODIN (Zannone et al., 2019) presents an approach to combine the model-based and model-free RL. for the active feaure acquisition problem. They train a Partial VAE (Ma et al., 2018) as the dynamics model to generate synthetic data and learn a policy and a prediction model using those generated data. In our method, however, surrogate model, which estimates both the state transitions and the prediction in unified model, is utilized to provide intermediate rewards and auxiliary information.
Active Perception Active perception is relevant field in robotics where a robot with a mounted camera is planning by selecting the best next view (Bajesy, 1988; Aloimonos et al., 1988). Re- inforcement learning based active perception models (Cheng et al., 2018; Jayaraman & Grauman, 2018) have recently been proposed to select the next view. In future work, we will explore applying our method for active vision.
Model-based and Model-free RL Reinforcement learning can be roughly grouped into model- based methods and model-free methods depending on whether they use a transition model (Li, 2017). Model-based methods are more data efficient but could suffer from significant bias if the dynamics are misspecified. On the contrary, model-free methods can handle arbitrary dynamic sys- tem but typically requires substantially more data samples. There have been works that combine model-free and madel-based methods to compensate with each other. The usage of the model in- cludes generating synthetic samples to learn policy (Gu et al., 2016), back paock-ropgggaing the reward to the policy along a trajectory (Heess et al., 2015), and planning (Chebotar et al., 2017; Pong al., 2018). In this work, we rely on the model to provide intermediate rewards and side information.
EXPERIMENTS
In this section, we evaluate our method on several benchmark environments built upon the UCI repository (Dua & Graff, 2017) and MNIST dataset (LeCun, 1998). We compare our method to another RL based approach, JAFA (Shim et al., 2018), which jointly trains an agent and a classifier. We also compare to greedy policy EDDI (Ma al., 2018) that estimates the utility for each candidate feature using VAE based model and selects one feature with the highest utility at each acquisition step. As a baseline, we also acquire features greedily using our surrogate model that estimates the utility following equation 6, equation 8 and equation 9. We use fixed cost for each feature and report multiple results with different o in equation to control the trade-off between task performance and acquisition cost. We cross validate the best architecture and hyperparameters for baselines. Architectural details, hyperparameters and sensitivity analysis are provided in Appendix.
Classification We first perform classification on the MNIST dataset. We downsample the original images to 16 16 reduce the action space. Fig. 4 illustrates several examples of the acquired
7

---

Under review as a conference paper at ICLR 2021
90
95
0iptstt8..=-10)
0.20
Gasid=10:
COVRI
Weitowineldde
JAFA
170
55
65
75
3.0
3.5
6.0
G3MRL
EDCI
GGMM+
5.0
50
400
20
30
3,003 233 503,
755 004 250.504 725.00
6:14
0.12
6.30
0.00
0.06
of Acquired Toutures
F001
GoV--reedy
Acqundd Peatures
10
Figure 8: Classification on time series.
Figure 9: RMSE for unsupervised tasks.
0
3
4
6
4.0
MNISTId=258)
3.8
CSMAL
3.6-
JAFA
1 3.4
GSM-Greedy
3
4
10
15
25
35
Acquired Festures
Figure 10: Example of acquired features and inpaintings.
Figure 11: RMSE Zig
features and their prediction probability for different images. We that model acquires different subset of features for different images. Notice the checkerboard patterns of the acquired features, which indicates our model is able exploit the spatial correlation of the data. Fig. shows the acquisition process and the prediction probability along the acquisition. We can see the prediction become certain after acquiring only small subset of features. The test accuracy in Fig. 5 demonstrates the superiority of our method over other baselines. It typically achieves higher accuracy with a lower acquisition cost. It is worth noting that our surrogate model with a greedy acquisition policy outperforms EDDI. We believe the improvement is due to the better distribution modeling ability of ACFlow so that the utility and the prediction can be more accurately estimated. We also perform classification using several UCI datasets. The test accuracy is presented in Fig. 6. Again, our method outperforms baselines under the same acquisition budget.
Regression We also conduct experiments for regression tasks using several UCI datasets. We re- port the root mean squared error (RMSE) of the target variable in Fig. 7. Similar to the classification task, our model outperforms baselines with a lower acquisition cost.
Time Series To evaluate the performance with constraints in action space, we classify over time series data where the acquired features must follow chronological ordering. The datasets are from the UEA& & UCR time series classification repository (Bagnall et al., 2017). For GSMRL and JAFA, we clip the probability of invalid actions to zero; for the greedy method, we use a prior to bias the selection towards earlier time points. Please refer to appendix B.3 for details. Fig. 8 shows the accuracy with different numbers of acquired features. Our method achieves high accuracy by collecting a small subset of the features.
Unsupervised Next, we evaluate our method on unsupervised tasks where features are actively acquired to impute the unobserved features. We use negative MSE as the reward for GSMRL and JAFA. The greedy policy calculates the utility following equation 9. For low dimensional UCI datasets, our method is comparable to baselines as shown in Fig. 9; but for the high dimensional case, as shown in Fig. 11, our method is doing better. Note JAFA is worse than the greedy policy for MNIST. We found hard to train the policy and the reconstruction model jointly without the help of the surrogate model in this case. See Fig. for example of the acquisition process.
5 ABLATIONS
In this section, we conduct series of ablation studies to explore the ability of our GSMRL model.
Surrogate Models Our method relies on the surrogate model to provide intermediate re- wards and auxiliary information. To better understand the contributions each component does to the overall framework, we conduct ab- lation studies using the MNIST dataset. We gradually drop
MNISTio-256k=100
with
90
GSM
80
FOT Model
Reword
wilc m
Who wixe
65
component from the full model and report the results in Fig. 12. The Full Model' uses both intermediate rewards
16 18
24
500
1000
Figure 12: Ablations
Figure 13: Rewards
8

---

Under review as a conference paper at ICLR 2021
and auxiliary information. We then drop the intermediate rewards and denote it as "w/o rm'. The model without auxiliary information is denoted as 'w/o aux'. We further drop both components and denote it as 'w/o rm & aux'. From Fig. 12, see these two components contribute significantly to the final results. We also compare models with and without the surrogate model. For models with- out surrogate model, we train classifier jointly with the agent as in JAFA. We plot the smoothed rewards using moving window average during training in Fig. 13. We can see the agent with a surrogate model not only produces higher and smoother rewards but also converges faster.
Dynamic vs. Static Acquisition Our GSMRL acquires features fol- lowing a dynamic order where it eventually acquires different features for different instances. A dynamic acquisition policy should perform better than static one (i.e., the same set of features are acquired for each instance), since the dynamic policy allows the acquisition to be specifically adapted to the corresponding instance. To verify this is ac- tually the case, we compare the dynamic and static acquisition under a greedy policy for MNIST classification. Similar to the dynamic greedy policy, the static acquisition policy acquires the feature with maximum utility at each step, but the utility is averaged over the whole testing set, therefore the same acquisition order is adopted for the whole test- ing set. Figure 14 shows the classification accuracy for both EDDI and GSM under greedy acquisition policy. We can the dynamic policy is always better than the corresponding static one. It is also worth noting that our GSM with a static acquisition can already outperform the EDDI with dynamic one.
ECD Dyyammy ECD (Stati) GSMrDynamic)
10 15 20 25
Figure 14: Comparison between dynamic and static acquisition under greedy policies.
Greedy vs. Non-greedy Acquisition Our GSMRL will terminate the acquisition process if the agent deems the current acquisition achieves the optimal trade-off between the prediction performance and the ac- quisition cost. To evaluate how much the termination action affects the performance and to directly compare with the greedy policies under the same acquisition budget, we conduct an ablation study that removes the termination action and gives the agent a hard acquisition budget (i.e., the number of features acquires). Figure 15 shows the accuracy of MNIST classification with different acquisition budget. We can see our GSMRL consistently outperforms the greedy policy under all different budget, which verifies the benefits of using the non-greedy policy.
GSMBL GSM-Grewdy
of Accusee Features
Figure 15: Acquisition with fixed budget.
Small vs. Large Action Space play a downsampled version of MNIST in the experiment section. Here, we show that our GSMRL model can be casily scaled up to a large ac- tion space. We conduct experiments using the original MNIST of size 28 28. We observe that JAFA has difficulty in scaling to this large ac- tion space, the agent either acquires no feature or acquires all features. The greedy approaches are also hard to scale, since at each acquisition step, the greedy policy will need to compute the utilities for every un- observed features, which incurs a total (d2) complexity. In contrast, our GSMRL only has ((d) complexity. Furthermore, with the help of the surrogate model, our GSMRL is pretty stable during training and converges to the optimal policy quickly. Fig. 16 shows the accuracy with certain percent of features acquired. The task is definitely harder for large action space as can be seen from the drop in performance when the agent acquires the same percentage of features for both small and large action space, but our GSMRL still achieves high accuracy by only acquiring small portion of features.
For the sake of comparison, we em-
MN
80
70
18x15
30
Acqutre Features
Figure 16:
Acquisition with large action space.
CONCLUSION
In this work, we formulate the active feature acquisition problem as an MDP and propose to combine model-based and model-free RL into a holistic framework to resolve the problem. We leverage a generative surrogate model to capture the state transitions across arbitrary feature subsets. Our surrogate model also provides auxiliary information and intermediate rewards to assist the agent. We evaluate our framework both supervised and unsupervised AFA problems and achieve state- of-the-art performance on both tasks. In future work, we will extend our framework to actively acquire features in spatial-temporal setting, where features indexed with continuous positions.
9

---

Under review as a conference paper at ICLR 2021
REFERENCES
John Aloimonos, Isaac Weiss, and Amit Bandyopadhyay. Active vision. International journal of computer vision, 1(4):333-356, 1988.
Anthony Bagnall, Jason Lines, Aaron Bostrom, James Large, and Eamonn Keogh. The great time ries classification bake off: a review and experimental evaluation recent algorithmic advances. Data Mining and Knowledge Discovery, 31(3):606-660 2017.
Ruzena Bajesy. Active perception. Proceedings of the IEEE, 76(8):666-100 1988.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Jie Cai, Jiawei Luo, Shulin Wang, and Sheng Yang. Feature selection in machine learning: A new perspective. Neurocomputingg 300:70-79, 2018.
Xiaoyong Chai, Lin Deng, Qiang Yang, and Charles X Ling. Test-cost sensitive naive bayes classi- fication. In Fourth IEEE International Conference Data Mining (ICDM'O4 04), pp. 51-58. IEEE, 2004.
Yevgen Chebotar, Karol Hausman, Marvin Zhang, Gaurav Sukhatme, Stefan Schaal, and Sergey Levine. Combining model-based and model-free updates for trajectory-centric reinforcement learning. arXiv preprint arXiv: 1exXiv:703.000 1703.73073. 2017.
Ricson Cheng, Arpit Agarwal, and Katerina Fragkiadaki. Reinforcement learning of active vision for manipulating objects under occlusions. arXiv preprint arXiv:1811.0800 2018
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL ics. uci. eddumml
Yifan Fu, Xingquan Zhu, and Bin Li. A survey instance selection for active learning. Knowledge and information systems, 35(2):249-28) 283, 2013.
Wenbo Gong, Sebastian Tschiatschek, Sebastian Nowozin, Richard E Turner, José Miguel Hernández-LLoatt, and Cheng Zhang. Icebreaker: Flement-wise efficient information acqui- sition with bayesian deep latent gaussian model. In Advances in Neural Information Processing Systems, 14820-14831, 2019.
Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration. In International Conference on Machine Learning, pp. 2829- 2838, 2016.
He He, Jason Eisner, and Hal Daume. Imitation learning by coaching. In Advances in Neural Information Processing Systems, pp. 3149-3157, 2012.
He He, Paul Mineiro, and Nikos Karampatziakis. Active information acquisition. arXiv preprint arXiv:1602.02111/ 2016.
Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa. Learn ing continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems, pp. 2944 2952, 2015.
Oleg Ivanov, Michael Figurnov, and Dmitry Vetrov. Variational autoencoder with arbitrary condi- tioning. arXiv preprint arXiv: 02382, 2018.
Dinesh Jayaraman and Kristen Grauman. Learning to look around: Intelligently exploring unseen environments for unknown tasks. Proceedings of the IEEE Conference on Computer Vision and Pateem Recognition, pp.
2018.
Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua. Learning active learning from data. In Advances in Neural Information Processing Systems, pp. 4225 -2355, 2017.
10

---

Under review as a conference paper at ICLR 2021
Jan Kybic. High-dimensional entropy estimation for finite accuracy data: R-nn entropy estimator, In Biennial International Conference on Information Processing Medical Imaging, pp. 569-580. Springer, 2007.
Yann LeCun. The mnist database of handwritten digits.
1998.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: framework for attention-based permutation-invarre neural networks. In Interna- lional Conference on Machine Learning, pp. 3744-3753. PMLR, 2019.
Jundong Li, Kewei Cheng, Suhang Wang, Fred Morstatter, Robert P Trevino, Jiliang Tang, and Huan Liu. Feature selection: A data perspective. ACM Computing Surveys (CSUR), 50(6):1-45, 2017.
Yang Li, Shoaib Akbar, and Junier B Oliva. Acflow: Flow models for arbitrary conditional likeli-
hoods. arXiv preprint 1rXiv:190.0063/1 2019.
Yuxi Li. Deep reinforcement learning: An overview. arXiv preprint a:Xiv:1701.07277, 2017.
Charles X Ling, Qiang Yang, Jianning Wang, and Shichao Zhang. Decision trees with minimal costs. In Proceedings of the twenty-first international conference on Machine learning, pp. 69, 2004.
Chao Ma. Sebastian Tschiatschck, Konstantina Palla, José Miguel Hernández-Lobato. Sebastian Nowozin, and Cheng Zhang. Eddi: Efficient dynamic discovery of high-value information with partial vae. arXiv preprint arXiv:/809.11 1112, 2018.
Jianyu Miao and Lingfeng Niu. A survey on feature selection. Procedia Computer Science, 91: 919-926, 2016.
Marvin Minsky. Steps toward artificial intelligence. Proceedings of the IRE, 49(1):8-330 1961.
Feng Nan, Joseph Wang, Kirill Trapeznikov, and Venkatesh Saligrama. Fast meargin-based cost- sensitive classification. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2952-2956. IEEE, 2014.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. 1999.
Vitchyr Pong. Shixiang Gu, Murtaza Dalal, and Sergey Levine. Temporal difference models: Model- free deep rl for model-based control. arXiv preprint arXiv: /00..800220... 2018.
Thomas Rückstieß, Christian Osendorfer, and Patrick van der Smagt. Sequential feature selection for classification. In Australasian Joint Conference on Artificial Intelligence, pp. 132-141. Springer, 2011.
Daniel Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, and Zheng Wen. tutorial on thompson sampling. arXiv preprint (tt.::770000038 2017.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
Hajin Shim. Sung Ju Hwang, and Eunho Yang. Joint active feature acquisition and classification with variable-size set encoding. In Advances in neural information processing systems, pp. 1368- 1378, 2018
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, (1):9 44, 1988.
William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of samples. Biometrika, 25(3/4):285-994 1933.
Donggeum Yoo and In So Kweon. Learning loss for active learning. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 93-102, 2019.

---

Under review as a conference paper at ICLR 2021
Sara Zannone, Jose Miguel Hernandez Lobato, Cheng Zhang, and Konstantina Palla. Odin: Optimal discovery high-value information using model-based deep reinforcement learning. In Real- world Sequential Decision Making Workshop. ICML, June 2019.
Valentina Bayer Zubek, Thomas Glen Dietterich, et al. Pruning improves heuristic search for cost- sensitive learning. 2004.
12


Reviews:
[{"review_id": "xBaCrn6HGLp", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper operates in the setting where there is a possibility to adaptively acquire features for the prediction on each datapoint. The authors study classification, regression, time series and feature completion problems. The proposed solution relies on RL and introduces additional hand crafted rewards and features.\n\nStrong points:\n- The paper tries to solve an important problem that deserves additional attention from the researchers.\n- The experimental results are promising and the proposed method outperforms the baselines in several settings. The experimental results include several datasets and types of the problems.\n- The approach has shown itself being quite generic in a sense that it is applicable to different prediction problems, such as classification, regression, time series and feature completion.\n\nWeak points:\n\n- The datasets for the experiments are rather simple and have only a handful of dimensions (except for MNIST, but even MNIST is downsampled to 16x16). It makes me wonder how the proposed method would scale to high-dimensional realistic datasets.\n\n- The approach is rather heuristic and relies heavily (as shown in the ablations) on the hand-crafted features for the state representation and engineered rewards. Another heuristic part is the prediction model f_\\theta that is sometimes used to make predictions and sometimes not.\n\n- In the experiments with MNIST the features are the different pixels in the images. However, this is a very special type of the features and many methods that rely on the inductive bias about images could be used in addition. As it is the most complex experiment in this paper, I think the related works that are specific for images could be mentioned. For example, see [1] (but there are many works in this problem setting).\n\n- The auxiliary loss used in the paper seems to be quite intuitive and helps to eliminate the problem of sparse rewards. However, I am wondering if this auxiliary reward actually modifies the optimal solution to the original MDP [2]. \n\n- The experimental section is quite short and does not provide empirical evidence towards understanding of the benefits of the proposed method. For example, the authors state that the approach learns a \"non-greedy policy\". It would be informative to see how different the queried features are from the greedy selection. Then, the authors mention the superiority of the adaptive feature acquisition over selecting a fixed set of features for all datapoints. While the argument seems to be very logical, it would be nice to see the empirical confirmation of this and how big the gain of the proposed adaptive approach is.\n\n- While the paper is reasonably well written at the sentence level, the structure of the paper is a bit hard to follow. For example, the abstract and introduction go into many details about the method, which are hard to understand at that point. Then, it feels that the methods section repeats a lot of things that are already described (at the same level of details). The experimental section is on the contrary very short (for example, it does not even mention what the dataset for time series experiments is and refers directly to the appendix).\n\nI am leaning towards the rejection of this paper. While I appreciate the challenges of the studied problem and the proposed solution based on RL, I think the paper would benefit from 1) better motivation of the proposed techniques, 2) more experimental analysis to support the claims of the paper, 3) experimental evaluation on more complex datasets, and 4) some restructuring to improve the reader's understanding.\n\nQuestions:\n- What is the computational complexity of this method and how would it scale to more realistic datasets with many features?\n- Could the authors elaborate on what effect the auxiliary loss has on the original MDP?\n- Could the authors elaborate on the prediction function f_\\theta and explain when it is used and when not and why?\n- I am not entirely convinced that using the proposed \"surrogate model\" shares a lot in common with model-based RL methods. For example, \"surrogate model\" does not encode much about the transition dynamics in the environment. Could the authors elaborate more on this?\n\nAdditional comments:\n- I am not completely convinced by the argument with \"limited power\" of the sensors in time series predictions, especially in the context of the potential computational cost of the method.\n- Scales in figures vary, maybe the most informative way of selecting the scale would be to put the upper/lower bound that a method with all features can achieve?\n- I didn't understand how varying \\alpha is reflected in the plots.\n\n[1] Learning to Look Around: Intelligently Exploring Unseen Environments for Unknown Tasks, Dinesh Jayaraman, Kristen Grauman. CVPR, 2018.\n[2] Policy invariance under reward transformations: Theory and application to reward shaping, Andrew Ng, Daishi Harada, Stuart Russell. ICML, 1999.\n\n--- After reading the authors' response ---\n\nThe authors' response addressed some of my concerns. However, I could see that some of the concerns regarding novelty, relation to the prior work and experimental results are shared among several reviewers. Thus, I keep my original score and I believe that the revised manuscript would benefit from another round of reviewing.", "extra_scores": {}, "extra_text": {}}, {"review_id": "qUo0-a7nA5Q", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work present a RL and ACFlow based solution for feature wise information acquisition. \n\nPros:\n1. Clarity: The paper is very well presented. Very clear presentation of the problem formulation  and methods.  \n2. significance: the paper is addressing a very important question in real-world applications where information are associated with cost (which was clearly motivated)\n3. significance: the method obtained improved results comparing to current state-of-the-arts methods. \n\nCons:\noriginality: very incremental contribution:\n* The RL formulation of the problem has been presented in JAFA (Shim et.al 2018). Comparing to Shim et.al, the contribution seems mainly the additional of the surrogate model to do probabilistic imputation/prediction and use the accumulated reward instead of the final ine\n* All the claimed contribution comparing to Shim et.al has been done in \nODIN: Optimal Discovery of High-value INformation Using Model-based Deep Reinforcement Learning  https://realworld-sdm.github.io/paper/21.pdf \nIn ODIN, the surrogate model PVAE was used and it also used PPO. \nThus, the only main difference comparing this paper to ODIN is the choice of surrogate model\n* Lastly, the surrogate model is not new. It has been used in paper such as Dynamic Feature Acquisition with Arbitrary Conditional Flows\n Thus, the contribution is really incremental. \n\n\n Other small questions:\n1. How to read the points on your method in e.g. Figure 5,6,7.  As your method learns the stopping states. Isn't you just get an mean and std for the steps that your method stops. \n2. How did you choose the stoping points for GSM-greedy. Did you set some heuristic for greedy method to stop or it is just manual chosen points to fit the plot. \n3. I would wish to see how much Non-Greedy contribute and how much stopping action contributed separately. You can remove the stopping action and let it run to the end and compare GSM-RL and GSM-greedy. In that case, it shows how myopic is not so optimal. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "ScfUbIlLmnO", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Paper summary:\n\nIn this work, a reinforcement learning (RL) approach is proposed to solve the active feature acquisition (AFA) problem (as well as the active instance recognition problem). Comparing to existing RL approaches for AFA, the main difference of the proposed approach is to introduce a generative model (utilizing the existing ACFlow model) to learn the transition function, in order to provide additional rewards and auxiliary information. The proposed approach is evaluated on MNIST and UCI datasets, which can outperform two existing baselines.\n\n#### Advantages:\n\n- I think the paper is well-written. The proposed approach is clearly explained. In my view, the overall workflow of the algorithm is clear. It is a reasonable approach to solve the AFA problem.\n\n- The idea of learning a generative model is interesting. It can be viewed as a good realization of the general idea of active RL: try to identify uncertain states for querying.\n\n#### Disadvantages:\n\n- The advantages of existing approaches are not clearly explained. There are many existing works on AFA as cited in the paper, while they are just listed instead of explaining the relationships to the current work clearly. What is the main challenge for AFA? What are the main drawbacks of the existing approaches? Why RL is necessary? Adding more explanations on these points can be very beneficial.\n\n- From the technical perspective, the proposed approach builds upon existing RL and generative learning algorithms, thus the technical novelty is limited.\n\n- The experiments are not sufficient for the following perspectives:\n\n    - Datasets and comparison methods are limited. What are the reasons to select these specific datasets and comparison methods? In my view, adding more could be more persuasive to show the effectiveness of the proposed algorithm.\n    - No results are shown to show what the generative model learned in these experiments. This is crucial to understand what help can the generative model provides in learning.\n    - There are no datasets for real AFA tasks included (e.g. the medical treatment task introduced in section 1). This is very important for understanding the true effective tasks for the proposed approach.\n\n- Citations and the discussions on the below AFA works are missing:\n   - TEFE: A Time-Efficient Approach to Feature Extraction. https://ieeexplore.ieee.org/document/4781137\n   - Unsupervised Sequential Sensor Acquisition. http://proceedings.mlr.press/v54/hanawal17a.html\n\n#### Overall evaluation:\n\nI think the paper proposes a reasonable solution to the AFA problem, while the contributions and novelty are somehow below the standard of ICLR.\n\n#### Minor:\n$y'$ in equation 5 might be $y$.", "extra_scores": {}, "extra_text": {}}, {"review_id": "MJxoW_vccV", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the problem of active feature acquisition (AFA). The authors formulate AFA as a Markov decision process (MDF) and use reinforcement learning to resolve it. In order to overcome the sparse reward and complicated action space in this situation, the authors combine a generative surrogate model into their framework (GSMRL) to provide more feedback to the agent. Additionally, the authors adapt GSMRL into the supervised, unsupervised task (AIR) domain and introduce the corresponding dealing process in detail. Finally, the authors conduct lots of experiments to validate the superiority of GSMRL and the necessity of each component in GSMRL.\n\nThe research direction of this paper is important and interesting. In real scenarios, there are many situations in which people cannot observe all features before making a decision or constructing a model. The framework of this paper is flexible and adaptable to these situations. Therefore, this paper is useful for solving real problems.\n\nPros:\n1.\tThe presentation of this paper is good. It is easy for readers to follow the logic and the main idea of the paper.\n2.\tThe authors provide many experimental details. Readers can reproduce the experimental results based on them easily.\n3.\tThe authors conduct a lot of experiments from different perspectives to validate the effectiveness of their work.\nCons:\n1.\tThe authors should modify their symbol system. For example, when they introduce the surrogate model. They need to construct a model to produce conditional distribution. At there, X_o and X_u are a subset of the features. I recommend making the font of X_u and X_o bold. I think this modification will make people get the idea of the formula easier.\n2.\tAFA For Time Series in Section 2 Methods is too simplified. But from the chronological of this paper, this part is paralleled with AFA and AIR with GSMRL. I recommend the authors to consider how to organize the component of this part to make the paper more logical.\n\nIn summary, this paper conducts interesting and vital research. GSMRL is flexible and effective for many tasks. The authors provide an organized and qualified introduction for their framework. Therefore, I recommend accepting this paper.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studied an interesting and important problem in active learning/information acquisition (AFA), and provided an RL-based active learning scheme for a broad spectrum of AFA tasks, in both supervised (active classification/regression) and unsupervised (feature completion/recovery) domains. The reviewers generally find the paper well presented, and all appreciate the broad applicability of the proposed approach, which leverages reinforcement learning and a generative surrogate model to learn the acquisition/reward function of AFA. However, there are also shared concerns among several reviewers on the novelty and positioning of the proposed approach, as well as on whether the proposed experiments results well demonstrated the significance of the algorithm. Given that this is a purely empirical paper, both aspects are important to be properly addressed in a revision. 
Decision: Reject