Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper studies the problem of active feature acquisition (AFA). The authors formulate AFA as a Markov decision process (MDF) and use reinforcement learning to resolve it. In order to overcome the sparse reward and complicated action space in this situation, the authors combine a generative surrogate model into their framework (GSMRL) to provide more feedback to the agent. Additionally, the authors adapt GSMRL into the supervised, unsupervised task (AIR) domain and introduce the corresponding dealing process in detail. Finally, the authors conduct lots of experiments to validate the superiority of GSMRL and the necessity of each component in GSMRL.

The research direction of this paper is important and interesting. In real scenarios, there are many situations in which people cannot observe all features before making a decision or constructing a model. The framework of this paper is flexible and adaptable to these situations. Therefore, this paper is useful for solving real problems.

Pros:
1.	The presentation of this paper is good. It is easy for readers to follow the logic and the main idea of the paper.
2.	The authors provide many experimental details. Readers can reproduce the experimental results based on them easily.
3.	The authors conduct a lot of experiments from different perspectives to validate the effectiveness of their work.
Cons:
1.	The authors should modify their symbol system. For example, when they introduce the surrogate model. They need to construct a model to produce conditional distribution. At there, X_o and X_u are a subset of the features. I recommend making the font of X_u and X_o bold. I think this modification will make people get the idea of the formula easier.
2.	AFA For Time Series in Section 2 Methods is too simplified. But from the chronological of this paper, this part is paralleled with AFA and AIR with GSMRL. I recommend the authors to consider how to organize the component of this part to make the paper more logical.

In summary, this paper conducts interesting and vital research. GSMRL is flexible and effective for many tasks. The authors provide an organized and qualified introduction for their framework. Therefore, I recommend accepting this paper.

review_title: Active Feature Acquisition with Generative Surrogate Models

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper operates in the setting where there is a possibility to adaptively acquire features for the prediction on each datapoint. The authors study classification, regression, time series and feature completion problems. The proposed solution relies on RL and introduces additional hand crafted rewards and features.

Strong points:
- The paper tries to solve an important problem that deserves additional attention from the researchers.
- The experimental results are promising and the proposed method outperforms the baselines in several settings. The experimental results include several datasets and types of the problems.
- The approach has shown itself being quite generic in a sense that it is applicable to different prediction problems, such as classification, regression, time series and feature completion.

Weak points:

- The datasets for the experiments are rather simple and have only a handful of dimensions (except for MNIST, but even MNIST is downsampled to 16x16). It makes me wonder how the proposed method would scale to high-dimensional realistic datasets.

- The approach is rather heuristic and relies heavily (as shown in the ablations) on the hand-crafted features for the state representation and engineered rewards. Another heuristic part is the prediction model f_\theta that is sometimes used to make predictions and sometimes not.

- In the experiments with MNIST the features are the different pixels in the images. However, this is a very special type of the features and many methods that rely on the inductive bias about images could be used in addition. As it is the most complex experiment in this paper, I think the related works that are specific for images could be mentioned. For example, see [1] (but there are many works in this problem setting).

- The auxiliary loss used in the paper seems to be quite intuitive and helps to eliminate the problem of sparse rewards. However, I am wondering if this auxiliary reward actually modifies the optimal solution to the original MDP [2]. 

- The experimental section is quite short and does not provide empirical evidence towards understanding of the benefits of the proposed method. For example, the authors state that the approach learns a "non-greedy policy". It would be informative to see how different the queried features are from the greedy selection. Then, the authors mention the superiority of the adaptive feature acquisition over selecting a fixed set of features for all datapoints. While the argument seems to be very logical, it would be nice to see the empirical confirmation of this and how big the gain of the proposed adaptive approach is.

- While the paper is reasonably well written at the sentence level, the structure of the paper is a bit hard to follow. For example, the abstract and introduction go into many details about the method, which are hard to understand at that point. Then, it feels that the methods section repeats a lot of things that are already described (at the same level of details). The experimental section is on the contrary very short (for example, it does not even mention what the dataset for time series experiments is and refers directly to the appendix).

I am leaning towards the rejection of this paper. While I appreciate the challenges of the studied problem and the proposed solution based on RL, I think the paper would benefit from 1) better motivation of the proposed techniques, 2) more experimental analysis to support the claims of the paper, 3) experimental evaluation on more complex datasets, and 4) some restructuring to improve the reader's understanding.

Questions:
- What is the computational complexity of this method and how would it scale to more realistic datasets with many features?
- Could the authors elaborate on what effect the auxiliary loss has on the original MDP?
- Could the authors elaborate on the prediction function f_\theta and explain when it is used and when not and why?
- I am not entirely convinced that using the proposed "surrogate model" shares a lot in common with model-based RL methods. For example, "surrogate model" does not encode much about the transition dynamics in the environment. Could the authors elaborate more on this?

Additional comments:
- I am not completely convinced by the argument with "limited power" of the sensors in time series predictions, especially in the context of the potential computational cost of the method.
- Scales in figures vary, maybe the most informative way of selecting the scale would be to put the upper/lower bound that a method with all features can achieve?
- I didn't understand how varying \alpha is reflected in the plots.

[1] Learning to Look Around: Intelligently Exploring Unseen Environments for Unknown Tasks, Dinesh Jayaraman, Kristen Grauman. CVPR, 2018.
[2] Policy invariance under reward transformations: Theory and application to reward shaping, Andrew Ng, Daishi Harada, Stuart Russell. ICML, 1999.

--- After reading the authors' response ---

The authors' response addressed some of my concerns. However, I could see that some of the concerns regarding novelty, relation to the prior work and experimental results are shared among several reviewers. Thus, I keep my original score and I believe that the revised manuscript would benefit from another round of reviewing.
review_title: Feature acquisition method that relies on RL with auxiliary rewards

final_score: rating: 4: Ok but not good enough - rejection
review_content: #### Paper summary:

In this work, a reinforcement learning (RL) approach is proposed to solve the active feature acquisition (AFA) problem (as well as the active instance recognition problem). Comparing to existing RL approaches for AFA, the main difference of the proposed approach is to introduce a generative model (utilizing the existing ACFlow model) to learn the transition function, in order to provide additional rewards and auxiliary information. The proposed approach is evaluated on MNIST and UCI datasets, which can outperform two existing baselines.

#### Advantages:

- I think the paper is well-written. The proposed approach is clearly explained. In my view, the overall workflow of the algorithm is clear. It is a reasonable approach to solve the AFA problem.

- The idea of learning a generative model is interesting. It can be viewed as a good realization of the general idea of active RL: try to identify uncertain states for querying.

#### Disadvantages:

- The advantages of existing approaches are not clearly explained. There are many existing works on AFA as cited in the paper, while they are just listed instead of explaining the relationships to the current work clearly. What is the main challenge for AFA? What are the main drawbacks of the existing approaches? Why RL is necessary? Adding more explanations on these points can be very beneficial.

- From the technical perspective, the proposed approach builds upon existing RL and generative learning algorithms, thus the technical novelty is limited.

- The experiments are not sufficient for the following perspectives:

    - Datasets and comparison methods are limited. What are the reasons to select these specific datasets and comparison methods? In my view, adding more could be more persuasive to show the effectiveness of the proposed algorithm.
    - No results are shown to show what the generative model learned in these experiments. This is crucial to understand what help can the generative model provides in learning.
    - There are no datasets for real AFA tasks included (e.g. the medical treatment task introduced in section 1). This is very important for understanding the true effective tasks for the proposed approach.

- Citations and the discussions on the below AFA works are missing:
   - TEFE: A Time-Efficient Approach to Feature Extraction. https://ieeexplore.ieee.org/document/4781137
   - Unsupervised Sequential Sensor Acquisition. http://proceedings.mlr.press/v54/hanawal17a.html

#### Overall evaluation:

I think the paper proposes a reasonable solution to the AFA problem, while the contributions and novelty are somehow below the standard of ICLR.

#### Minor:
$y'$ in equation 5 might be $y$.
review_title: a reasonable solution to active feature acquisition, while the advantages to existing approaches are not clear and the novelty is not enough

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work present a RL and ACFlow based solution for feature wise information acquisition. 

Pros:
1. Clarity: The paper is very well presented. Very clear presentation of the problem formulation  and methods.  
2. significance: the paper is addressing a very important question in real-world applications where information are associated with cost (which was clearly motivated)
3. significance: the method obtained improved results comparing to current state-of-the-arts methods. 

Cons:
originality: very incremental contribution:
* The RL formulation of the problem has been presented in JAFA (Shim et.al 2018). Comparing to Shim et.al, the contribution seems mainly the additional of the surrogate model to do probabilistic imputation/prediction and use the accumulated reward instead of the final ine
* All the claimed contribution comparing to Shim et.al has been done in 
ODIN: Optimal Discovery of High-value INformation Using Model-based Deep Reinforcement Learning  https://realworld-sdm.github.io/paper/21.pdf 
In ODIN, the surrogate model PVAE was used and it also used PPO. 
Thus, the only main difference comparing this paper to ODIN is the choice of surrogate model
* Lastly, the surrogate model is not new. It has been used in paper such as Dynamic Feature Acquisition with Arbitrary Conditional Flows
 Thus, the contribution is really incremental. 


 Other small questions:
1. How to read the points on your method in e.g. Figure 5,6,7.  As your method learns the stopping states. Isn't you just get an mean and std for the steps that your method stops. 
2. How did you choose the stoping points for GSM-greedy. Did you set some heuristic for greedy method to stop or it is just manual chosen points to fit the plot. 
3. I would wish to see how much Non-Greedy contribute and how much stopping action contributed separately. You can remove the stopping action and let it run to the end and compare GSM-RL and GSM-greedy. In that case, it shows how myopic is not so optimal. 
review_title: Well-written paper; very incremental contribution 

Meta-review: The paper studied an interesting and important problem in active learning/information acquisition (AFA), and provided an RL-based active learning scheme for a broad spectrum of AFA tasks, in both supervised (active classification/regression) and unsupervised (feature completion/recovery) domains. The reviewers generally find the paper well presented, and all appreciate the broad applicability of the proposed approach, which leverages reinforcement learning and a generative surrogate model to learn the acquisition/reward function of AFA. However, there are also shared concerns among several reviewers on the novelty and positioning of the proposed approach, as well as on whether the proposed experiments results well demonstrated the significance of the algorithm. Given that this is a purely empirical paper, both aspects are important to be properly addressed in a revision. 
Decision: Reject