Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper tackle the task of evaluating active feature acquisition (AFA) agent when the AFA agent cannot always access the complete set of features during decision making and there is distribution shift between training and evaluation stages. Specifically, it considers a subset of AFA problems where the features acquired by AFA agent is connected to a classification downstream task and the available dataset contains missing entries that could not be acquired. This work mainly tackles non time series AFA problems where the authors consider the task of feature acquisition from a fixed feature set with a sequence view, for which they claim they are the first one to propose such sequence view.

In the methodology part of this paper, the authors first illustrate the difference between set and sequence views through a hospital example where a task-specific missing data graph (m_graph) is plotted for each view. Then the authors present causal graph for AFA problem, where they identify the feature variable X as the confounder because they claim there exits a path $\bar{R} \leftarrow X \rightarrow C$. They also declare that AFA agent is allowed to implement causal interventions on $\bar{R}$ so that the problem of AFA PE can be formulated as a task of evaluating the cost objective $\mathcal{J}$ under causal intervention on $\bar{R}$. To correct the distributional shift for training and testing stages, the authors propose to integrate an existing technique for unbiased estimation, i.e., importance sampling, to their performance evaluation method.  

The authors make strong claim that their method could work on various type of missing observation patterns, including missing completely at random (MCAR), missing at random (MAR), and missing not at random (MNAR). In the main paper, they show empirical results only for MNAR.
strength_and_weaknesses: **Strength**

- The problem this paper attempts to tackle is a challenging yet understudied one which is related to many important real-world problem domains such as cost-sensitive diagnosis in healthcare.

**Weaknesses**

[*Scope*] This paper studies the task of evaluating evaluating AFA (Active Feature Acquisition) agents on a very narrow subset of AFA problems where the AFA strategies are defined as reinforcement learning (RL) policies and there is distribution shift between the training and evaluation AFA distributions. I do think the distribution shift for AFA problems is a general challenge existing in both RL-based as well as non RL-based   AFA problems . However, the authors directly connect the AFA problem with off-environment reinforcement learning without claiming the underlying relationship in the two fields, which might mislead the audiences who are new to the AFA literature when reading this paper. Moreover, the main technical solution considered in this work (causal graphs with intervention) is not specifically related to RL. Therefore, only studying performance evaluation (PE) on RL-based AFA problems lacks generality and the work should consider AFAPE with a larger scope.


[*Methodology - Sequence View*]   One major property introduced in the work is about the *sequence view* to select the features, instead of a *set view*.  First, the authors make false claims in several places in the paper, stating the *sequence view* presented in the paper is the *new view on missingness*.  To the best of my knowledge, there are some works from the existing literature studying AFA problems with such *sequence view* already, e.g., [Yin et al, 2020] (cited by the paper). The *sequential view* presented in this work is simply a special case of  the *sequence view* from existing literature. It's unfair to claim the *sequence view* is new when the authors are aware of the work. Second, I do not think the *sequence view* is a better design choice than *set view* for the non-time series AFA problems considered in this paper. For example, when test A and B can be taken in any order, the *sequence view* fail to represent homogeneous property of the sequences $A\rightarrow B$ and $B\rightarrow A$.  Would it be more appropriate to consider an alternative design which combines the *set view* with *sequence view* and let the AFA policy acquire a subset of features at each step in the *sequence view*? 

[*Methodology - Observation Missingness*]  (1)  I suggest the authors to rephrase the term. In conventional AFA literature, people refer *observation missingness* as the natural property for any AFA method, since when employing active learning  by default the algorithm would receive partial observation which comes *observation missingness*. I think the *observation missingness* referred in the paper refers to the missingness between train/evaluation stages which is a different type from that in conventional literature. (2) Though generally *observation missingness* could be classified into missing completely at random, missing at random, and missing not at random, it's suspicious if a performance evaluation (PE) method is able to work perfectly well on all the scenarios. Overall I think it's more possible a PE method could work well on a well defined subrange of AFA problems. 

[*Methodology - RL perspective*] I think the RL problem considered in the paper is offline RL problem, rather than off-policy RL, because the authors describe the task as off-environment policy training. I think the authors claim their work as off-policy RL mostly in the main paper, even though offline RL has been briefly mentioned from one place in appendix. The authors even claim offline RL is data inefficient, which I think might not make sense. I'm concerned if the RL part of its methodology is correct/clear. 

[*Methodology - Causal Graph*] I feel the presented causal graph is too high-level and not significantly novel. It is simplified for some purpose but I expect to see some reasoning over the distribution shift to play around with, such as representing the AFA policy and other factor leading to distributional shift with different variables to derive the formula with. The authors seem to have combined everything leading to *observational missingness* (AFA policy and sources of distributional shift) in one variable $R$, so that it is still unclear what happens to AFA PE model when we deal with the AFA policy and other assumed sources of distributional shift (other than AFA policy) respectively.

[*Evaluation - Choice of baselines*] I feel the choice of  AFA RL policies are insufficient. So far the authors only adopt CATE and DQN, which is insufficient. CATE is an unpublished work and DQN is a weak off-policy baseline. It would be strong if the authors consider state-of-the-art RL methods proposed for AFA problems like SeqVAE. 

[*Evaluation - Results*] Though the paper makes rather strong claim saying the method can tackle various of missing patterns, i.e., missing completely at random, missing at random, and missing not at random, the empirical evidence is quite weak.  (1)  the evaluation results in main paper only consists of the MNAR domain, which is insufficient to support the authors' claim. (2) The authors claim their method could obtain the unbiased estimate with Figure 5, but I do not find so. It seems AFAIS with IPW base results in considerable biases compared with the baseline.  (3) There should be some suggestions on which type of methods fit for wha type of missingness, but the authors only present partial view on MNAR.  


[*Evaluation - Scenarios*] I think it would be good if the authors could claim their method could work well under various types of interventions which could well cover the observation missingness types. In the current setting, it's unclear how sensitive the proposed PE method would be. 

**Other questions**:
- a ML - > an ML
- $\bar{U}^{(k)} = observe(X_i)$: this definition is ambitious as the authors do not introduce the relationship between $i$ and $k$. 
clarity,_quality,_novelty_and_reproducibility: **Clarity**

The writing of the paper is unclear and the organization of its content is not that great. The paper is formulation heavy but they are introduces in a unclear/confusing way, e.g., some characters like X and R are used to define various different variables like $X$, $\hat{X}$ and $\bar{X}$, which lack clear explanation and appear to be hard to understand when reading. The organization is not good as well, since many important parts of this work, e.g., literature for AFA methods, math derivations for the method and empirical evaluation results, are presented at appendix. The main paper is not self contained to a great extent and it not fair to let the future audiences to read through the 20+ pages to understand this work.

**Quality**

Overall I feel quality of this paper is limited at the current state because of its developed for solving a small subset of problems in AFAPE, with a very high-level causal graph derivation and weak evaluation result. 

n. Also I feel the derived causal graph is too high-level not problem-specific. I do not see apparent components representing the force of *observation missingness* and the authors seem to combine the AFA policy together with environmental factors leading to the *observation missingness* in one variable $R$. 

**Novelty**

I feel the novelty is very limited, and the authors overclaim the novelty of various important part of the paper. The authors claim they are the first to introduce the sequence view, but there is a referred work that already done so [Yin et al. 2020]. Also, since the proposed technical contribution of this paper is mostly build on the existing well adopted important sampling regime, the novelty of the solution to their highlighted AFAPE problem is also limited. 

**Reproducibility**

The reproducibility of the paper is low. The experiment engages AFA agent trained by two methods, CATE and DQN, implemented on various domains. I do not see sections introducing the important empirical settings, such as hardware/software config, architectural config for the agents, training iterations, etc, on each testified domain if they differ.  
summary_of_the_review: This paper learns an interesting problem of AFAPE, but I have concerns on its writing, problem scope, the capability of tackling all different types of *observational missingness* in one causal graph, the over simplified causal graph and its insufficient empirical evaluation.  There's a possibility the proposed method could work well on a well specified subset of AFA problems with certain property.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces AFAPE, the first approach active feature acquisition performance evaluation under missing data. It also introduces AFAIS (i.e.,  active feature acquisition importance sampling) a novel estimator that is more efficient than existing approaches. 
strength_and_weaknesses: The paper introduces a novel approach to an important, real-world problem: active feature acquisition. It sets up the problem, provides a novel framework for solving it, and validates it empirically against existing approaches.

However, the paper could be significantly improved by 
(i) a better use of the available space: in its current form, most experimental results (including all on the real-world datasets) are in the appendix. The authors should shrink the current "section 3" by at least two pages (from 5 to at most 3) and use the freed space to bring forward the experiments in the appendix and to thoroughly discuss insights on their significance and weaknesses.
(ii) the paper would greatly benefit from a motivating, real-world dataset. The three datasets in Appendix A-10 appear to be real-world, but the experimental setup is quite vague and confusing (did the author(s) chose arbitrary are the levels of missingness and miss-classification costs? if not, please explain). 
clarity,_quality,_novelty_and_reproducibility: The paper appears to introduce a novel, first-of-its-kind approach to active feature acquisition. Unfortunately, the paper is not written with a general audience in mind, which limits its potential impact across other communities. Investing half a page on intuitively introducing all key AFA concepts would go a long way to address this issue. Also the paper over-emphasizes the method's detail (5 pages) and, within its main body, it barely touches on the empirical validation (one page on a single, synthetic domain; per the comments above, the experiments from A.10 should be brought within the main paper). The paper would also greatly benefit (in both clarity and significance) if it had an in-depth discussion of the empirical insights, with their strengths and weaknesses).   
   
summary_of_the_review: Overall, the proposed approach has clear potential, but the paper needs quite a bit of re-writing/re-organizing in order to clarify its impact. As most of the required information is already within the appendices, the comments in this review should quite easy and fast to address.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: - The authors tackled an important problem called active feature acquisition (AFA) which is critical but under-explored.
- More specifically, the authors focused on the setting where the training data includes missing components which is also practical in the setting when active feature acquisition is necessary (measurement costs are high).
- The authors do not focus on proposing a new AFA model. Instead, the authors focused on evaluation of AFA models with the training data with missing components.
- The experimental results are promising, showing consistently better evaluation results in comparison to alternatives in both synthetic and real-world datasets.
strength_and_weaknesses: Strength:
- The authors tackled an important problem, active feature acquisition. Especially, the authors focused on evaluation of AFA model with missing components which is highly practical problem.
- The proposed solution makes sense and have sample efficiency and generally applicable to any missing patterns. 

Weakness:
- The experimental sections are weak. The datasets are too simple; thus, hard to say that the proposed evaluation method can be well generalized to the real-world datasets.
- In experiments, the authors only focused on one-time AFA. However, in the introduction, the authors mainly discussed about the sequential AFA.
clarity,_quality,_novelty_and_reproducibility: - The paper is easy to read and clear (except the experimental section).
- I think the proposed framework is somewhat novel.
- Some details are missing for the experiments and many details are deferred to the appendix; thus, I cannot provide high score for the reproducibility of this paper.
summary_of_the_review: 1. Experiments on real-world data
- With real world data we cannot do experiments because the assumption 1 does not hold with logged data.
- So, we should state that the real-world data experiments are actually somewhat semi-synthetic data experiments.

2. IPW
- When we use IPW, one another way is computing the probability of the agent that selects the same subset of logged features.
- Then, use that as the propensity score to compute the unbiased estimator of the cost.
- In other words, we can do weighted average of the logged costs where weights come from the probability that our AFA agent can select the exact subset of logged policy.
- In that case, we do not need to worry about only utilizing the samples with complete cases and this can significantly improve the sample efficiency of the propensity method. 

3. Small number of data points
- In many medical domains, there are only a small number of samples in the datasets.
- It would be great if the authors can provide the results with a small number of samples ranging from 0-5000 as well.

4. Number of features
- It seems like the authors used the synthetic data with 4 features.
- It would be better if the authors can provide the results with more features. 
- Based on the introduction, the authors mentioned emergency care examples with many possible measurements. To make this result more realistic, it would be better to provide the results with at least 20 features.
- Note that all the real-world experiments have less than 10 features.

5. Completeness of the experiments
- As we know, the missing patterns are quite important in this paper.
- In that case, it would be good if the authors provide MCAR, MAR, MNAR settings for one synthetic data and all 3 real-world datasets.
- Currently, the authors only provide one set of experimental results.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors consider the task of active feature acquisition, i.e., the requirement of an actor to choose whether and if so which features to acquire during run time, in the context of missing data during the training stage (in all three variants of missingness). Their proposal formulates the task as a causal graph and formulates it as a reinforcement learning problem. Their final estimator of the true cost function is then evaluated in several experimental settings.

strength_and_weaknesses: The authors tackle an important task with a novel and principled approach.  

One weakness of the paper is in the theoretical section 3. The diversity yet similarity of the notation requires close attention to follow. The authors provide a very helpful glossary in the appendix, however, it is never mentioned in the main text. Adding a reference there would allow for greater readability. The second weakness similarly relates to the structure of the paper. Given the length of Section 3, almost all, especially all experiments with real-world data, are hidden in the appendix.


clarity,_quality,_novelty_and_reproducibility: The quality of the writing is high, the same goes for the clarity apart from the points mentioned above. The approach itself is novel as far as I can say due to a somewhat limited overview of the literature in that domain. Some training details seem to be missing in the appendix to allow for full reproducibility (e.g., which gradient descent method, activation functions in the neural nets,...).
summary_of_the_review: A paper tackling an important task with a principled solution, whose weaknesses lie primarily in its structural presentation and less in its theoretical contribution.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the problem of data acquisition at test time when the training data contains missing data. This work tackles the question: which testing policy should be performed to minimize the cost but maximize performance? The paper shows that traditional ways to handle this problem lead to biased results when training data presents non completely at random missingness. The authors propose to an unbiased estimator of the cost associated with a testing policy.
strength_and_weaknesses: This paper explores an important problem in machine learning, particularly relevant to healthcare applications. This work provides a clear formalisation of the problem.

However, the model makes a key assumption that weakens the paper: "the order of acquisitions does not matter for the evaluation step". In the medical context, a first test, if positive, might make all other tests unnecessary. So, could further clarify this assumption?
clarity,_quality,_novelty_and_reproducibility: The paper is really clear and the literature gives a good overview of the field (I would recommend including it in the main text). The method is well introduced.

The paper proposes a novel approach to tackle the problem of data acquisition. My only concern is about the assumption of non-sequentiality in the testing procedure.

summary_of_the_review: This work tackles a central issue in machine learning for healthcare. However, one of the central assumptions weakens the claims of the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper considers the problem of performance evaluation for active feature acquisition (AFA). In particular, they highlight the distribution shift that typically arises in this setting and how unbiased estimation can be performed. In particular, they consider an importance sampling-based estimator (inverse probability weighting, IPW) but observe that it might be very data efficient. To this end, they propose a novel approach, AFAIS (active feature acquisition importance sampling), exploiting the particular properties of the AFA problem. They validate their approach and several baselines in synthetic and semi-real-world experiments, illustrating that their approach can perform better than the baselines.

There was no author-reviewer discussion on OpenReview unfortunately but the reviewers considered the authors' rebuttal in our virtual discussion.

Strengths of the paper.
The paper addresses an important problem that is sometimes incorrectly approached in the literature and the careful causal analysis sheds light on how to do a better job (in some cases). IPW is a standard technique but in contrast to most literature, they apply it properly to the AFA setting. AFAIS is novel and more efficient than the baselines in some cases.

Weaknesses of the paper.
The biggest weakness of the paper is its presentation/clarity and limited experiments / limited insights from experiments. All reviewers shared concerns in that regard. In particular, the authors should make all necessary assumptions explicit (in particular regarding their experimental findings) and also concisely explain their theoretical results in the main paper (e.g. regarding the unbiasedness and consistency of AFAIS). More relevant experiments should be presented in the main body of the paper, also regarding the limits of the method (which missingness cases work, etc.). More generally, the authors should consider stronger real-world-inspired experiments regarding missingness. Also, AFAIS cannot be applied in all cases - the main paper should clearly detail what AFAIS can do and what it cannot do.

In the virtual discussion, we came to the consensus that the presentation of the paper must be improved before publication. But we also want to emphasize that the paper is really interesting and will be of great interest to the community. So the authors are encouraged to improve their paper in line with the reviews and submit their work to an upcoming venue.
justification_for_why_not_higher_score: As said, the paper addresses an important problem and in general, has interesting contributions. So the paper could be accepted - but a future version of the paper is likely much better.
justification_for_why_not_lower_score: n/a
Decision: Reject