Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduce a novel approach to evaluate a black-box classifier and capture it's performance in different regions of the input space. This is done by modelling an accuracy surface over the space of combinations of data attributes. The model is a beta-GP with variational inference. It can also be used for active learning. Experimental evaluation shows that the model compares favourably against competing methods and illustrates the rationale behind its gains.
main_review: This is a well written paper and worthwhile contribution to a practical problem of great and increasing importance. I appreciate the detailed appendix and comparisons.
limitations_and_societal_impact: adequately addressed
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: - This paper proposed to use Gaussian processes in the embedded space of the attribution combinations to estimate the accuracy surface with uncertainty, even when faced with extreme sparsity and heteroscedasticity of labeled instances.
- The paper shows that straightforward GPs do not address this problem, and then proposed Dirichlet regularization and pooling to ease the over-smoothing and over-fitting.
- The proposed method (AAA) also tackles uncertain attribute inference, and applied the model to fractionally assign instances across arms with both labeled data and unlabeled data.
main_review: Overall the paper is easy to read and well-argued. 

\- In MF dataset, 12 out of 39 attributes are hand-picked. The selection was made according to how important they are are gender classification. I am curious if uninformative features are included into the feature space, how will the model perform, will the accuracy surface be smooth along these informative axises? And what are the consideration to remove gender-neutral features?

\+ The comparison against BetaGP and Beta-I in Section 4.5 and 4.6 clearly verified the effectiveness of the proposed methods.

\+ I agree with the limitations mentioned in in the last paragraph, and agree that they can be left to be addressed in future work.
limitations_and_societal_impact: NA
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary:         This paper describes a method to estimate the predictive performance of a black-box classifier on different tasks depending on the subsets of attributes considered. Some level of smoothness is assumed depending on the attributes considered for prediction. This smoothness is captured using a Gaussian process. The model also uses a beta distribution as the generative model for the probability of correctly predicting a label for a data instance. Further improvements are introduced in the model using a Dirichlet distribution. The proposed method is evaluated on several tasks and compare with several base-lines that are variants of the proposed method. An active learning strategy that queries the points with the highest associated variance is suggested.

main_review: I believe that this is a well written paper that proposes an interesting model to address the problem described.


The problem considered is also interesting. The experimental section is robust and the only point of criticism is that most of the methods compared are simply variants of the proposed approach.

 My main concern with this paper is, however, that the novelty is very limited. It consists in simply applying a collection of well known methods. There is no specific contribution to the problem besides the model described. In terms of core-Machine Learning, the contribution is very small, hence.


limitations_and_societal_impact: Yes, about the limtiations. The authors do not forsee any negative impact.
review_title: null

Meta-review: The reviewers are in consensus that this submission represents an interesting, well-motivated contribution to a field that I personally find is under-represented and -valued at conferences: model evaluation. The reviewers are slightly concerned around the explanation of some results and how they are presented -- I hope the authors will take these comments to heart and encourage them to incorporate these changes in preparing the camera-ready version of their manuscript.
Decision: Accept (Poster)