Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper is an empirical study of deep active learning (especially for margin based approaches). This paper consider different tasks (openml) and how model pre-training affect the performance.
strength_and_weaknesses: Strength: this empirical study tests many DAL approaches (mostly are margin-based approaches) on openml datasets. A meaningful finding is that margin-based approachs perform consistently better than random sampling across multiple various tasks.

Weakness: some directions need to dig deeper.

1. For w/ and w/o pre-training techniques, this paper only provide an overall performance, however, how w/ and w/o pre-training techniques affect the AL performance is complex, it is related to the similarity btw target and source domain and perform different on different tasks like [r1] (section 3.4, considering dataset with distribution shift). The author should dig more informations either from existing experimental results or conduct more experiments.

2. For evaluation methods, both win matrix and box plots only present the performance on one round. As for Figure 4, there are too many curves with mixing colors. It's really hard to distinguish which is better across varying batches. Can use Area Under the Budget Curve (AUBC) [r1] or AUCLOG, which is the AUC of the learning curve when the X-axis is in log-scale [r2] or the average performance across multiple rounds (not recommended) to present an overall performance. Aslo, Figure 4 (and the similar figures) need to be re-designed by selecting more distinguishable color schemes or zoom in on some areas like [r3].

3. A deeper and more confidential analysis of the experimental results is needed. For instance, at the end of Page 7, "Methods that tied
or slightly underperformed margin on the win plots have a comparable relative gain over random, whereas the gains for others are near zero or negative (in the case of BALD). We observe that Power-BALD outperforms BALD – this is, most likely, not because BALD is ill-suited for the batch AL setting but because PowerBALD’s additive noising process mimics (biased) random sampling." Firstly, "most likely" seems not confidential. Comprehensive study needs reasonable and confident analysis. Then, Why BALD is ill-suited for the batch AL setting? Does the author try different batch settings or there are some publications support the opinion? 

[r1] Zhan X, Wang Q, Huang K, et al. A comparative survey of deep active learning[J]. arXiv preprint arXiv:2203.13450, 2022.

[r2] Mozafari B, Sarkar P, Franklin M, et al. Scaling up crowd-sourcing to very large datasets: a case for active learning[J]. Proceedings of the VLDB Endowment, 2014, 8(2): 125-136.

[r3] Kothawade S, Beck N, Killamsetty K, et al. Similar: Submodular information measures based active learning in realistic scenarios[J]. Advances in Neural Information Processing Systems, 2021, 34: 18685-18697.
clarity,_quality,_novelty_and_reproducibility: This paper is clealy written and easy to follow, it tests many margin-based DAL approaches across 69 datasets. From the aspect of the empirical study, the quality is high, the novelty is not appliable for empirical study. The author does not provide the re-implementations of their experiments thus it is not reproducible. 
summary_of_the_review: This is a well-written emprical study with sufficient comparative experiments about margin-based DAL approaches, however, the experimental analysis is somewhat weak and need to be improved.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper compares a variety of active learning algorithms on 69 tabular datasets using a multilayer perceptron model (with and without unlabeled pretraining). The results show that margin-based (i.e. best versus second best) uncertainty sampling performs the best.
strength_and_weaknesses: Strengths:
 - This paper empirically analyzes active learning algorithms (which is much needed) rather than defining yet another algorithm.
 - This paper studies tabular datasets which are under-explored in active learning experiments. 
 - This paper compares a large variety of active learning algorithms and large number of datasets.


Weaknesses:
 - This paper only evaluates with one model, a multilayer perceptron, which may be suboptimal for tabular datasets compared to tree-based methods.
 - With the very large number of active learning algorithms, it is admittedly hard to pick a list. However, I was surprised to see BADGE and BAIT were not on the list. While BADGE is computationally intensive, for image classification tasks, it works very well.
 - The evaluation criteria (performance after k rounds in 3 different batch settings) seems a bit arbitrary without much discussion.

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and is original in its choice of models/datasets for an active learning empirical survey. 

In terms of writing, I think more discussion regarding choices in the evaluation would be very helpful. For example:
 - Why the specific settings for small, medium, large?
 - Why performance after k iterations rather than area under the active learning curve, or some other metric?
 - Why the particular MLP model rather than tree-based methods?
 - What was the process for choosing the active learning methods?
summary_of_the_review: This type of active learning evaluation paper (especially for tabular datasets) is a very useful addition to the active learning literature. However, the quality of the evaluation could be improved in some ways though.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper performs a comparison of different active learning strategies on tabular data sets when trained with deep learning models that are pre-trained with self-supervised learning. The key result is that margin sampling, which is also one of the easiest active learning strategies, consistently outperforms other strategies.

strength_and_weaknesses: Strengths
- The experiments cover a large number of data sets and seeds
- The paper concludes with clear guidelines for practitioners

Weaknesses
- The focus of the study seems narrow. The paper focuses on (1) tabular data with (2) deep learning models. This is limited as the vast majority of tabular ML in practice is done not with DL but with GBT, RF, etc.
- The statistical testing seems inappropriate in that the number of tests that are being made is massive. My understanding may be incorrect so please correct me if I am wrong, but there are 49 x 20 x 3 tests performed and then the results are collated. At the very least it should be possible to reduce the number of tests to 16 x 16 by considering multiple measurements (i.e., seeds) and just doing a single test per method x method pair. An even simpler analysis would be for each data set, to perform a single ANOVA (if assumptions are met) with margin and other methods to demonstrate that margin actually outperforms all methods on each data set. This reduces the number of tests down to 49. There are statistical setups that could also be considered.
- Why do you use a larger p-value in one setting and not in the other? The p-value is supposed to be a priori fixed and not modified to increase the number of wins. If the result is not significant for the pre-determined p-value then that in itself is a result

clarity,_quality,_novelty_and_reproducibility: The paper is mostly clear, but does require prior knowledge of the experimental setup in a previous paper.
summary_of_the_review: Overall, the paper seems limited in scope and performs statistical analysis that is not clear. It could be my understanding of the stat analysis is wrong and if I am incorrect, I am happy to increase my score, but I am not yet sure about how to address the scope limitations.


==============
AFTER REBUTTAL

I appreciate the authors for carefully considering my comments and exploring a more robust statistical setup to compare their methods. I have raised my score accordingly.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors conduct a large-scale empirical study of active learning methods on tabular data using neural networks, showing that 'simple' margin-based methods outperforms (or is at least non-inferior) to more recent methods in most experimental settings. Thus, at least for practical settings, in the absence of additional problem-specific information, margin-based AL seems easy and performant. More specifically, experiments are conducted on the OpenML-CC18 benchmark (69 datasets) to compare 15+ AL querying strategies under three different {initial data, batch size} settings with and without pre-training -- showing 'simple' multi-class margin to generally be the best AL querying strategy.

The consensus strengths of this work include:
- This is ostensibly the largest empirical study of active learning querying strategies, resulting in a finding that many practitioners have claimed/suspected, but haven't had strong evidence to support.
- For tabular data, this work provides an empirical framework that can be used to evaluate future claimed advanced convincingly for practical settings.
- The core conclusion is convincing; 'simple' margin-based active learning works best in practice (possibly modulo incorporating dataset-specific domain knowledge, etc.)

The weaknesses discussed in the reviews (and my own reading) of this work include:
- While the quantity of empirical results are significant and the claimed findings are convincing, there was limited discussion regarding how/why particular methods were chosen (even if the list is relatively exhaustive). It would be a more rigorous result if the authors followed the format of a systematic review in terms of specifying inclusion criteria, etc.
- From a novelty perspective, there (obviously) aren't any methodological contributions (as this isn't the point of the paper). However, also from an empirical methods perspective, this is using an available benchmark and evaluation methods. Thus, while a potentially interesting finding and good benchmark for future work, there also aren't any methods proposed for finding out when some querying strategies are particularly strong (e.g., dimensionality of the data). 
- In the same vein as the previous item, there is limited interpretation of the results beyond the primary conclusion. There must have been additional observations that point the way to future work.
- While neural networks are increasingly used in tabular data settings, tree-based ensembles remain widely used, especially in the 'no additional information specific to the application" setting that this paper emphasizes. Thus, while not strictly necessary, adding these results would strengthen the impact amongst practitioners (who are a key consumer for this work).
- It isn't clear if the authors are going to release the code, etc. to make this a benchmark for future work on active learning. It would be helpful to minimize friction for testing future proposed methods with a simple way to plug the method in and get the same tables/figures out.

Overall, the paper is well-written, easy to follow, and convincingly supports an interesting result. Additionally, the paper was improved during the rebuttal period. After discussion, the consensus was that the findings in this paper would be of interest to the community, but additional interpretation of the results are needed (i.e., secondary findings that point to future AL work; experimental methods that facilitate deep analysis of contrasts in methods) for this to be adopted as the standard benchmarking evaluation for AL research. Additionally, while secondary, all of the reviewers would like to see tree-based ensemble results (possibly more than the pre-training results) for the tabular setting.

justification_for_why_not_higher_score: To have a higher score, we would like to see:
- A more rigorous statement regarding the inclusion criteria for different methods (and associated rationale)
- Increased interpretation of the results, including a possible deep dive into secondary experiments.
- Results using tree-based ensembles.
Decision: Reject