On the Interdependence between Data Selection and Architecture Optimization in Deep Active Learning
Authors: Pradeep Bajracharya
Rui Li
Linwei Wang
Venue: Accepted by TMLR ()

Abstract: Deep active learning (DAL) studies the optimal selection of labeled data for training deep neural networks (DNNs). While data selection in traditional active learning is mostly optimized for given features, in DNN these features are learned and change with the learning process as well as the choices of DNN architectures. How is the optimal selection of data affected by this change is not well understood in DAL. To shed light on this question, we present the first systematic investigation on: 1) the relative performance of representative modern DAL data selection strategies, as the architecture types and sizes change in the underlying DNN architecture (Focus 1), and 2) the effect of optimizing the DNN architecture of a DNN on DAL (Focus 2). The results suggest that the change in the DNN architecture significantly influences and outweighs the benefits of data selection in DAL. These results cautions the community in generalizing DAL findings obtained on specific architectures, while suggesting the importance to optimize the DNN architecture in order to maximize the effect of active data selection in DAL.

Published in Transactions on Machine Learning Research (12/2024)
On the Interdependence between Data Selection and Archi- tecture Optimization in Deep Active Learning
Pradeep Bajracharya
pb8294 @rit. edu
College of Computing and Information Sciences Rochester Institute of Technology
Rui Li
rikis Srit. edu
College of Computing and Information Sciences Rochester Institute of Technology
Linwei Wang
knwwei, war.eiiwwngrri...oc edu
College of Computing and Information Sciences Rochester Institute of Technology
Reviewed on OpenReview: /ttps://openneee net/ forum? id- difcoMrmWcE
Abstract
Deep active learning (DAL) studies the optimal selection of labeled data for training deep neural networks (DNNs). While data selection in traditional active learning is mostly op- timized for given features, DNN these features are learned and change with the learning process as well as the choices of DNN architecturos. How is the optimal selection of data affected by this change not well understood DAL. To shed light on this question, we present the first systematic investigation on: 1) the relative performance of representative modern DAL data selection strategics, the architecture types and sizes change in the un- derlying DNN architecture (Focus 1), and 2) the effect of optimizing the DNN architecture of DNN on DAT, (Focus 2). The results suggest that the change in the DNN architecture significantly influences and outweighs the benefits of data selection in DAL. These results cautions the community in generalizing DAL findings obtained on specific architectures, while suggesting the importance to optimize the DNN architecture order maximize the effect of active data selection in DAL.
1 Introduction
Active learning (AL) is long standing rescarch area that studics how to carefully select the most informative
samples to label in order to best improve learning given limited labeling budget (Cohn et al. 1996 Settles
2009). As the success of deep neural networks (DNNs) continues, there have seen substantial developments in bringing the success of AL to DNNs, creating a fast-growing subarea known as deep active learning (DAL) [Ren et al. 2021). In its essence, DAL starts with training a DNN on an initial pool of labelled data, followed by optimizing an acquisition function to select new data to be labelled to improve the DNN. This iteration continues until labeling budget desired accuracy achieved (Ren al. 2021.
The main progress in DAL has been revolving around the design and developments of acquisition functions to improve the selection of training data. Most data acquisition strategies
be categorized into efforts
that exploit DNN uncortainties (Ranganathan et al.
2017
Wang et al.
2016 Li ct al.) 2017 He et al. 2019:
Ostapuk et al. 2019 Gal et al. 2017b; Freylag et al.
2014
Käding el a..
2016 Yoo & Kweon 2019 Huang
et al. 2022 explore the diversity of unlabeled data
(Wang et al. 2017
Sener & Savarese 2017 Geifman
& El- Yaniv 2017 Shui et al.] 2020 Zhang et al. 2020b) Sinha et al.
2019
Kim al. (021). or combine
the advantages the above two hybrid fashion (Liu al. 2016
Coletta et al.] 2019 Zhdanov
2019
Ash et. al. 2020 Shui et al.
2020 Kong et. al.
2022 Wang et al. 2022
Settles al. 2007 Shukla 2022

---

Published in Transactions on Machine Learning Research (12/2024)
Additional approaches also consider the generation of informative examples (Mahapatra et al. 2018
Mayer
& Timofte 2020 Zhu & Bento 2017), utilizing the unlabeled data to pretrain the DNN feature extractor (Siméoni et al., (021) or in a semi-supervised fashion during DAL (Siméoni et al. 2021 Gao et al. 2020).
or exploiting the training dynamics of the DNN (Wang et al. 2022
In this paper, we bring to the research community an under-explored yet important question in DAL: the interdependence between choice of DNN architecture on data optimization during active learning. Tradi- tional AL developments are typically associated with pre-defined features, where data selection is primarily optimized for task decision boundaries on the given feature space. In contrast, for DNNs, the feature space is unknown and is learned as training progresses. How does this affect optimal data selection strategies? This raises questions about the potential interdependence between DNN architectures and data acquisition strategies that are not yet well understood.
First, not clear how DAL evaluation may be dependent on (or agnostic to) the choices of DNN architec- lures, considering that the vast majority of DAL works are based on predefined and fixed DNN archilectures 2018 Indeed, several recent works have investigated the reproducibility of DAL evaluations BBeck et al. 2021 Munjal et al. 2020 Mittal et al. 2019), noting that performance gaps among various DAL methods are inconsistent across experimental settings. Some also noted that the gain of DAL over random acquisition is in general marginal compared to the use of other regularization techniques (Beck et al. 2021 Munjal et al. 2020). These studies however have not focused on the effect of DNN architecture choice. In Focus of this paper, we address this knowledge gap by systematically evaluating the relative performances of seven representative DAL acquisition functions for a given DNN architecture, across two convolution-based DNN architecture types on image datasets and three transformer-based DNN architecture types on text datasets, each with different sizes.
(Geifman & EI- Yaniv]
Second, it is not clear how DAL developments can be improved by DNN architectural optimization, or what may be the most effective approach to combine DNN training data optimization with architecture optimization. Until now, only one previous work examined the effect of incremental DNN architectural search as DAT proceeds, showing that architecture optimization brought improvements compared to the use of fixed architectures during DAL (Geifinan & E1- E1- Yaniv 2017). It however was limited to the investigation of particular architecture optimization approach, designed for a particular type of DNN architectures (RESNET-18). Nor did il have a focus the relative importance between optimizing the DNN archilecture versus optimizing data selection. In parallel, there have been increasing DAL works that put an explicit focus on DNN feature space, mostly done via pre-training a feature extractor using unlabeled data (Siméoni et al., 2021 and/or simultaneously improving the learned feature during DAL using unlabeled data in addition to the acquired labeled data (Kong et al., 2022: Hacohen et al. 2022 Wang et al. 2022). All of these works, however, are performed with fixed DNN architectures. In Focus 2 of this paper, we address this knowledge gap by systematically evaluating how architecture optimization of the DNN feature extractor may affect DAL, considering three possible approaches: 1) jointly optimizing the DNN architecture with the acquired labeled data throughout the course of DAL, or 2) pre-optimizing the DNN architecture using the unlabeled data prior to DAL, or 3) pre-optimizing the DNN architecture using the initial label data prior to DAL. We conduct this investigation with three representative approaches to DNN architecture optimization that allow manageable computation when used repetitively over the course of DAL: two Bayesian architecture inference methods (KC et al.] 2021 Lee et al. 2018) and one neural architccture search (NAS) method (Chen et al., 2019), all in comparison to their fixed counterparts on the same spectrum of DAL strategies considered in Focus 1.
Experimental results from Focus 1 demonstrated that the relative performance of DAL acquisition functions was substantially influenced by the underlying DNN architecture, cautioning the community in generalizing DAL findings obtained on specific architectures. Results from Focus showed that optimizing DNN archi- tecture substantially benefited DAL. More interestingly, the gain of performance induced by an optimized vs. pre-defined DNN architecture appeared to significantly outweighed the choice of DAL acquisition strategies. This leaves a potentially important implication for DAL rescarch: while the importance of DNN architecture choices is widely accepted in standard 'passive' learning, it may be especially crucial and may suggest an under-explored research avenue in DAL in order to maximize the effect of active data selection when the labeling budget is limited.
2

---

Published in Transactions on Machine Learning Research (12/2024)
2
Related Works
Deep active learning (DAL): DAL rescarch has flourished over the years with the design of various strategies to select data from the unlabelled pool. The strategies could be broadly divided intro three categories. Uncertainty-based strategies seek examples that a DNN is most uncertain about. A variety of measures has been proposed to represent this broadly-defined uncertainty, including entropy (Joshi et al,
2009 BALD (Houlsby et al.) 2011 [Shelmanov et al.) 2021), least confidence based on softmax outputs
(Settles es 2009
margin sampling (Scheffer et al. 2001), p expected gradient length (Settles et al.
2007 Huang
et al.
2016
Zhang et al., 2017 Shukla (022). changes in outputs in response input perturbation Freytag
el al.
2014 Käding el al.) 2016), and estimation of DNN loss (Yoo & Kweon 2019 Huang et al.
2022)
Diversity- based strategies seek samples that are representative of the unlabelled data using approaches such
as density clustering (Wang et al. 2017], coresct optimization (Sener & Savaresc 2017 Geifman & El- Yaniv
2017j, and leveraging adversarial networks (Zhang et al. 2020b; (Sinha et al. 2011 Kim et al. (001).. Hybrid
strategies combine these two approaches to sample diverse data which the DNN is most uncertain about (Ash)
jet al. 2020 Shui et al.) 2020 Wang et al.) 2022 Kong et al. 2022,, such as by considering the magnitude
as well as diversity of DNN gradients (Settles et al. 2007 Shukla 2022).
In addition to these pool-based active learning where new training data is obtained by querying an unlabelled pool, there are generative approaches (Zhang et al.,
2020b) that generate examples informative to the current model. These approaches leverage generative adversarial network (GAN) to generate informative
data examples that has high entropy (Mayer & Timofie, 2020 or are closer to the decision boundary (Zhu &
Bento 2017) MMahapatra et al. 2018 Mayer & Timofte
2020 Zhu & Bento
2017). Additional approaches
include utilizing the unlabeled data to pretrain the DNN feature before DAL (iiméoni et al. 2021 or using
it in a semi-supervised fashion during DAL (Siméoni et al., 2021 Gao et al., 2020), as well as exploring DNN
training dynamics (Wang et al. 2022) measured by the derivative of training loss with respect to the
nunber of iterations assuming that models training faster generalize better. These developments are seen in
both image as well as lext data domains.
Most of these existing works were conducted on specific choices of DNN architectures, such as MLPs (Ash ct al. 2020 LeNet (Geifman & El- Yaniv,
2017
Hu et al., 2021b, CNNs (Gal et al. 2017b), and different Shui et al. 2020 on image data, or BERT (Zhang et al.
versions of VGG and RESNET (Ash et al. 2020
2020a; Schröder et al., 2021 Wertz et al.] 2022) and its two variants DistiBBERT (Schröder al.) 2001
2022) and RoBERTa (Lu & MacNamee] 2020) on text data. A lack of consistency regarding the choices of DNN architectures exist across existing studies, and it is not clear how the reported DAL evaluations may be dependent on (or agnostic to) the choices of DNN feature extractors, a critical question that will be systematically investigated in this paper.
Kirk et al.
An observation emerging in recent works (Mittal et al. 2019 Munjal et al.] 2020 Beck et al.] 2021) is the inconsistency and reproducibility of the relative performance DAL methods across experimental settings. The lack of unified experimental setting, such as size of the initial labeled pool, acquisition size, total labeling budget, random seeds, batch size, and optimizers have been credited for the inconsistencies of results reported (Munjal et al., 2020 Beck et al. 2021). It was further shown that the gain of DAL over random acquisition is in general marginal compared to other strategies, such as network regularization, data augmentation, and semi-supervised techniques (Munjal et al.) 2020; Beck et al., p This paper will add to these findings focusing on the effect of optimizing the architecture of DNN feature extractor on DAL.
Systematic evaluation of DAL methods:
DNN architecture optimization in DAL: There is large body of literature in deterministic optimiza-
tion or Bayesian inference of DNN architectures ZZoh & Le 2016 Zoph et al. 2018 Kasim et al.) 2020) KC ct al. 2021), supporting the notion that the complexity of DNN feature exitactors has substantial impact when passively learning from given data. To date, only one work investigated the effect of optimizing DNN architecture in the context of active data selection as DAL proceeds (Geifman & El- Yaniy 20188). Specifically, an incremental architectural scarch method was formulated over a modularly reduced search space customized for RESNET-18, integrated and evaluated with three existing DAL data acquisition strategies. This paper will substantially expand the scope
Feng & Darrell 2015 Lee et al.) 2018 Dikov & Bayer 2019
3

---

Published in Transactions on Machine Learning Research (12/2024)
of this previous study by: 1) investigating both supervised joint training well as unsupervised pre-training as alternative approaches to combine DNN architecture and data optimization, 2) including a variety of DNN architectures especially convolutional architectures (VGG- and RESNNET-variants) for image data and transformer architectures (BERT, DistilBERT, and RoBERTa) for text data, 3) including three architecture optimization methods representative of both Bayesian architecture inference (KC et al. 2021 Lee et al.
2018) and deterministic NAS (Chen et al., 2019) approaches, and eventually 4) deriving insights into the relationship between DNN architecture optimization and data optimization during DAL.
3 Focus 1: The Effect of DNN Architectures on DAL Performances
3.1 Methodology
In Focus 1, we systematically investigate whether and how the relative performance existing DAL acqui- sition strategics may depend on the underlying DNN feature extractor, espocially its difference choices of architecture types and sizes. We consider classification lasks on both image and lext data. On image tasks, we consider two convolutional DNN architecture types (RESNET and VGG) that are mostly monly used in DAL literature and cach with three different sizes (RESNET-18/34/55 and VGG-11/16/19), On text tasks, we consider three transformer architectures (BERT, DistiBBERT, and RoBERTa) that are most commonly used in text-based active learning tasks. We consider seven state-of-the-art DAL acquisition functions representative of uncertainty-based (Settles et al. 2007 Scheffer et al. 2001 Joshi et al. 2009) Houlsby al. 2011). diversity-based (Sener & Savarese 2017 Ash al.) 2020), and hybrid strategies (Ash)
20200, each in combination with without unsupervised pretraining the seventh DAL strategy presented in (Siméoni et al. 2021
al.
As our main focus is to understand how the performance gap among various acquisition strategies may be affected by the choice of DNN architectures, we devise metrics to measure the relative performance of different acquisition strategics, and examine its changes across choices of DNN architcctures. We further examine how such effects change as the data selection is affected by the size of the initial labeled pool, the size of each acquisition, and the presence of data augmentation. We leave out investigations on other DNN hyperparameters or regularization techniques that have been studied in previous works (Munjal et al. 2020 Beck et al. 2021
DAL acquisition strategies: We consider the following acquisition functions, against random sampling as the baseline. For all below, we consider L as the labeled datasets, U as the unlabeled pool of datasct and D as the complete dataset i.c. D = U. These are further explained in Section 3.2
Least confidence sampling (Settles al. 2007) chooses instances (x*) with the least probability scores of the predicted class p(c) among class C C (1,2, ...,,... k} in the output for given unlabelled data an i.e. x* arplan...
Margin sampling [Scheffer et al. 2001 selects instances (x*) with the smallest difference between the first and sccond largest class label probability, assuming that confident model is characterized by substantial gap between the predicted label probability and the second-highest label probability:
x
auglin ca(2)) ply = c2|2x)]
(1)
3. Entropy sampling (Joshi et al. 2009 selects instances (x*) DNN is most uncertain about as measured by the entropy (H) calculated from the output softmax probabilities:
11 Σ Pe +lo((p)) aEC
x
argmarz,et H ((x))
(2)
4

---

Published in Transactions on Machine Learning Research (12/2024)
4. BALD sampling (Houlsby et al., 2011 selects instances (x*) that generates disagrecing predictions that the model is the most uncertain on average information gain (7) (Gal et al. 2017b) i.e.
Ily w|x L)
H(yz L) Eptw] 1)) (yla, w, L)]
x* argmaTzzeeU [y; w/ 22 ,
(3)
Coresel Sampling (Sener & Savarese 2017 selects unlabeled instances that are the most different from existing labelled sample based on their feature distances:
argmazzeu minjet (g))
(4)
6. BADGE sampling (Ash et al., 2020 selects instances that generates diverse but also high gradient magnitudes in the penultimate layer of the DNN.
Unsupervised Pretraining: Based on Simeoni et al. (2021), we adopt two-step pretraining strategy for both image and text datasets. This pretraining involves a combination of alternate unsupervised clustering task and classification task supervised by the clustering labels. We begin with random initialization of the network parameters and the features from the penultimate layers are clustered using k-means clustering. These generated pseudo- labels are then utilized the ground truth for a subsequent supervised classification task which in turn updates the network parameters. The networks, once fully trained, serve as the initial models for all subsequent active learning experiments. This strategy is used in combination with all acquisition functions described above.
These acquisition functions are the commonly used benchmarks in DAL research. Among them, the first four representative of uncertainty-based acquisition strategies, with Entropy and BALD calculated based Bayesian drop-out strategies (Gal et al. 2017b Coreset is representative of diversity-basee strategies, and BADGE is a representative hybrid strategy. Each of these acquisition functions are tested without and with unsupervised pretraining as the seventh DAT. strategy considered.
Network architectures: On image data, consider two convolutional DNN architecture types that are mostly used in DAL literature, each with three different sizes: VGG-11, -16, -19; (Simonyan & Zisserman 2014), and RESNET-18, -34, -50 (He et al. 2016). On text data, we consider three transformer architectures namely BERT Deelin et al., 2018), 1 RoBERTa (Liu et al. 2019 and DistilBERT (Sanh et al. 2019) due to their prominent use in text based active learning tasks. Details of DNNs are described in the Appendix B
Evaluation metrics: We consider two quantitative metrics: 1) labeling efficiency as described in (Beck et al. 2021), which measures the amount of data required in comparison to random acquisition (as a ratio) achieve the same test accuracy; and 2) a new metric that measures the percentage of gain in test-accuracy random acquisition at each acquisition round averaged over all acquisition rounds. We use these two metrics to compare the relative performance of the considered acquisition functions across DNN architecture types and sizes.
3.2 Experiments and Results
Experiments in Focus 1 were performed on four image datasets including MNIST MNIST (Xiao et al., 2017), CIFAR10 (Krizhevsky 2009b), and SVHN (Netzer et al. 2011 and four text datasets including AGNEWS (Zhang et al., 2015). Banks77 (Casanueva et al.] 2020). DBPedia (Auer al., 2007) and QNLJ (Wang et al. 2018). Tet D be the complete dataset divided into initially labelled data L = (21,81111 and unlabelled pool of data U = 1111111114 of size [L] and |D] [L], respectively. We initialized the networks on the labelled set L. In each acquisition round, the acquisition function of choice was used choose |A] number of data points from U to be labelled next. The newly labelled data was added back the labelled set L to retrain the network and the process was repeated until [B] number of unlabelled data has been labelled.
Deng 2012) Fashion
5

---

Published in Transactions on Machine Learning Research (12/2024)
LABELING EFFICIENCY
CIFAR10
4-
RESNET18
RESNET34
RESNET50
VGG11
VGG16
VGG19
NETWORKS
LABELING EFFICIENCY
FASHIONMNIST
RESNET18
RESNET34
RESNETS0
VGG11
VGG16
VGG19
NETWORKS
LABELING EFFICIENCY
MNIST
4-
RESNET18
RESNET34
RESNET50
VGG11
VGG16
VGG19
NETWORKS
LABELING EFFICIENCY
SVHN
4-
2-
RESNET18
RESNET34
RESNET50
VGG11
VGG16
VGG19
NETWORKS
RANDOM
LEAST CONFIDENCE
MARGIN
ENTROPY
BALD
BADGE
CORESET
NORMAL TRAINING
UNSUPERVISED PRETRAINING
Figure 1: Comparison of labeling efficiency over random acquisition, for all six acquisition function with (hatchod) and without unsupervised protraining across four datasots and all DNN architectures on image classification tasks. The dashed horizontal line shows reference of random acquisition. Results show that the relative performance of acquisition strategies varies both across DNN choices on the same dataset, as well across datasets.
On image dataset, for consistent experimental settings, used an initial balanced labelled data size of 1000 unless otherwise stated. The remaining data were divided into 90-10% saining-validatte split and the size of the unlabelled pool was 30,000. A lotal of 25 acquisition rounds were performed with an acquisition size of 1000. Each round is set to train for 500 epochs with early stopping added when the training accuracy reached 99% or validation accuracy remained unchangod for 50 epochs. On text data, we used an initial labelled data size of 100 with 25 rounds of acquisition rounds performed with an acquisition size of 100. The experiments for epochs following (Devlin et. al. 2018). All experiments
for three seeds

---

Published in Transactions on Machine Learning Research (12/2024)
TEST ACCURACY GAIN
CIFAR10
0.05
0.00
RESNET18
RESNET34
RESNET50
VGG11
VGG16
VGG19
NETWORKS
TEST ACCURACY GAIN
FASHIONANIST
.0.05
0.00
RESNET18
RESNET34
RESNETS0
VGG11
VGG16
VGG19
NETWORKS
TEST ACCURACY GAIN
0.05
0.00
RESNET18
RESNET34
RESNETS0
VGG11
VGG16
VGG19
NETWORKS
TEST ACCURACY GAIN
0.05
0.00
RESNET18
RESNE 134
RESNEISH 150
VGG11
VGG16
VGG19
NETWORKS
RANDOM
LEAST CONFIDENCE
MARGIN
ENTROPY
BALD
BADGE
CORESET
NORMAL TRAINING
UNSUPERVISED PRETRAINING
Figure 2: Comparison of gain in test-accuracy over random acquisition (as 0.00 test accuracy gain). for all six acquisition function with (hatched) and without unsupervised pretraining across four datasets and all DNN architectures on image classification tasks. Similar to Figure T results show that the relative performance of acquisition strategies varies both across DNN choices on the same dataset, as well as across datasets.
workstations with RTX 2080Ti GPU and 32 GB of RAM as well as P8 and V100 GPU provided by Research Computing at Rochester Institute of Technology (Rochester Institute of Technology (024).. We used the
DISTIL github repo (Dani et al. 2021) as base skeleton for image experiments and dal-toolbox github
repo (Rauch et al. 2023] lor text. experiments. We used dbViz github repo (Somepali et al. 2022 to visualia
the decision boundaries images across different networks and acquisition functions for image dataset.
Relative performance acquisition functions: Figure summarizes the labeling efficiency and
Figure 2 summarizes the gain of DAL over random acquisition for all network architectures on the image
datasets. Figure 3 Figure summarize similar results the text datasets. The numerical values for
7

---

Published in Transactions on Machine Learning Research (12/2024)
LABELING EFFICIENCY
AGNEWS
1
BERT
ROBERTA NETWORKS
DISTILBERT
LABELING EFFICIENCY
BANKS77
1
BERT
ROBERTA NETWORKS
DISTILBERT
LABELING EFFICIENCY
DBPEDIA
BERT
ROBERTA NETWORKS
DISTILBERT
LABELING EFFICIENCY
QNLI
2
1
0
BERT
ROBERTA NETWORKS
DISTILBERT
RANDOM
LEAST CONFIDENCE
MARGIN
ENTROPY
BALD
BADGE
CORESET
NORMAL TRAINING
UNSUPERVISED PRETRAINING
Figure Comparison of labeling efficiency random acquisition, for all six acquisition function with (hatched) and without unsupervised pretraining across four datasets and all transformer architectures text tasks. The dashed horizontal line shows reference of random acquisition. Results show that the relative performance of acquisition strategies varies both across architecture choices on the same dataset, as well as across datasots.
Figure and Figure can also be found in Appendix C.5 As shown, across datasets and image text tasks, both the performance gain of all DAL strategies random acquisition, well as the relative perfor- mance among the different DAL acquisition functions, varied substantially. For instance, DAL seemed to demonstrate less benefits over random acquisition on CIFAR10 than the other image datasets, or stronger benefits on AGNEWS than the other text datasets. This is confirmed by the relative average ranking of acquisition function (Appendix C.I.I across different networks (RESNETs and VGGs for image data and BERT, ROBERTA, DISTILBERT for text data), showing that different acquisition functions have different benefits depending the datasets. Given the same DNN architecture, FASHION-MNIST (and DBPEDIA
8

---

Published in Transactions on Machine Learning Research (12/2024)
TEST ACCURACY GAIN
AGNEWS
0.02
0.00
BERT
ROBERTA NETWORKS
DISTILBERT
TEST ACCURACY GAIN
BANKS77
0.25
0.00
-0.25
BERT
ROBERTA NETWORKS
DISTILBERT
TEST ACCURACY GAIN
DBPEDIA
0.0
BERT
ROBERTA NETWORKS
DISTILBERT
TEST ACCURACY GAIN
QNLI
0.2
0.0
BERT
ROBERTA NETWORKS
DISTILBERT
RANDOM
LEAST CONFIDENCE
MARGIN
ENTROPY
BALD
BADGE
CORESET
NORMAL TRAINING
UNSUPERVISED PRETRAINING
Figure 4: Comparison gain in test-accuracy over random acquisition (as 0.00 test accuracy gain), for all six acquisition function with (hatched) and without unsupervisod pretraining across four datasots and all transformer architectures text tasks. Similar to Figure 3 results show that the relative performance of acquisition strategies varies both across architecture choices the same dataset, well across datasets.
and QNLI) seemed to induce more evident performance difference among the acquisition function tested, compared to CIFAR10 and SVHN (AGNEWS and BANKS77) where the performance gap among the tested acquisition functions were small. More importantly, even within the same dataset, the relative performance among the acquisition functions varied with DNN types and sizes: for instance, MNIST (in Figure 2h. data acquisition with least confidence criteria appeared to be the most advantageous using the larger RESNETs (RESNET34 and RESNET50), yet coreset based data acquisition appeared to outperform the others on RESNET18 and the variants of VGGs; similarly, on QNLI (in Figure 4), the hybrid BADGE acquisition seemed be favored on the BERT architecture followed by DISTILBERT, while becoming the less com- petitive the ROBERTA architecture. Furthermore, the of DISTILBERT in general seemed to induce

---

Published in Transactions on Machine Learning Research (12/2024)
RESNET 18
RESNET
RESNET 50
0.94
0.90
0.9
0.90
TEST ACCURACY
TEST ACCURACY
TEST ACCURACY
0.90
0.86
0,96
0.52
0.82
6.82
0.70
0.78
6.78
20
30
20
10
20
LABELED DATA (X1000)
LABELED DATA (x1000)
LABELED DATA (x1000)
VGG 11
VGG 16
VGG 19
0.92
0.92
0.92
TEST ACCURACY
TEST ACCURACY
TEST ACCURACY
0.50
2.05
0.50
0.86
0.80
0.04
0.04
0.82
0.62
0.82
0.78
0.78
0.78
10
20
35
15
20
10
20
LABELED DATA (x1000)
LABELED DATA (x1000)
LABELED DATA (x1000)
RANDOM
LEAST CONFIDENCE
MARGIN
ENTROPY
BALD
BADGE
CORESET
5000 INITIALIZATION SIZE
1000 INITIALIZATION SIZE
Figure 5: Comparison of test accuracy of different acquisition functions for different DNNs trained on FashionMNIST with an initialization size of 1000 (dashed) us. 5000 (solid). Similar results on other datasets can be found in the Appendix C.3
larger gaps among the different acquisition functions compared to the other two architectures on the same datasets, although favoring different acquisitions on different datasets (least confidence based acquisistion on AGNEWS, margin based acquisistion on BANKS77, and BADGE on DBPERDIA and QNLI). Note that depending on the metric used (labeling efficiency us. gain in test accuracy), the relative performance among the various acquisition functions as well as their gain over random were also different.
The use of pre-training also appeared to not only influence the relative performance of the tested acquisition functions, but also induced different gains over random acquisition in a architecture-dependet fashion, positively on some architectures and datasets (e.g., RESNET50 on CIFARIO, RESNET18 on FASHION- MNIST, ROBERTA on AGNEWS and BANKS77, and DISTILBERT on QNLI), and negatively on some (e.g. all VGGs on MNIST, all RESNETs on SVHN, and BERT on AGNEWS and BANKS77).
Ultimately, no acquisition function consistently outperformed others across datasets and architecture. This indicates that the optimality of data selection strategy depends on the choice of DNN architectures.
Effect of initialization and acquisition size: We further varied the initial label size between 1000 and 5000, and the acquisttion size among 500, 1000, 1500. As shown in Figure the increase in the initial labeled data size did not show a benefit compared to the smaller initialization size. Increasing the initial labeled data size effectively shifts the performance curve to the right compared a smaller initial labeled data size. We observe that the performance of BADGE, Entropy, and Margin was worse compared to lower initialization
10

---

Published in Transactions on Machine Learning Research (12/2024)
TEST ACCURACY GAIN
CIFAR10
0.05
0.00
RESNET18
RESNET34
RESNETSO
VGG11
VGG16
VGG19
NETWORKS
TEST ACCURACY GAIN
FASHIONMNIST
0.05
0.00
RESNET18
RESNET34
RESNETSO
VGG11
VGG16
VGG19
NETWORKS
TEST ACCURACY GAIN
MNIST
0.05
0.00
RESNET18
RESNET3M
RESNETSO
VGG11
VGG16
VGG19
NETWORKS
TEST ACCURACY GAIN
SVHN
0.05
0.00
RESNET1B
RESNET34
RESNETSO
VGG11
VGG16
VGG19
NETWORKS
RANDOM
LEAST CONFIDENCE
MARGIN
ENTROPY
BALD
BADGE
CORESET
NORMAL TRAINING
UNSUPERVISED PRETRAINING
Figure 6: Gain in test-accuracy over random acquisition (as 0.00 test accuracy gain) for different acquisition functions across DNN architectures on four datasets, with (hatched) and without data augmentation. With data augmentation, most acquisition functions exhibited similar gain over the random acquisition, and their relative performance appeared to be more clearly scparated into two groups (high- us. low-performing).
size the same number of labeled data while that BALD, and Coreset were unchanged. The change acquisition size did not produce noticeable differences the results (see Appendix C.4
Effeet of data augmentation: Fig. 6 shows the Lest accuracy gain over random acquisition achieved by the same six acquisition functions when we applied data augmentation, including horizontal flips and random crops, to all experiments. Compared the results without augmentation, two main differences can be observed: with data augmentation, most acquisition functions tested exhibited either a substantially larger gain the random acquisition (e-g., MNIST for RESNETS and VGG11), reduced error in the
11

---

Published in Transactions on Machine Learning Research (12/2024)
gain (SVHN for all networks). CIFAR10 saw both effects across all networks. 2) the gap in the performance among different acquisition functions appeared to be minimal (e.g., BADGE, entropy, least confidence, and margin) across all datasets, in comparison to the no augmentation counterpart. For FashionMNIST, the difference between with and without augmentation appears to be minimal. Additionally (not shown on Fig. G consistent lo existing findings (Beck el al., 2021 Mittal et al. 2019), the performance gain obtained by data augmentation was more substantial than the use of DAL.
4 Focus 2: DNN Architecture Optimization during DAL
4.1 Methodology
In this Focus, we investigate how DAL may be affected by the optimization of the architecture of DNN feature extractors. We consider three approaches optimize the DNN architecture, namely supervised joint-training, supervised pre-training, and unsupervised pre-training, In supervised joint-truinéng, we utilize the labeled data that increasingly acquired during DAL. This translates to setting that is similar to (Geifman & El-Yaniv 2018), where DNN architecture and weight parameters are simultaneously optimized as DAL proceeds. In supervised pre-training, we utilize the initial labelled data to optimize the DNN architecture prior to active learning. In unsupervised pre-training, we utilize the unlabeled data to optimize DNN architecture prior to active learning. The pre-trained DNN is then used to initialize DAL during which the DNN's weight parameters are updated while the optimized archilecture kept fixed. This was motivated by the recent DAL work that advocated for unsupervised DNN pre-training (Siméoni et al. (021)) but fixed architectures. In all three scttings, we consider three represcntative architecture optimization approaches.
DNN architecture optimization approach during DAL: We consider three approaches to optimize the DNN architecture during DAT. In supervised joint-oplimization of DNN architectures and data acquisi- tion, at each acquisition round within DAL, we iterated between data selection given the choice of acquisition function, and the optimization of the CNN architectural and weight parameters given the new data. In un- supervised pre-oplimizatin (UPO) of DNN archätectures, we adopt the idea from recent work (Siméoni let al. 2021 Caron et al.) 2018) that pre-trains DNN by iteratively clustering all unlabeled data and us- ing the obtained clusters as pseudo-labels to train the DNN. In supervised pre-oplimization (SPO) of DNN architectures, we use a classification task on initial labelled data to pre-train the DNN. While the original work (Siméoni et al., 2021 Caron et al. 2018)) utilized this to pre-train the weight parameters of the DNN, we use this to simultaneously optimiie the architecture and weight parameters of the DNN.
The optimized DNN architecture is then kept fixed while the pre-trained weight parameters are used to initialize the DNN at each DAL acquisition.
DNN architectures: The computational cost associated with architccture optimization is high, especially for complex architectures such as RESNET and transformers. Existing works in architecture inference optimization on transformers are also limited. Therefore we consider CNN the choice of architecture in Focus 2.
DNN architecture optimization: We consider three methods for architecture optimizations, including two Bayesian inference methods and one NAS method, considering mainly the computational feasibility in including these methods in DAL.
For NAS, we follow the PDARTS method described in (Chen et al. 2019 that defines over-complete network with L cells each with N nodes. Each node signifies a feature layer and two nodes are connected by operations o E O. We define each subsequent node xy as the linear combination of operations on node 24 defined by architecture paramcter ".
Ijj -= Σ explas))
(CJ oEO.,, Eveo exp(a"")) o(T:)
(5)
12

---

Published in Transactions on Machine Learning Research (12/2024)
The optimization is split multiple steps where for each step u, the number of operations in the operations space is reduced in comparison to previous step In addition, the number of skip-connects are controlled by
setting dropout rate which increases as we go deeper. As described in (Chen et al. 2019), : the architecture parameters are optimized on validation data and the weights of the optimal network are tuned on training data. Training time varies with the amount of data available during DAI, ranging approximately from 1.5 hours using 1000 data samples to hours for 30000 data samples.
We also include two Bayesian architecture inference methods described in (KC et al. 2021 (Depth-Dropout) and (Lee et al. 2018) (BBDropout) that translates to the optimization of the number of CNN layers (depth) and the number of CNN filters in each layer (width). In Depth-Dropout, the hidden layer is observed as:
hu = o 11-1) X h-1
(6)
and for BBDropout, we have:
h =o((W h1-1) 8 21) h4-1
where by is the feature map of the 1th hidden layer, Wi is the weight matrix for CNN filters in the 7 layer, zt the activation mask for 1th layer, O is a convolution operation, and o(.) is an activation function. We define a beta process as a prior over the number of hidden layers. A beta process sample can be denoted (hy, mi); where T. [0, denotes the activation probability of a hidden layer function hy. A stick-breaking construction of the beta process can be represented as:
Tt = -T U3, Vi Beta(a, B)
(8)
For BBDropout, in contrast, the Indian Buffet Process (IBP) is defined over neurons per hidden layer i.e.,
T = Beta(ay.
We then define a conjugate Bernoulli process inducing layer-wise binary vectors to drop out neurons per layer. The prior over the network structure variable Z can thus be formulated as
M
p(Z,vla, -l)
Bern(zmulan (x) Il
11
Beta(vila,
(10)
1=1
The variational distribution with truncation number of layers K to approximate the true posterior distribu- tion for the Depth-Dropout method is defined
M_ ula ) ((Z)))q()) = -_ ConBern 2md (m) K Beta aa bt)
(11)
where Con Bern is continuous relaxation of Bernoulli distribution. For simplicity, the number of filters in each layer is limited to M
In BBDropout, the posterior is approximated with the variational distribution of form:
q(z n X)
(())))(())))))))))))))))))))(( ammmmmmmm mm)) m12m
11
(12)
where we use Kumaraswamy distribution (Kumaraswany 1980) for ((Tm) and continuous relaxation of
Bernoulli distribution for q(2m) Fm) (Lee et al. 2018 Maddison al.) 2016 Jang al., 2016 Gal et al. 2017a)
The distinction lies in the utilization of a beta process and its corresponding Bernoulli process independently by DepthDropout, enabling the inference of both the number of layers and nodes per layer. On the other hand, BBDropout marginalizes the beta process, inferring the number of nodes in each layer but not the depth. For both DepthDropout and BBDropout, the evidence lower bound (ELBO) of the marginal likelihood of observed data D can be derived and optimized via structured stochastic variational inference (SSVI) described in
KC et al. 2021
13

---

Published in Transactions on Machine Learning Research (12/2024)
0.7
CIFAREO
TEST
TEST
0.3
0.3
TM ACQUISITION
TA ACOUISITON
ACQUUSITION
FASHORMNIST
TisT ACCURACY
TRST ACCURACT
8.3
TGST ACCURACY
0.7
0.9
TO
17
ACQUISTION
ACQUISTIIO
ACQUISITION
JOINT OPTIMIZATION
FIXED LAYER
UNSUPERVISED PRE-OPTIMEATION (UPO)
FXED LAYER
SUPERVEED PRE-OPTIMZATION (SPO)
FXED LAYER
Figure 7: Spread of performance acToss different acquisition functions (described in Section 3 given pre- defined (fixed) CNN architecture jointly-optimized, supervised pre-optimized (SPO), unsupervised pre- optimized (UPO) using MNSST (bottom row) Optimization the DNN architecture, either jointly or during pre-training, in general improved over pre-defined fixed 1, and layer CNN networks performance.
(A), BBDropout(B) and PDART&(C) for CIFAR10 (top row) and
Evaluation metrics: Here we focus on the relative performance of DAL on fixed us. optimized DNN architectures, as well as the relative contribution of DNN architecture optimization us. data acquisition optimization to the final DNN performance. We measure this by the spread of performance due to the optimization one factor when the other is controlled. This is visualized as well as quantified as the absolute difference between the maximum and minimum accuracy achieved when one these two factors varies.
4.2 Experiments and Results
Experiments in Focus 2 were performed four image datasets including MNIST [Dng 2012 Fashion MNIST (Xiao et al.) 2017), CIFAR10 (Krizhevsky 2009b), and SVHN (Netzer et al. 2011). For Depth- Dropout and BBDropout method, we used CNN with truncation K = 20 and 64 filters in each layer. A uniform distribution (0, 1.1) and (00, 1.0) was used to initialize the prior of architecture parameters a and in equation [ for Depth-Drow Dropout method. The parameters a and b were initialized to 1.3133 and 1.000, respectively, equation g for BBDropout. For PDARTS, we define total of eight operations (Max pool 3x3, Average Pool 3x3, Skip-connect, Identity, Separable Convolution 3x3 and 5x5, Dilated Convolution 3x3 and 5x5) which dropped off with dropout rate of 0.1, 0.4 and 0.7 in the subsequent training steps
14

---

Published in Transactions on Machine Learning Research (12/2024)
PlAclivation)
DEPTH
PlActivation)
DEPTH
m
ROUND
ROUND
ROUND
ROUND
ROUND
INITIALIZATION
INITIALIZATION
15
Figure 8: Visualization of optimized CNN architecture as DAL proceeds using Depth-Dropout. A: CIFARIO; FNSH.
as used in Chen et al. (2019). All the models were trained for 500 epochs. Experiments were performed on workstations with RTX 2080Ti GPU and 32 GB of RAM as well as P8 and V100 GPU provided by Research Computing at Rochester Institute of Technology (Rochester Institute of Technology 2024h. The
active learning experiments are run on four image dataset in same setting as described 3.2
The effect of optimized DNN architectures: Fig- summarizes the DAL performance across different acquisition functions when applied to architecture optimization Depth- (A), BBDropout (B), and PDARTs(CC- - and fixed DNN architectures with 1, 5, and 9 convolutional layers (between input and output layers of the net work) on CIFARIO and MashionMNIST. The fixed architectures are used as baselines to compare with the networks with optimized architectures. The DNN architectures were optimized either jointly or during pre-training using both supervised and unsupervised methods. Complete results on the rest of the datasets can be found in Appendix D.1 The figure depict that the optimized networks, particularly those optimized jointly through supervised pre-optimization (SPO) approach, consistently outperform fixed pre-defined networks. SPO and joint optimization approaches, in general, showed consistent gains in all architecture optimization methods (i.e. Fig. 7 B and C) in comparison to unsupervised pre-optinization (UPO) approach which performed least favorably. This may be attributed to the fact that the network optimization with unlabelled data was based on a clustering task different for primary DAL task.
In case of joint omiimization approach, the continuous change in the size of labeled data during DAL resulted in changes in optimal DNN architecture. This, in turn, induces continuous modifications the optimal architecture, complicating the joint optimization with data selection. To corroborate this, experimented with more extreme changes of data size by considering initial size of 200 and acquisition size of 100 in the DAL
same experiments. Figure g (A) and (B) shows the optimized architecture using Depth Dropout proceeds for CIFARIO and FashionMNIST respectively, where the bar graph on top indicates the probability of activation a layer and the column below indicates the activation of filters in each layer. As shown, both the depth and width of the CNN increased as DAL proceeded with adding labeled data. These results suggested that the joint optimization approach via small and growing labeled data in addition to weight optimization and active learning may complicate optimal data acquisition due to the continued change in the architecture, whereas leveraging initial labeled data to pre-optimize the feature space may offer simpler and yet more competitive solution.
The limited performance with unsupervised pre-training using PDARTS may be altributed to the sensilivily of the unsupervised labeling task to the number of clusters used for generating pseudo-labels. Through our experiments, we observed that employing larger cluster size with PDARTS resulted in an optimizod network with minimal test accuracy (< 30%), which improved as the number of clusters approached the actual number of labels in the dataset. In contrast, supervised pre-training of PDARTS with an initially
15

---

Published in Transactions on Machine Learning Research (12/2024)
CIFAR10
FASHIONANIST
0.22
0.40
ACCURASY
2.0
ACCUNACT
0.80
0.15
0.55
ACQUISITION
ACQUISITION
ROUNDS
LEAST CONFIDENCE
MARGIN
ENTROPY
BALD
BADGE
CORESET
Figure Spread of performance across different architectures pre-defined architecture and architeure optimization of CNN architectures given acquisition functions for CIFARI0 and FashionMNIST, in the form of spread of test accuracy over the course of DAL. Comparison with Fig. 7 shows that the optimization of DNN urchitecture outweighed the elfect of data selection in the overall DAL performance. Similar results for MNIST and SVHN are shown in Appendix D.2
RESNET 18
RESNET 34
RESNET 50
VG6
VGG 16
VGG19
The class boundaries of six architectures (RESNET 18/34/50 and VGG 11/16/19) shown across columns, plotted on the plane spanning three randomly selected images Airplane, x Bird, Frog classes) in CIFAR10 dataset for a single seed. The figure shows that change in architecture introduces change in the decision boundary.
Figure 10:
labeled dataset yielded a siguificantly superior optimized network compared to the unsupervised approach. This suggests that unsupervised pre-optimization of DNN architectures may be difficult.
Relative contribution of architecture versus data optimization: The shade or spread in Fig. describe the spread of performance between different acquisition functions for different architecture opti- mizations. A closer look into Fig. 7 suggest that, with the exception of UPO on PDARTS, the performance spread across different acquisition functions is reduced using an optimized DNN architecture compared to fixed pre-defined networks. Figure g further summarizes the spread performance across all optimized or fixed DNN architectures, for each given acquisition function on CIFAR1O and FashionMVISTT. Complete results the rest of the datasets be found in Appendix D.2. As shown, different choices of architecture parameters induced large performance gap of the DNN at any given data size for any acquisition function used. This performance spread changed as DAL proceeded, although the trend of change not consistent among datascts: on FashionMNISST, the gap among different CNN architectures appcared to be larger at the earlier stages of DAL when the data size was smaller, whereas on CIFAR this gap appeared to increase as DAL proceeded Contrasting Fig. 7 with Figure g it is evident that the impact of the DNN architectures substantially outweighed and even reduced the impact of acquisition strategies. This further suggests that, when the labeling budget small, the effort identify optimal architecture of DNN feature extractor may be critical in order to maximize the efficacy of active data selection.
16

---

Published in Transactions on Machine Learning Research (12/2024)
LEAST CONFIDENCE
BADGE
BALD
CORESET
ENTROPY
MARGIN
RANDOM
RESNET 18
RESNET 34
RESNET 50
VGG
VGG
VGG
T-SHIRT
TROUSER
PULLOVER
DRESS
COAT
SANDAL
SHIRT
SNEAKER
BAG
BOOT
Figure 11: The class boundaries of six architectures (RESNET 18/34/50 and VGG 11/16/19) along the rows trained with seven acquisition functions shown across columns, plotted on the plane spanning three randomly selected images decision boundary for the three example classes vary both across the networks as well as acqusisitinn functions. Complete result. for remaining image dalaset are added in appendix.
T-Shirt, x Pullover, Shirt) in FashionMNIST for a single seed. The
5 Discussion
5.1
Relation between DAL acquisition, DNN architecture, and decision boundaries
What may explain the observed interdependence between the DNN architecture and data acquisition? [Kolossov et al. 2023) showed in their work that better performing model in general is not always
ideal choice during active learning. Recent studies (Mickisch al. 2020 Lei et al. 2023) showed that the task decision boundary of network changes continuously during training to generalize to the available data. (Lei et al. 2023) showed that the variability in the decision boundary of network inversely affects the generalization and reproducibility
the results. Furthermore, when we consider different network archi- tectures train on the same data, the decision boundary appears to visibly vary (Somepalli et al. 2022 Illustration of this is shown in Figure 10 where the class decision boundary of six architectures plotted the plane spanning three randomly selected images (Plane, Frog and Bird) of CIFAR-10. We further extended the visualization in Figure I0 to include different acquisition functions shown for FNST in Figure 11 Complete results remaining image datasets can be found in Appendix C.2 The decision boundary appeared to change with both acquisition function as well as the underlying network architec- ture considered. The change in decision boundary appeared more prominent with change in the network architecture in comparison to the change in acquisition function.
To further understand the effect of change in decision boundary on data acquisition, we added a simple experiment half moon dataset with MLP (Multi Layer Perceptron) architectures of varying sizes (1,2
17

---

Published in Transactions on Machine Learning Research (12/2024)
X-2-y
X-4-2-y
X-4-4-2-y
Features
Data
Features
Data
Features
Data
Supervised by 1000 Cample
(A)
0.007
B1
B2
Active learning
(B)
Test: 0.871 Gain: -0.022
Test: 0.988 0.033 Gain: 0.013
Test: 0.982 = 0.022 Gain: 0.002
Figure 12: Illustrations the change of optimal decision boundary due to differences the DNN feature space (A) and its impact on DAL data selection and gain over random acquisition of data (B). Each column (separated by dotted vertical lines) represents decision boundaries achieved by different architecture of neural networks visualized in the feature (left) and data. (right) space, where 2, 4-2, 4-4-2 in between input and output y denote the number of nodes used in each hidden layer. The figures show that change in architecture changes the features and thus optimal decision boundary, which in turn affects optimal data selection in DAL.
and 3 layers) in Figure 12 We observed, as the architecture of the model changed, the optimal decision boundary changed accordingly. When aiming to actively capture such decision boundary with a small amount carefully-selected labeled data (using Entropy), the differences in feature space as shown in Figure B had clear impact. First and foremost, data selection for the same acquisition strategies (red) became very different, even for overly expressive DNNs (Fig. 12 that eventually arrived at similar decision boundaries. Furthermore, the gains of active learning over random acquisitions not only became substantially different among the various DNN architectures, but also appeared marginal compared to the performance gap induced by architecture differences.
Since optimal decision boundary plays an important role in data acquisition (Kim et al. 2021 Tharwat Schenck 2023) it may be the mechanism through which DNN architectures affects data acquisition. Future works investigating the theoretical underpinning for this relation may shed light on how to best. address the interdependence between these two optimizations in DAL.
5.2 Effect of the dataset on data acquisitions
Additionally, though not explicitly mentioned, different datasct differ from each other owing to the complex- ity of the dataset which includes underlying structure of dataset, dimensionality, noise, redundancy in data, decision boundary complexity, etc. These affect several factors that are important to the design of acquisition functions, including the uncertainty of DNN that will affect uncortainty-base basod acquisition functions, the diversity of the data samples that will affect diversity-based acquisition functions, and the decision boundary which will affect all acquisition functions. Recent work (Kim al. 2021 has shown that the ranking of
18

---

Published in Transactions on Machine Learning Research (12/2024)
acquisition functions vary depending on the nature of the dataset (balanced / imbalanced) Similarly, the scalability of acquisition functions has also been found to change with dataset Ji et al. 2023). Additionally, many existing works Mittal et al. (2019); Mayer & Timofte (2020); [eck et al. (2021) ZZhnng et al. (2024) in the active learning community have implicitly shown in their experiment that the performance of differ- ent. acquisition functions differ for different datasets. Our results presented in Figure | to Figure and Appendix CT..I confirmed this effect. Because the optimal DNN architecture depends on the underlying dataset, this may also contribute to the observed dependence between the DAL performance and choices of DNN architecture.
5.3 Limitations and Future work
Investigations in Focus 2 of the current study is focused relatively small CNN based architecture op- timization. To further generalize the findings, future studies need to extend to larger overparameterized architectures such as RESNET, VGG, and transformers. Such extension will provide a more comprehensive understanding of how optimization of diffcrent network architectures inffuence data optimization, provid- ing insights into the applicability of joint architect and data optimization strategies across a variely of DNN types. Additionally, current work is focused on relatively smaller datasets like MNIST, FashionM- NIST, SVHN and CIFAR10. Broadening the scope of datasets to include larger datasets like CIFAR100 Krizhevsky 2009a), Imagenet (Deng et al., 2009), CelebA (Liu et al. 2015 etc will help generalizing the observations across more complex datasets space.
There is also room for incorporating additional data acquisition strategies in the presented study. The current study examined a range of acquisition functions, covering prevalent strategies representative of uncertainty- basod, diversity-basedl and hybrid strategies found in the existing literature of DAL. Future work can broaden the spectrum of acquisition strategies more recent strategies, such as those incorporating neural tangent kernels to assess DNN training dynamics (Wang et al. 2022). and those incorporating the concept of semi-superrised learning in DAL.
In terms of approaches for optimizing DNN architectures, we considered a cell-based (NAS) method, specifi- cally PDARTs, along with two Bayesian DNN archilecture inference methods. For CNN net net works considered in the Focus of this study, the connections between operations within a search cell exhibit a large influ- ence on the architecture's performance. However, cell-based NAS methods are not dircctly transforrable transformers that operate on attention mechanism rather than convolution. Future work will incorporate NAS methods for transformers, such AdaBERT and NAS-BERIS to further test the generalizability of the findings obtained in Focus 2 regarding the effect of DNN architecture optimization on DAL. Future work can also include additional NAS methods for CNN-based architecture, such as dynamii-exploration DARTs where the dynamic architecture varies the kernel size or network path of CNN according to input data.
Finally, to investigate potential strategies for simultaneously optimizing DNN architecture and data selection, we considered unsupervised pre-optimizaiton of DNN artechitecture prior DAL, versus supervised joint- optimization of DNN during DAL. Both approaches, however, essentially considered DNN architecture and data optimization two separate optimization problems with their respective objective functions. An interesting yet much more challenging future research direction may be the developments of joint DNN architecture and data optimization theories, methods, and algorithms that integrate these two optimization objectives in a more coherent formulation.
6 Conclusion
In this work, we examine the influence of DNN architectures on optimal data selection in DAL. We show that the choices of DNN architecture substantially influence and outweigh data omiimization in DAL, and that its optimization helps increase the benefits of active data selection, with supervised pre-optimization being most beneficial followed by joint optimization. We hope that the findings help inform the rescarch community in improving the reproducibility of DAL evaluations by taking into account the important role of DNN architecture choices in DAI, and in opening up new research avenues that better integrate DNN
19

---

Published in Transactions on Machine Learning Research (12/2024)
architecture optimization to maximize the benefits of active data selection when the labeling budget is limited.
Acknowledgements
This work is supported by the National Science Foundation funding NSF OAC-2212548 and the NSF award no. 2045804.
References
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In International Conference on Learning Representations, 2020. URL httpps///pperreeeeeeeeet/forrorp
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. Dbpe- dia: A nucleus for web open data. In international semantic web conference, pp. 222-735. Springer, 2007.
Nathan Beck, Durga Sivasubramanian, Apurva Dani, Ganesh Ramakrishnan, and Rishabh Iyer. Effective evaluation of deep active learning on image classification tasks. arXiv preprint orXfc:2106.15324 2021.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Malthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV) pp. 132 149, 2018.
Iñigo Casamevaa, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. Efficient intent detection with dual sentence encoders. Xixee preprint urXiv:2003.0400 2020.
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1294-1303, 2019.
David A. Cohn, Zoubin Ghahramani, Michael I. Jordan, and David Cohn. Active learning with statistical models. Journal of Artificial Intelligence Research, 1996.
Luiz FS Coletta, Moacir Ponti, Eduardo R Hruschka, Ayan Acharya, and Joydeep Ghosh. Combining clustering and active learning for the detection and learning of new image classes. Neurocompuling, 358: 150-165, 2019.
Apurva Dani, Durga Sivasubramanian, and Rishabh Iyer. Distil. http::///ittuub.cco//ellllll-0 distil 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pallern recognilion, pp. 248 255. Teee, 2009.
Li Deng. The mnist database of handwritten digit images for machine learning rescarch. IEEE Signal Processing Magazine, 1911)1141-14 2012.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. orXiv preprint arXiv:1810.04500, 2018
Georgi Dikov and Justin Bayer. Bayesian learning of neural network architectures. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 730-738. PMLR, 2019.
Jiashi Feng and Trevor Darrell. Learning the structure of deep convolutional networks. In Proceedings of the IEFE international conference compuler vision, pp. 2749 2757, 2015.
20

---

Published in Transactions on Machine Learning Research (12/2024)
Alexander Freytag, Erik Rodner, and Joachim Denzler. Selecting influential examples: Active learning with expected model output changes. In Computer Vision Vision-FCCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 1V 13, pp. 562 577. Springer, 2014.
Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout. Advances in neural information processing systems, 30, 2017a.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International conference on machine learning, pp. 1192..99. PMTR, 2017b.
Mingfei Gao, Zizhao Zhang, Guo Yu, Sercan Ô Arık, Larry S Davis, and Tomas Pfister. Consistency- based semi-supervised active lcarning: Towards minimizing labeling cost. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceediiggs, Parl X 16, pp. 510 526. Springer, 2020.
Yonatan Geifman and Ran EI- Yaniv. arXiv:1711.0004,, 2017.
Deep active
learning over the long tail.
arXie preprint
Yonatan Geifman and Ran El-Yaniv. Deep active learning with neural architecture search. arXiv preprinl arXiv:1811007579, 2018.
Guy Hacohen, Avihu Dekel, and Daphna Weinshall. Active learning budget: Opposite strategies suit high and low budgets. orXiv preprint arXiw:2202.02794, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognilion, pp. 770 778, 2016.
Tao He, Xiaoming Jin, Guiguang Ding, Lan Yi, and Chenggang Yan. Towards better uncertainty sampling: Active learning with multiple views for deep convolutional neural network. In 2019 IEEF International Conference Multimedia and Expo (ICME), pp. 1360 1365. IEEE, 2019.
Neil Houlsby, Ferene Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classifi- cation and preference learning. arXiv preprinl xiv: 1112. 112.5745, 2011.
Qiang Hu, Yuejun Guo, Maxime Cordy, Xiaofei Xie, Wei Ma, Mike Papadakis, and Yves Le Traon. Towards exploring the limitations of active learning: An empirical study. In 2021 36th IEEE/ACM International Conference on Automated Software Engincering (ASE), pp. 917-929. IEEE, 2021.
Jiaji Huang, Rewon Child, Vinay Rao, Hairong Liu, Sanjeev Satheesh, and Adam Coates. Active learning for speech recognition: the power of gradients. arXiv preprint arxiv: 1xX.::661..0226,, 2016.
Siyu Huang, Tianyang Wang, Haoyi Xiong, Bihan Wen, Jun Huan, and Dejing Dou. Temporal output discrepancy for loss ciiimatiio-bbsed active learning. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Eric Jang, Shixiang Gu, and Ben Poole. Calegorical reparameterization with gumbel-softmax. arXiv preprinl arXiv:1611.0114 2016.
Yilin Ji, Daniel Kaestner, Oliver Wirth, and Christian Wressnegger. Randomness is the root of all evil: more reliable evaluation of deep active learning, In Proceedings of the IEEE/CVF Winter Conference Ayplications of Compuler Vision, pp. 3943-3333 -9522, 2023.
Ajay Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos. Multi-class active learning for image classifi- cation. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2372-22222222 2379. IEFE, 2009.
Christoph Käding, Erik Rodner, Alexander Freytag, and Joachim Denzler. Active and continuous exploration with deep neural works and expected model output changes. Xr preprint arXiv:1622.00 06129, 2016.
21

---

Published in Transactions on Machine Learning Research (12/2024)
MF Kasim, D Watson-Parris, L Deacomu, S Oliver, P Hatfield, DH Froula, G Gregori, M Jarvis, S Khatiwala, J Korenaga, et al. Building high accuracy emulators for scientific simulations with deep neural architecture search. arXiv preprint arXie:2001. 08055, 2020.
Kishan KC, Rui Li, and MohammadMahdi Gilany. Joint inference for neural network depth and dropout regularization. Advances in Neural Information Processing Systems, 34, 2021.
Kwanyoung Kim, Dongwon Park, Kwang In Kim, and Se Young Chun. Task-aware variational adversarial active learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8166-8866777 8175, 2021.
Hannah Rose Kirk, Bertie Vidgen, and Scott A Hale. Is more data better? re-thinking the impor- tance of efficiency in abusive language detection with hausformers-basee active learning, arXiv preprint arXiv:2209.1011 10193, 2022.
Germain Kolossov, Andrea Montanari, and Pulkit Tandon. Towards a statistical theory of data selection under weak supervision. arXiv preprint arXiv:2309. 14563, 2023.
Seo Taek Kong, Soomin Jeon, Dongbin Na, Jaewon Lee, Hong-Seok Lee, and Kyu-Hwan Jung, A neural pro-conditioning active learning algorithm to reduce label complexity. Advances in Neural Information Processing Systems, 25:32842-32855. 2022.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009a.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009b.
Ponnambalam Kumaraswamy. A generalized probability density function for double-bounded random pro- cesses. Journal of hydrology, 46(1-2):79 88, 1980.
Juho Lee, Saehoon Kim, Jachong Yoon, Hae Beom Lee, Eunho Yang, and Sung Ju Hwang. Adaptive network sparsification with dependent variational beta-bernoulli dropout. arXiv preprint arXiv:1805. 10896, 2018.
Shiye Lei, Fengxiang He, Yancheng Yuan, and Dacheng Tao. Understanding deep learning via decision boundary. IEEE Trunsactions Neural Networks and Learning Systems, 2023.
Ya Li, Keze Wang, Lin Nie, and Qing Wang. Face recognition via heuristic deep active learning. Chinese Conference on Biometric Recognition, pp. 97-107. Springer, 2017.
Peng Liu, Hui Zhang, and Kie B Fom. Active deep learning for classification of hyperspectral images. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 10(2):712-722) 2016.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimizod bert protraining approach. arXiv preprint Xiv: 1907. 11692, 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Jinghui Lu and Brian MacNamee. Investigating the effectiveness of representations based on pre- trained transformer- based language models in active learning for labelling text datasets. arXie preprint arXiv:200,.13138, 2020.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint or 161::611100000 2016.
Dwarikanath Mahapatra, Behzad Bozorgtabar, Jean-Philippe Thiran, and Mauricio Reyes. Efficient active learning for image classification and sogmentation using a sample selection and conditional generative adversarial network. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pp. 580-588. Springer, 2018.
22

---

Published in Transactions on Machine Learning Research (12/2024)
Christoph Mayer and Radu Timofte. Adversarial sampling for active learning. In Proceedings of the IREE/CVP Winter Conference on Applications of Computer Vision, pp. 3071-3079, 2020.
David Mickisch, Felix Assion, Florens GreBner, Wiebke Günther, and Mariele Motta. Understanding the decision boundary of deep neural networks: An empirical study, arXiv preprint arXi::2002. 01810, 2020.
Sudhanshu Mittal, Maxim Tatarchenko, Özgün Çiçek, and Thomas Brox. Parting with illusions about deep active learning. arXiv preprint SSI::192.00300 2019.
Prateek Munjal, Nasir Hayat, Munawar Hayat, Jamshid Sourati, and Shadab Khan. Towards robust and reproducible active learning using neural networks. arXiv preprint ar Xrir:2002.00564, 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Natalia Ostapuk, Jie Yang, and Philippe Cudré-Mauroux. Activelink: deep active learning for link prediction 2019.
in knowledge graphs. In The World Wide Web Conference, pp.
Hiranmayi Ranganathan, Hemanth Venkateswara, Shayok Chakraborty, and Sethuraman Panchanathan, Deep active learning for image classification. In 2017 IEEE International Conference Image Processing (ICIP), pp. 3934 3938. IEEE, 2017.
Lukas Rauch, Matthias Aßenmacher, Denis Huseljic, Moritz Wirth, Bernd Bischl, and Bernhard Sick. dal- toolbox. atttps://githie com//hhueeljicttialttoooooo.. 2023.
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Brij B Gupta, Xiaojiang Chen, and Xin Wang. A survey of deep active learning. ACM Computing Surveys (CSUR), 54(9):1-40, 2021.
Rochester Institute of Technology. Research computing services, 2024. researchcoomptiing/
URL https://ww..rtt edu/
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint 1xX::1910.01108, 2019.
Tobias Scheffer, Christian Decomain, and Stefan Wrobel. Active hidden markov models for information extraction. In International Symposium on Intelligent Dala Amalysis, pp. 309 318. Springer, 2001.
Christopher Schröder, Andreas Niekler, and Martin Potthast, Revisiting uncertainty-based query strategies for active learning with transformers. arXiv preprint arXiv:2107.05687, 2021.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint Xr:: 170.: 00080 2017.
Burr Settles. Active learning literature survey. 2009.
Burr Settles, Mark Craven, and Soumya Ray. Multiple-instance active learning. Advances in neural infor- mation processing systems, 20:1289- 1296, 2007.
Artem Shelmanov, Dmitri Puzyrev, Lyubov Kupriyanova, Denis Belyakov, Daniil Larionov, Nikita Khro- mov, Olga Kozlova, Ekaterina Artemova, Dmitry V Dylov, and Alexander Panchenko. Active learning for sequence tagging with deep pre-trained models and bayesian uncertainty estimates. arXiv preprint arXiv:2101.0813.. 2021.
Changjian Shui, Fan Zhou, Christian Gagné, and Boyu Wang. Deep aclive learning: Unified and principled method for query and training. In International Conference Artificial Intelligence and Statistics, pp. 1308-1318. PMLR, 2020.
Megh Shukla. Bayesian uncortainty and expected gradient length-regression: Two sides of the same coin? In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2367-2376, 2022.
23

---

Published in Transactions on Machine Learning Research (12/2024)
Oriane Siméoni, Mateusz Budnik, Yannis Avrithis, and Guillaume Gravier. Rethinking deep active learning: Using unlabeled data at model training. In 2020 25th International Conference on Pallern Recognation (ICPR), pp. 1220 1227. IEEE, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint Xrv: 1556, 2014.
Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. In Proceedings of the IEEE/CVF International Conference on Compuler Vision, pp. 5972 5981, 2019.
Gowthami Somepali, Liam Fowl, Arpit Bansal, Ping Ych-Chiang, Yehuda Dar, Richard Baraniuk, Micah Goldblum, and Tom Goldstein. dbviz. httpss///ggtaaub..mo/oo////aaae
2022.
Gowthami Somepalli, Liam Fowl, Arpit Bansal, Ping Yeh-Chiang, Yehuda Dar, Richard Baraniuk, Micah Goldblum, and Tom Goldstein. Can neural nets learn the same model twice? investigating reproducibility and double descent from the decision boundary perspective. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 18699-13708 2022.
Alaa Tharwal and Wolfram Schenck. A survey on active learning: State-of-the-arl, practical challenges and research directions. Mathematics, 11(4):820, 2023.
Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461 2018.
Haonan Wang, Wei Huang, Ziwei Wu, Hanghang Tong, Andrew J Margenot, and Jingrui He. Deep active learning by leveraging training dynamics. Advances in Nearol Information Processing Systems, 35:25171- 25184, 2022.
Keze Wang, Dongyu Zhang, Ya Li, Ruimao Zhang, and Liang Lin. Cost-cffective active learning for deep image classification. IEEE Transactions Circwits and Systems for Video Technology, 27(1)):2591 2600, 2016.
Min Wang, Fan Min, Zhi-Heng Zhang, and Yan-Xue Wu. Active learning through density clustering. Erpert systems with applications, 85:305 317, 2017.
Lukas Wertz, Katsiaryna Mirylenka, Jonas Kuhn, and Jasmina Bogojeska. Investigating active learning sampling strategies for extreme multi label text classification. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 4597-4605, 2022.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Pashion-mnist: novel image dataset for benchmarking machine learning algorithms, 2017.
Donggeun Yoo and In So Kweon. Learning loss for active learning. Tn Proceedings of the IEFE/CVF conference on computer vision and pattern recognition, pp. 93 10-102, 2019.
Anman Zhang, Bohan Li, Wenhuan Wang, Shuo Wan, and Weitong Chen. Mii: A novel text classification model combining deep active learning with bert. Computers, Materials & Continua, 63(3):1499 1514, 2020a.
Beichen Zhang, Liang Li, Shijie Yang, Shuhui Wang, Zheng-Jun Zha, and Qingming Huang. State-relabeling adversarial active learning. In Proceedings of the IEFE/CVF conference on computer vision and pattern recognition, pp. 8756 8765, 2020b.
Jifan Zhang, Shuai Shao, Saurabh Verma, and Robert Nowak. Algorithm selection for deep active learning with imbalanced datasets. Advances in Neural Information Processing Systems, 36, 2024.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classifi- cation. In NIPS, 2015.
24

---

Published in Transactions on Machine Learning Research (12/2024)
Ye Zhang, Matthew Lease, and Byron Wallace. Active discriminative text representation learning. In Proceedings of the AAAT Conference Artificial Intelligence, volume 31, 2017.
Fedor Zhdanov. Diverse mini-batch active learning. arXiv preprint arXiv:1901.05954 2019.
Jie Zhu and José Bento. Generative adversarial active learning. arXiv preprinl
2017.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. ar preprint arXiv:1611.01578 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc Le. Learning transferable architectures for seal- able image recognition. In Proceedings of the TEFE conference on compuler vision and pattern recognition, pp. 8697 8710, 2018.
25

---

Published in Transactions on Machine Learning Research (12/2024)
A Appendix
B Architectures of RESNET and VGG Networks
B.1 RESNET
Structure of VGG-11/16/19 and RESNET-18/34/50 shown below in format Conv x, y where is filter size and y represents number of filters.
RESNET-18
RESNET-31 Conv 3, 64
RESNET50
CONV LAYER
Conv 3, 64
Conv 3, 64
[Conv 1, 64
[Conv 3, 64
[Conv 3, 64
BLOCK
Conv 3, 64
Conv 3, 64] 2
Conv 3, 64] x 3
Conv 1, 256] x 3
[Conv 1, 128
[Conv 3, 128 Conv 3, 128] x 2
[Conv 3, 128 Conv 3, 128] x 4
BLOCK 2
Conv 3, 128
Conv 1, 512] x 4
[Conv 1, 256
[Conv 3, 256
[Conv 3, 256
BLOCK 3
Conv 3, 256
Conv 3, 256] x 2
Conv 3, 256] x 6
Conv 1, 1024] x 6
[Conv 1, 512
[Conv 3, 512
[Conv 3, 512 Conv 3, 512] 3
BLOCK 4
Conv 3, 512
Conv 3, 512] 2
Conv 1, 2048] x
FULLY CONNECTED LAYERS
FC width 512 FC width C
FC width 512 FC width C
FC width 2048
FC width C
B.2 VGG
VGG-11
VGG-16
VGG-19
Conv 3, 64
x
[Conv 3, 64)
2x
[Conv 3, 64]
Max-pool
Max-pool
Max-pool
Conv 3, 128
[Conv 3, 128
2
[Conv 3, 128
Max-pool
Max-pool
Max-pool
CONVOLUTION LAYERS
2 x [Conv 3, 256)
3
[Conv 3, 256]
4 x
[Conv 3, 256)
Max-pool
Max-pool
Max-pool
2 x
[Conv 3, 512]
3 x
[Conv 3, 512]
4
[Conv 3, 512)
Max-pool
Max-pool
Max-pool
2 [Conv 3, 512]
x [Conv 3, 512)
4 x
[Conv 3, 512
Max-pool
Max-pool
Max-pool
2 x FC width 1096)
2 x
FC width 4096)
x
FC width 4096)
FULLY CONNECTED LAYERS
FC width 1000 FC width C
FC width 1000
FC width 1000
FC width C
FC width C
26

---

Published in Transactions on Machine Learning Research (12/2024)
C Focus 1:
The Effect of DNN Architectures on DAL Performances
C.1 Effect of Acquisition Functions
C.1.1 Ranking of Acquisition function
ACQUISITION FUNCTION RANKING
CIFAR10
FashionMNIST
MNIST
SVHN
DATASETS
RANDOM
LEAST CONFIDENCE
MARGIN
ENTROPY
BALD
BADGE
CORESET
NORMAL TRAINING
UNSUPERVISED PRETRAINING
Figure 13: Average ranking plot for acquisition functions for image datascts averaged across six networks
(RESNET 18/34/50 and VGG 11/16/19). The ranking of different acquisition functions vary with dalaset
considered
ACQUISITION FUNCTION
1
2
3
RANKING
AGNEWS
BANKS77
DBPEDIA
QNLI
DATASETS
RANDOM
LEAST CONFIDENCE
MARGIN
ENTROPY
BALD
BADGE
CORESET
NORMAL TRAINING
UNSUPERVISED PRETRAINING
Figure 14: Average ranking plot for acquisition functions for text datasets averaged across three networks (BERT, ROBERTA, DISTILBERT). The ranking of different acquisition functions vary with dataset con- sidered
27

---

Published in Transactions on Machine Learning Research (12/2024)
C.2
Effect Decision Boundary
LEAST CONFIDENCE
BADGE
BALD
CORESET
ENTROPY
MARGIN
RANDOM
18
RESNET 34
RESNET
11
VGG 16
VGG 19
AIRPLANE
AUTO
BIRD
CAT
DEER
DOG
FROG
HORSE
SHIP
TRUCK
Figure 15: The class boundaries of six architectures (RESNET 18/34/50 and VGG 11/16/19) along the rows trained with seven acquisition functions shown across columns, plotted on the plane spanning three randomly selected images
Airplane, x Bird, Frog) in CIFAR10 for single seed.
28

---

Published in Transactions on Machine Learning Research (12/2024)
LEAST CONFIDENCE
BADGE
BALD
CORESET
ENTROPY
MARGIN
RANDOM
RESNET 18
RESNET 34
RESNET 50
VGG 11
VGG 16
VGG 19
Figure 16: The class boundaries six architectures (RESNET 18 18/34/50 and VGG 11/16/19) along the rows trained with seven acquisition functions shown across columns, plotted on the plane spanning three randomly selected images (classes 0, 6) in MNIST for single seed.
29

---

Published in Transactions on Machine Learning Research (12/2024)
LEAST CONFIDENCE
BADGE
BALD
CORESET
ENTROPY
MARGIN
RANDOM
RESNET 18
RESNET 34
RESNET 50
VGG 11
VGG 16
VGG 19
Figure 17: The class boundaries six architectures (RESNET 18/34/50 and VGG 11/16/19) along the rows trained with seven acquisition functions shown across columns, plotted on the plane spanning three randomly selected images (classes 0, 6) in SVHN for single soed.
30

---

Published in Transactions on Machine Learning Research (12/2024)
C.3
Effect of Initial Training Size
C.3.1
CIFAR10
RESNET 18
RESNET 34
RESNET 50
0.8
0.5
TEST ACCURACY
0.7
TEST ACCURACY
0.7
TEST ACCURACY
04
0.6
0.5
0.5
0.4
0.4
10
15
20
30
36
15
20
25
30
10
15
20
30
LABELED DATA (x1000)
LABELED DATA (x1000)
LABELED DATA (x1000)
VGG 11
VGG 16
VGG 19
0.55
0.80
0.8
0.75
TEST ACCURACY
ACCURACY
0.75
TEST TEST ACCURACY
0.70
0.70
0.7
0.05
0.60
0.60
0.55
0.50
050
0.45
10
20
30
15
20
10
15
20
LABELED DATA (x1000)
LABELED DATA (x1000)
LABELED DATA (x1000)
RANDOM
LEAST CONFIDENCE
MARGIN
ENTROPY
BALD
BADGE
CORESET
5000 INITIALIZATION SIZE
1000 INITIALIZATION SIZE
Comparison of test accuracy of different acquisttion functions for different network trained CIFAR10 with initialization size of 1000 (dashed) and 5000(solid).
Figure 18:
31

---

Published in Transactions on Machine Learning Research (12/2024)
C.3.2
FashionMNIST
RESNET 18
RESNET 34
RESNET 50
0.92
0.92
0.92
TEST ACCURACY
TEST ACCURACY
TEST ACCURACY
0.59
0.88
0.58
0.54
0.84
0.54
0.78
0.78
0.78
10
15
20
30
10
15
20
10
15
20
30
LABELED DATA (x1000)
LABELED DATA (x1000)
LABELED DATA (x1000)
VGG 11
VGG 16
VGG 19
0.92
0.92
0.92
TEST ACCURACY
TEST ACCURACY
TEST ACCURACY
0.50
2.05
6,80
0.84
0.84
0.84
0.50
080
0.50
10
15
20
30
30
15
20
50
10
15
20
25
35
LABELED DATA (x1000)
LABELED DATA (x1000)
LABELED DATA (x1000)
RANDOM
LEAST CONFIDENCE
MARGIN
ENTROPY
BALD
BADGE
CORESET
5000 INITIALIZATION SIZE
1000 INITIALIZATION SIZE
Figure 19: Comparison of test accuracy of different acquisttion functions for different network trained MNSS with initialization size of 1000 (dashed) and 5000(solid).
32

---

Published in Transactions on Machine Learning Research (12/2024)
C.3.3
MNIST
RESNET 18
RESNET 34
RESNET 50
0.98
TEST ACCURACY
TEST ACCURACY
TEST ACCURACY
0.90
0.85
0.54
0.90
0.70
0.90
16
15
20
20
15
26
10
20
LABELED DATA (X1000)
LABELED DATA (x1000)
LABELED DATA (x1000)
VGG 11
VGG 16
VGG 19
0.98
TEST ACCURACY
TEST ACCURACY
TEST ACCURACY
0.96
0.55
0.92
0.90
D10
0.75
0.89
10
15
20
10
10
15
20
25
30
10
15
20
35
LABELED DATA (x1000)
LABELED (x1000)
LABELED DATA (x1000)
RANDOM
LEAST CONFIDENCE
MARGIN
ENTROPY
BALD
BADGE
CORESET
5000 INITIALIZATION SIZE
1000 INITIALIZATION SIZE
Figure 20: Comparison of test accuracy of different acquissitinn functions for different network trained on MNIST with initialization size of 1000 (dashed) and 5000(solid).
33

---

Published in Transactions on Machine Learning Research (12/2024)
C.3.4
SVHN
RESNET 18
RESNET 34
RESNET 50
0.5
0.9
0.9
TEST ACCURACY
TEST ACCURACY
TEST ACCURACY
0.55
0.8
0.0
0.2
0.75
0.6
065
0.5
16
15
20
20
15
26
10
20
30
LABELED DATA (X1000)
LABELED DATA (x1000)
LABELED DATA (x1000)
VGG 11
VGG 16
VGG 19
0.85
0.55
TEST ACCURACY
TEST ACCURACY
TEST ACCURACY
0.80
0,80
0.75
0.75
10
15
20
10
10
15
20
25
30
10
15
20
35
LABELED DATA (x1000)
LABELED (x1000)
LABELED DATA (x1000)
RANDOM
LEAST CONFIDENCE
MARGIN
ENTROPY
BALD
BADGE
CORESET
5000 INITIALIZATION SIZE
1000 INITIALIZATION SIZE
Figure 21: Comparison of test accuracy of different acqisisitinn functions for different network trained on SVHN with initialization size of 1000 (dashed) and 5000(solid).
34

---

Published in Transactions on Machine Learning Research (12/2024)
C.4 Effect of Acquisition size
C.4.1
CIFAR10
RESNET
RESNET
RESNET
0.7
0.7
0.7
TEST TEST ACCURACY
TEST ADCURADY
TEST ACCURACY
0.5
0.5
0.5
0.3
5
10
25
20
0.3
5
10
15
20
5
10
ACQUISTION ROUNDS
ACQUISITION ROUNDS
ACQUISITION ROUNDS
VGG 11
VGG 16
VGG
0.7
0.7
0.7
TEST ACCURACY
TEST ACCURACY
TEST ACCURACY
0.5
0.5
3.5
0.4
ACQUISIIIN ROUNDS
ACQUISITION ROUNDS
ACQUISITION ROUNDS
Figure 22: Visualization of performance of Entropy acquisition function on different networks trained on CIFAR10 with acquisition size of 500 (dotted blue), 1000 (solid black) and 1500 (dashed rod) respectively
35

---

Published in Transactions on Machine Learning Research (12/2024)
RESNET
RESNET
RESNET 50
0.7
0.7
9.7
TEST ACCURACY
TEST ACQURACY
ACCURACY
0.6
0.6
0.6
0.4
0.4
0.4
ACQUISITION ROUNDS
ACQUISITION ROUNDS
ACQUISITION ROUNDS
VS0 11
VGB 16
VGG 19
0.8
0.8
0.8
TEST ACCURACY
TEST ADCURACY
TEST ACCURACY
0.6
0.6
0.6
0.4
0.4
ACQUISIION ROUNDS
ROUNDS
ACQUISITION ROUNDS
ACOUISITION SIZE
ACQUISITION SIZE 1000 ACQUISITION SIZE 1500
Figure 23: Visualization of performance of BALD acquisition function on different networks trained on
CIFAR10 with acquisition size of 500 (dotted blue), 1000 (solid black) and 1500 (dashed red) respectively
36

---

Published in Transactions on Machine Learning Research (12/2024)
RESNET
RESNET 34
RESNET 50
0.7
0.7
9.7
TEST ACCURACY
TEST ACQURACY
ACCURACY
0.6
0.6
0.6
0.4
0.4
0.4
ACQUISITIO ROUNDS
ACQUISITION ROUNDS
ACQUISITION ROUNDS
VS0 11
VGB 16
VGG 19
0.8
0.8
0.8
TEST ACCURACY
TEST ACCURACY
TEST ACCURACY
0.6
0.6
0.6
0.4
0.4
ACQUISTIOO ROUNDS
ROUNDS
ACQUISITION ROUNDS
ACOUISITION SIZE
ACQUISITION SIZE 1000 ACQUISITION SZZE 1500
Figure 24: Visualization of performance of Coreset acquisition function on different networks trained on
CIFAR10 with acquisition size of 500 (dotted blue), 1000 (solid black) and 1500 (dashed red) respectively
37

---

Published in Transactions on Machine Learning Research (12/2024)
RESNET
RESNET
RESNET 50
0.7
0.7
9.7
TEST ACCURACY
TEST ACQURACY
ACCURACY
0.6
0.6
0.6
0.4
0.4
0.4
ACQUISITION ROUNDS
ACQUISITION ROUNDS
ACQUISITION ROUNDS
VS0 11
VGB 16
VGG 19
0.8
0.8
0.8
TEST ACCURACY
TEST ACCURACY
TEST ACCURACY
0.6
0.6
0.6
0.4
0.4
ACQUISTION ROUNDS
ROUNDS
ACQUISITION ROUNDS
ACOUISITION SIZE
ACQUISITION SIZE 1000 ACQUISITION SIZE 1500
Figure 25: Visualization of performance of Margin acquisition function on different networks trained on
CIFAR10 with acquisition size of 500 (dotted blue), 1000 (solid black) and 1500 (dashed red) respectively
38

---

Published in Transactions on Machine Learning Research (12/2024)
RESNET
RESNET
RESNET 50
0.7
0.7
9.7
TEST ACCURACY
TEST ACQURACY
ACCURACY
0.6
0.6
0.6
0.4
0.4
0.4
ACQUISITION ROUNDS
ACQUISITION ROUNDS
ACQUISITION ROUNDS
VS0 11
VGB 16
VGG 19
0.8
0.8
0.8
TEST ACCURACY
TEST ACCURACY
TEST ACCURACY
0.6
0.6
0.6
0.4
0.4
ROUNDS
ROUNDS
ACQUISITION ROUNDS
ACOUISITION SIZE
ACQUISITION SIZE 1000 ACQUISITION SIZE 1500
Figure 26: Visualization of performance of Least Confidence acquisition function on different networks trained
CIFAR10 with acquisition size of 500 (dotted blue), 1000 (solid black) and 1500 (dashed red) respectively
39

---

Published in Transactions on Machine Learning Research (12/2024)
C.5 Tabular representation for Labeling Efficiency
C.5.1
Image Dataset
Lead
Margle
Entropy
Leae
Margie
Entrapy
RESNETTS
1 0790
1.2299//6 8671
1.3114N0 0881
1.006-1001040
1.008+1000000
10861+10.9937
1,064H10O 5478
0.9932H-2.0055
O 0.304413.0004
0.0000H00 1091
1 1048
RESSETT3
C.I73H+0 0773
1.022HH4G.CO 0818
D888844- D506
10002+10.005T
1:00381-2.031
1.018++10.041
RESNETS)
0
1.020HH 6843
34117440 1006
1805--000011
T,00CHHI
1,00THNN
93888--3 1919
090SHH- 1609
1537
BALD
Cornent
Confocunce
Margie
BADGE
BALO
Conoset Pro
Cenfecrnue
Margle
BADGE
1.3822N0
RESNETS4
1,006HNG
0.8878NAG
3060
1.4787H- 2508
15447H0334411
14817H6013500
1.810HH6..299
RESNETSO
1.0030HAG
1211 2004
1.2000+00 2115
1.3990N-3330 2000
14800-40.3334
1.525H+4
VB611
1 0978
0.980TNA 1323
7.222H-- 5008
17882+000000
1,090AHH 1923
1,4049H-03 2437
1,5710+10
1,000000
10002H $34)
VBO15
0.0000HH 1978
1.0099H10. 1344
5444H440 5000
1.0 TITTSCHAA 9934
1.02PFHA 1778
1.4309H- 4016
1. 1.4817//04814
14000-1000000
1.000 4555
VGGIO
L.00AH- 2008
1.1357-A0 2570
1.459414 ADDO
14270--244477
TADOOOIO 3015
1.0DDHHO 0790
1-53274-044000
1 155756034887
1.455 44457
1.5085+0-44400
BALD
Conment
Margie
EntrOpy
BALD
Connent Pro
Conficence
Margie Pre
Entropy
BADGE
RESNET19
1.342HH0 2674
160E
5064410
1,2010H1 2242
1.3436H00 2206
1.432240
121714408866
19425+109720
5685
1
2.271
1:8698H11
0966
VBD19
TASTTING
8.0173HA11555 1558
5600
2.4889HA1
2.8104+00.8500
1,5840H10 3475
1.7129H-27808
1.849H13.8888
1.7406+10
1.2005+10..90
1,8000H10
Least
Conficence
Pre
RESNETTS
1,0002110 1284
1.799H--6 3401
1.2994110 5057
12220
1,3012410 3316
1,0754410 1942
1.1399H-21430 1438
1 1.084415.1888
1.1952H- 2006
1.075440000055
1.14THHI 1970
RESNETT4
1,00TOH4 1164
1.20 3HA 2006
1.2000440 2002
10000-4000000
101152H-02144
1221554-34006
1:1348434418
1.10244003254
1.00444-0.2554
12200HH0 4057
RESNETS)
1,074T++ 1570
1.3002H-0 2034
1.2566-94412
2706-+00 6900
1,3730+10.000 4250
1,080HH0 1983
1:0400H-31886
5 1.1653-02214
1 1.04440000077
1.0000+4002000
1,100HH0 2003
VBOT1
1,0022H0 1973
1:129H100 1990
1,4327419 6188
34664110
1,202210,3473 3473
1,5034H00 1379
1.2543H/0 2744
1.3444//3306
1.29861-000000
1,2022410.300
1,3005H04419 4190
VGD18
1,050HH0.079 0737
1,2229H10 2000
1.2328H-0 3048
13117H403140
12815+10.2201
12207760.3100
1,085HH00 2940
1.14349/32206
1.1659H/3.007
12002410:2400
1.1228444 1993
1.1200H002744 2742
VGG19
1.00H+0.004
1.1034A10 2374
1.19994-2220 2408
L.077440.200
107154102436
11003400.500
0.990H/45771
1:18664-32158
1.0446/222033
1.1988--32364 2388
1.005416.3000
Figure 27: Labeling elficiency over random acquisition, for all six acquisition function with and without unsupervised pretraining (represented with suffix 'pre" after the name) along the columns across four datasets (A CIFAR, B- FashionMNIST,, C - MNIST, D - SVHN) and six DNN architectures along each row. Numerical values for labeling efficiency comparison for image dataset in Figure [
40

---

Published in Transactions on Machine Learning Research (12/2024)
C.5.2 Text Dataset
(A)
Connent
Cenfidence
BADGE
Canfidence
Margin
BADGE
DERT
1.0002+-0
1.3019H-0
1.4922--G 2008
1-45574-2741
2072146
1,103
107774461777
1,000
1,1100
1.091Ti-
ROBERTA
22607 0.288
2260:110
1. 1:1700H0
1,2021+10 2063
1.2944H10 2007
17705H60 1944
17878-16 7106
66664 0.3000
1.00 0.653
1.78786000 7104
1.2010H/4
DISTILBERT
1:6700+1044901
1,0000HH-0 2293
18000000000
2019
3007
1.2399HHG 5131
1.13724-0440 4207
10429-4022073
1:7291+-0.2447 1.729
1.20000r- 3115
(B)
BALL
Cornent
Confidance
BADGE
BALL
Cornent
Caniidenco
Margin
Enstropy
BADGE
10400H4410707
1:0029110
1.009HH0LU006 0.80194401184
1.000HH400000
10846-140.0221
14848-4013441
1.004BH10
1.1000
DISTIBERT
0.0088+-0
(C)
BALD
Cornett
Lewet Confidance
Marget
Entrepyy
BADOR
BALL Pro
Corrett Pro
Canfilenne
Margin Pro
Enstroy
BADGE Pre
(D)
Leust
Entrapy
Least
Margin
Entragy
Pro
BERT
12206+10222
089544/0 0.104
08956410.188 0.1984
0.9154400.100 1084
1.6271H/0.119
0.8128H/6.1744
0.1790/661600
O 864444 0.187
0.9664H 1979
01966400,
ROBERTA
0.0860+00.1199
0.7844-6Q 1301
08314460 1453
0.831440
0.8073H10.445 1455
0.86334-6.0607 D807
1.1100H-2.4716
882211004744
0.80244044740
0.8812H-044488
0.9078H-0-4544
0.99436-
DISTILBERT
10002160:3117
1575
0.7547H0..8155
1.008H100 3378
0.84994-23422
2:181+10.3003
1.008-46..707
1.008-+--
2:109
1.0201
Figure 28: Labeling efficiency random acquisition, for all six acquisition function with and without unsupervised pretraining (represented with suffix 'pre" after the name) along the columns across four datasets (A - AGNEWS, B BANKS77, C DBPEDIA, D QNLI) and three DNN architectures along cach row. Numerical values for labeling efficiency comparison for text dataset in Figure
41

---

Published in Transactions on Machine Learning Research (12/2024)
D
Focus 2: DNN Architecture Optimization during DAL
D.1 Spread of performance across different acquisition function
A
[8]
ROUNDS
HOUNDS
ROUNDS
(AI
(R)
ADCUMACY
DWWN
ROUNDS
ROUNDS
ROUNDS
JOINT OPTIMIZATION
FIXED 1 LAYER
UNSUPERVISED PRE OPTIMIEZAIIIONN (UPOO)
FOXE LAYER
SUPERVISED PRE OPTIMIZATION (SPO)
FOXED LAYER
Figure 29: Spread of performance across different acquisition functions given pre-defined (fixed) CNN ar- chitecture, or jointly-optimized, supervised pre-optimized (SPO), unsupervised pre-optimized (UPO) using Depth-Dropout (A), BBDropout(B) and PDARTs(C) for MNIST (top row) and SVHN (bottom row) Opti- mization of the DNN architecture, either jointly during pre-training, general improved pre-defined CNN performance.
42

---

Published in Transactions on Machine Learning Research (12/2024)
D.2 Spread of performance across different network architectures
MNIST
SVHN
1.00
0.00
ACCURACY
ACCURACY
0.80
0.
0.00
0.70
0.00
16
ACQUISITOON
ACQUISITION
LEAST CONFIDENCE
MARGIN
ENTROPY
BALD
BADGE
CORESET
Figure 30: Spread of performance across different CNN architectures given acquisition functions for MNIST and SVHN, in the form of spread of test accuracy over the course of DAL.
43


Reviews:
[{"review_id": "SRP5teckc2", "reviewer": "Reviewer_Htho", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The problem is well-formulated and the past literature is summarized well. \n- The authors conduct extensive empirical experimentations throughout the paper, such as all results are presented across various datasets, model architectures, model optimization schemes, data augmentation and data acquisition schemes as applicable.\n- The inclusion of multiple approaches to architecture optimization, particularly supervised and unsupervised pretraining, adds an interesting dimension to the discussion of DAL and opens new avenues for future research.\n\n### Weaknesses\n Summary of weaknesses is provided, for details please look at requested changes.\n- I think many of the conclusions here are qualified for small networks working with small data given the size of the empirical experiments. It is unclear if similar findings hold for modern over-parametrized networks. Though this is not an issue, I think it should be specifically mentioned in the paper.\n- Some of the conclusions are not thoroughly backed by the experimental evidence.\n- Given the suggested future work of setting up a benchmark, it would significantly strengthen the paper if they could release the code for their experiments. This would enhance reproducibility and allow the community to build upon their findings.\n- There are minor errors throughout the paper, indicating the need for a more thorough review and proofreading.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors empirically investigate the role of Deep Neural Network (DNN) architectures in Deep Active Learning (DAL), addressing two complementary questions:\n\n1. Whether any single DAL acquisition strategy consistently outperforms others across various architectures and datasets. Through their experiments, they demonstrate that among the diverse schemes tested, no single strategy consistently prevails.\n2.  Whether optimizing the architecture can provide greater benefits than the improvements achieved through DAL itself, which they answer affirmatively.\n\nWhile the questions posed are intriguing, there are some concerns with the claims of the study, as outlined below.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Major \n- Focus 1:\n    - The error bars reported in Figures 1-4 appear to be quite large, and I assume these are calculated over three seeds. Increasing the number of seeds would likely make these findings more reliable, as the current error bars suggest that all methods perform statistically similarly. This may result in qualitative conclusions (such as those in the section “Relative performance of acquisition functions”) being based on visual differences in the bars rather than the error bars. Additionally, I have two minor suggestions:\n        - Consider using a log scale for the y-axis in the plots describing label efficiency. This could better represent the differences visually.\n        - Reporting the exact values in a table would also be helpful. Specifically, what is the random test accuracy? At what iso-accuracy levels are the labeling efficiencies reported?\n    - Regarding the conclusion on data augmentation in Figure 6, it seems that the models trained with data augmentation simply see more data points, which could explain the observed increase in accuracy. For a fair comparison, the x-axis should reflect the same number of data points seen across models.\n- Focus 2:\n    - One of the key issues with the conclusions drawn from this part (e.g., in Figure 8) is that the error bars represent the spread across different data acquisition strategies. To measure performance spread due to architecture optimization, a more appropriate experiment would compare each data acquisition strategy individually, alongside random sampling, while keeping the architecture either fixed or optimized for each strategy. The current experiments showing that architecture optimization has a stronger effect might be a case of “Simpson’s paradox.” Either the data for each acquisition strategy should be provided, or the claim should be rephrased to reflect the metric being evaluated (e.g., “averaging across data acquisition strategies, optimizing model architecture provides higher gains”).\n    - Additionally, as shown in Figure 8, supervised pre-optimization (SPO) performs comparably to joint optimization. The fact that using initially labeled data to optimize architecture before DAL performs competitively suggests an alternative interpretation: data acquisition from an optimized network can be as competitive as joint optimization. Although this point is mentioned in the paper, the main conclusions seem to overlook its significance.\n\n### Minor \n- Equation mistakes in DAL acquisition strategies described on Pg 5:\n    - The equation for margin sampling should use argmin instead of argmax to be consistent with the notation.\n    - The definition of entropy sampling is missing a minus sign in the entropy formula.\n    - Symbols used in the BALD sampling equation are not defined.\n- “Entroy” should be corrected to “Entropy” on Page 5.\n- Clarification on Evaluation Metrics for Part 1, Pg 6\n    - Does the second metric in Figures 3 and 4 represent an average over all acquisition rounds when it measures “the percentage of gain in test accuracy over random acquisition at each acquisition round”? This should be clarified in the “evaluation metrics” section.\n- Typographical error in Page 9 in “Effect of data augmentation” section: “1) most acquisition functions tested exhibited either a substantially larger gain over the random acquisition (e.g., CIFAR10 for all networks, MNIST for RESNETs and VGG11), or a reduced error in the gain (CIFAR10, SVHN for all networks)”. This is contradictory statement e.g. wrt CIFAR10.\n- Pg 14, a full stop is missing after “M” in the phrase “each layer is limited to M.”", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kNr4lcDME8", "reviewer": "Reviewer_x9aW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nS1 - This work has performed an extensive empirical investigation on several existing DAL methods under various settings, revealing that their effectiveness do not necessarily generalise across different settings.\n\nS2 - It has demonstrated that data augmentation and supervised- prior and joint network architecture optimisation are potentially beneficial in making DAL methods perform more consistently across different settings.\n\nWeaknesses\n\nW1 - Over-claim the contribution: In abstract, the work claims to shed light on the impact of changes in feature space on optimal data selection. However, the only content pertaining to the feature space is Figure.~1 and its description in ``Introduction\" section which only demonstrates that feature space corresponding to three small-scale networks are different. It does not establish a causal relationship between feature space changes and the data acquisition of DAL methods.\n\nW2 - Over-generalisation: In Section.~5 - 1st paragraph: this work suggests network architecture optimisation as a potential solution to ensuring the generalisation of DAL methods. However, the beneficial effects of joint/prior supervised network architecture optimisation on DAL methods are only observed for a single type of network with various depths. Shouldn't the work also be cautious in suggesting its generalisation just like it cautioning in overly generalising DAL methods across different network architectures?\n\nW3 - Presentation: \n\nW3.1 - In Focus 2, it is unclear what ``DNN architectures with 1, 5, and 9 layers\" are. It is also unknown which DNN architecture (number of layers) is used as baseline for the network architecture optimisation experiments.\n\nW3.2 - Figure. 10 and Figure. 21 are associated with caption \"Spread of performance across different architectures\". However, the plot only illustrates how test accuracy changes with increasing data acquisition rounds using six DAL methods on CIFAR and FASHIONMNIST datasets. Also, the shaded areas of different DAL methods are highly overlapped, it is difficult to tell the test accuracy ``spread\" of individual methods.\n\nW3.3 - Figure.~20 is associated with caption ``Spread of performance across different acquisition functions\". However, none of the DAL method is included.\n\nW3.4 - Figure.~15 - 19, the ``spread\" for red and blue plots are missing in all VGG models (second row).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work provides an empirical study on the performance of six Deep Active Learning (DAL) methods under various settings, including different network architectures, datasets, initial labelled data size and subsequent acquisition size, and the presence/absence of data augmentation, pre-training, and network architecture optimisation methods. It demonstrates that effectiveness of these DAL methods do not necessarily generalise across different network architectures and training/valuation datasets. Further, data augmentation and network optimisation (both joint and supervised prior methods) are generally beneficial when paired with DAL methods. In contrast, unsupervised pre-training and unsupervised network architecture optimisation are found less useful and sometimes even harmful to DAL methods.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical:\n\nPlease address the weaknesses W. 1 and W. 2.\n\nMild:\n\nPlease address the weakness W. 3.1 - W. 3.4.\n\nIn Figure.~1, it is interesting that DAL method selects data around the decision boundaries in both input and feature spaces at early stages. But at a late stage (last row), it seems to acquire data that are farther away from both decision boundaries, which is most noticeable with model X-4-2-y. Is there an explanation for such a phenomenon?\n\nIn Figure.~1, is it possible to disclose which DAL method is used here? Additionally, are the decision boundaries obtained before or after training on the newly acquired data on the same row?\n\nTrivial:\n\nTypo in Page~5 - last paragraph: ``Entroy\" -> Entropy.\n\nAcronym ``IBP\" undefined in Page. 13 above Equation. 9.", "broader_impact_concerns": "Nil.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "a2jTUh5LpW", "reviewer": "Reviewer_GpUt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n- Extensive experiments are conducted across different model structures and datasets.\n- The finding that architecture search outweighs DAL acquisition strategies is potentially impactful and guides further DAL research.\n\n**Weaknesses**:\n\n- The paper does not provide any theoretical insight into how the decision boundary is affected by model structure. An analysis explaining the theoretical underpinnings of the impact of architecture on feature space and decision boundaries would have been beneficial.\n- The conclusion that architecture search has more influence than acquisition functions is somewhat intuitive, given that a well-optimized architecture naturally improves learning efficiency.\n- Other data formats (e.g., time series) are not tested, although it is foreseeable that this would not significantly affect the conclusions drawn in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the problem of deep active learning (DAL) by empirically examining how deep neural network (DNN) architectures influence the feature space, and, consequently, the effectiveness of DAL. The study explores whether DAL performance is sensitive to model structures and investigates the impact of optimizing model architectures together with DAL on model performance gain. Extensive experiments are conducted using CNN and transformer models on both image and text datasets. The results demonstrate that DAL performance is highly dependent on the underlying model architecture. Intriguingly, the findings reveal that architecture search has a greater impact on DAL performance than the choice of acquisition function. This suggests that integrating architecture search with DAL could offer significant benefits for improving model performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Clarification of notations**: Some notations, like $L$ and $U$, are introduced in Section 3.1 but not explained until Section 3.2. It would be helpful to mention that these will be explained later.\n\n**Dataset Impact on Acquisition Function Performance**: From Fig. 3 and Fig. 7, it appears that the relative performance of acquisition functions may vary more depending on the dataset than on the model structure. It might be beneficial to discuss why dataset characteristics might have a larger impact on acquisition function effectiveness compared to model structures.\n\n**Ranking Plot (Optional)**: While not necessary, it could be interesting to plot an average ranking of different acquisition functions with respect to model structures. Additionally, plotting the rank of acquisition functions with respect to datasets may provide further insights into how relative performance is affected by the nature of the dataset. This could add more depth to the comparison and help identify patterns in how acquisition functions behave across various settings.", "broader_impact_concerns": "No broader impact concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers agree that the paper should be accepted and published in TMLR. I share their opinion and am therefore recommending acceptance.
Decision: Accept (poster)