SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lower bound minimum eigenvalue via Follow-The-Regularized-Leader (FTRL). We apply FTRL, which is a popular method for online optimization [16], to our problem because it can yield a lower bound for  $\lambda_{\min}\left(\sum_{t\in [b]}\widetilde{\mathbf{H}}(x_{i_t})\right)$  in our setting. FTRL takes  $b$  steps to finish. At each step  $t\in [b]$ , for a fixed learning rate  $\eta >0$ , we generate a matrix  $\mathbf{A}_t$  defined by
$$
\mathbf {A} _ {1} = \frac {1}{\tilde {d}} \mathbf {I} _ {\tilde {d}}, \quad \mathbf {A} _ {t} = \left(\nu_ {t} \mathbf {I} _ {\tilde {d}} + \eta \sum_ {l = 1} ^ {t - 1} \widetilde {\mathbf {H}} \left(x _ {i _ {l}}\right)\right) ^ {- 2} \quad (t \geq 2). \tag {17}
$$
Here  $\nu_{t}$  is the unique constant such that  $\operatorname{Trace}(\mathbf{A}_t) = 1$ . Using Eq. (17) we can guarantee a lower bound for  $\lambda_{\min}\left(\sum_{s\in [t]}\widetilde{\mathbf{H}}(x_{i_t})\right)$ , which is formalized below:
Proposition 8. Given  $A_{l}$ ,  $l \in [b]$ , defined by Eq. (17) and for all  $t \in [b]$
$$
\lambda_ {\min } \left(\sum_ {l = 1} ^ {t} \widetilde {\mathbf {H}} \left(x _ {i _ {l}}\right)\right) \geq - \frac {2 \sqrt {\widetilde {d}}}{\eta} + \frac {1}{\eta} \sum_ {l = 1} ^ {t} \operatorname {T r a c e} \left[ \mathbf {A} _ {l} ^ {1 / 2} - \left(\mathbf {A} _ {l} ^ {- 1 / 2} + \eta \widetilde {\mathbf {H}} \left(x _ {i _ {l}}\right)\right) ^ {- 1} \right]. \tag {18}
$$
Point selection via maximizing the lower bound in Eq. (18). Now we discuss our choice of point selection at each time step based on Eq. (18). Recall that our sparsification goal is to make  $\lambda_{\min}(\sum_{s=1}^{t} \tilde{\mathbf{H}}(x_{i_s})$  as large as possible. Since Eq. (18) provides a lower bound for such minimum eigenvalue, we can choose  $i_t \in [m]$  to maximize the lower bound, which is equivalent to choose
$$
i _ {t} \in \underset {i \in [ m ]} {\arg \min } \operatorname {T r a c e} \left[ \left(\mathbf {A} _ {t} ^ {- 1 / 2} + \eta \widetilde {\mathbf {H}} \left(x _ {i}\right)\right) ^ {- 1} \right]. \tag {19}
$$
Solving Eq. (19) directly can become computationally expensive when the dimension  $d$ , number of classes  $c$ , and the pool size  $n$  are large. This is due to the fact that the matrix  $\mathbf{A}_t^{-1/2} + \eta \widetilde{\mathbf{H}}(x_i) \in \mathbb{R}^{\widetilde{d} \times \widetilde{d}}$  (where  $\widetilde{d} = d(c - 1)$ ), requiring  $n$  eigendecompositions of a  $\widetilde{d} \times \widetilde{d}$  matrix to obtain the solution. Fortunately, we can reduce this complexity without losing accuracy. First, by Eq. (10) and Eq. (11), we have for any  $i \in [m]$ ,
$$
\mathbf {H} \left(x _ {i}\right) = \underbrace {\frac {1}{b} \sum_ {x \in X _ {0}} \mathbf {H} \left(x , \theta_ {0}\right)} _ {\triangleq \mathbf {D}} + \underbrace {\left[ \operatorname {d i a g} \left(\mathbf {h} \left(x _ {i} , \theta_ {0}\right)\right) - \mathbf {h} \left(x _ {i} , \theta_ {0}\right) \mathbf {h} \left(x _ {i} , \theta_ {0}\right) ^ {\top} \right]} _ {\triangleq \mathbf {V} _ {i} \boldsymbol {\Lambda} _ {i} \mathbf {V} _ {i} ^ {\top}} \otimes \left(x _ {i} x _ {i} ^ {\top}\right), \tag {20}
$$
where  $\mathbf{V}_i\boldsymbol{\Lambda}_i\mathbf{V}_i^\top$  is the eigendecomposition of  $\mathrm{diag}(\mathbf{h}(x_i,\theta_0)) - \mathbf{h}(x_i,\theta_0)\mathbf{h}(x_i,\theta_0)^\top$ . Define matrix  $\mathbf{Q}_i \triangleq \mathbf{V}_i\boldsymbol{\Lambda}_i^{1/2}$ , then  $\widetilde{\mathbf{H}}(x_i) = \mathbf{D} + (\mathbf{Q}_i\mathbf{Q}_i^\top) \otimes (x_i x_i^\top)$ . Substitute this into Eq. (15), we have a new expression for transformed Fisher information matrix  $\widetilde{\mathbf{H}}(x_i)$ :
$$
\widetilde {\mathbf {H}} (x _ {i}) = \underbrace {\left(\boldsymbol {\Sigma} _ {\diamond}\right) ^ {- 1 / 2} \mathbf {D} \left(\boldsymbol {\Sigma} _ {\diamond}\right) ^ {- 1 / 2}} _ {\triangleq \widetilde {\mathbf {D}}} + \underbrace {\left(\boldsymbol {\Sigma} _ {\diamond}\right) ^ {- 1 / 2} \left(\mathbf {Q} _ {i} \otimes x _ {i}\right)} _ {\triangleq \widetilde {\mathbf {P}} _ {i}} \left(\mathbf {Q} _ {i} \otimes x _ {i}\right) ^ {\top} \left(\boldsymbol {\Sigma} _ {\diamond}\right) ^ {- 1 / 2} = \widetilde {\mathbf {D}} + \widetilde {\mathbf {P}} _ {i} \widetilde {\mathbf {P}} _ {i} ^ {\top}. \tag {21}
$$
Now define  $\mathbf{B}_t \in \mathbb{R}^{\widetilde{d} \times \widetilde{d}}$  s.t.  $\mathbf{B}_t^{-1/2} = \mathbf{A}_t^{-1/2} + \eta \widetilde{\mathbf{D}}$ . By Eq. (21), we have  $\mathbf{A}_t^{-1/2} + \eta \widetilde{\mathbf{H}}(x_i) = \mathbf{B}_t^{-1/2} + \eta \widetilde{\mathbf{P}}_i \widetilde{\mathbf{P}}_i^\top$ . Applying Woodbury's matrix identity, we have
$$
\left(\mathbf {A} _ {t} ^ {- 1 / 2} + \eta \widetilde {\mathbf {H}} \left(x _ {i}\right)\right) ^ {- 1} = \mathbf {B} _ {t} ^ {1 / 2} - \eta \mathbf {B} _ {t} ^ {1 / 2} \widetilde {\mathbf {P}} _ {i} \left(\mathbf {I} _ {c - 1} + \eta \widetilde {\mathbf {P}} _ {i} ^ {\top} \mathbf {B} _ {t} ^ {1 / 2} \widetilde {\mathbf {P}} _ {i}\right) ^ {- 1} \widetilde {\mathbf {P}} _ {i} ^ {\top} \mathbf {B} _ {t} ^ {1 / 2}. \tag {22}
$$
Algorithm 1 FIRAL(b, U, S0, θ0)
Input: sample budget  $b$ , unlabeled pool  $U = \{x_{i}\}_{i\in [m]}$ , labeled set  $S_{0}$ , initial ERM  $\theta_0$   
Output: selected points  $X$   
1:  $X \gets \emptyset$   
2:  $z_{\diamond} \gets$  solution of Eq. (14),  $\Sigma_{\diamond} \gets \sum_{i=1}^{n} z_{*,i} \mathbf{H}(x_{i})$   
3:  $\mathbf{V}_i \boldsymbol{\Lambda}_i \mathbf{V}_i^\top \gets$  eigendecomposition of diag  $(\mathbf{h}(x_i, \theta_0)) - \mathbf{h}(x_i, \theta_0) \mathbf{h}(x_i, \theta_0)^\top, \forall i \in [m]$   
4:  $\widetilde{\mathbf{P}}_i \gets \Sigma_{\diamond}^{-1/2} (x_i \otimes (\mathbf{V}_i \boldsymbol{\Lambda}_i^{1/2}))$ ,  $\forall i \in [m]$   
5:  $\widetilde{\mathbf{D}} \gets$  defined in Eq. (21),  $\mathbf{A}_1^{-1/2} \gets \sqrt{d} \mathbf{I}_{\widetilde{d}}$ ,  $\mathbf{B}_1^{1/2} \gets (\mathbf{A}_1^{-1/2} + \eta \widetilde{\mathbf{D}})^{-1}$   
6: for  $t = 1$  to  $b$  do  
7:  $i_t \gets$  solution of Eq. (23),  $X \gets X \cup \{x_{i_t}\}$   
8:  $\mathbf{V} \boldsymbol{\Lambda} \mathbf{V}^\top \gets$  eigendecomposition of  $\eta \sum_{s=1}^{t} \widetilde{\mathbf{H}}(x_{i_s}) = \eta \sum_{s=1}^{t} (\widetilde{\mathbf{D}} + \widetilde{\mathbf{P}}_{i_s} \widetilde{\mathbf{P}}_{i_s}^\top)$   
9: find  $\nu_{t+1}$  s.t.  $\sum_{j \in [\widetilde{d}]} (\nu_{t+1} + \lambda_j)^{-2} = 1$   
10:  $\mathbf{A}_{t+1}^{-1/2} \gets \mathbf{V}(\nu_{t+1} \mathbf{I}_{\widetilde{d}} + \boldsymbol{\Lambda}) \mathbf{V}^\top$ ,  $\mathbf{B}_{t+1}^{1/2} \gets (\mathbf{A}_{t+1}^{-1/2} + \eta \widetilde{\mathbf{D}})^{-1}$   
11: end for
Now our point selection objective Eq. (19) is equivalent to
$$
i _ {t} \leftarrow \underset {i \in [ m ]} {\arg \max } \left(\mathbf {I} _ {c - 1} + \eta \widetilde {\mathbf {P}} _ {i} ^ {\top} \mathbf {B} _ {t} ^ {1 / 2} \widetilde {\mathbf {P}} _ {i}\right) ^ {- 1} \cdot \widetilde {\mathbf {P}} _ {i} ^ {\top} \mathbf {B} _ {t} \widetilde {\mathbf {P}} _ {i}. \tag {23}
$$
Since  $(\mathbf{I}_{c - 1} + \eta \widetilde{\mathbf{P}}_i^\top \mathbf{B}_t^{1 / 2}\widetilde{\mathbf{P}}_i)\in \mathbb{R}^{(c - 1)\times (c - 1)}$ , solving Eq. (23) is faster than solving Eq. (19). We summarize FIRAL for selecting  $b$  points in Algorithm 1.
Connection to regret minimization. Our algorithm is derived as the solution of a regret minimization problem in the adversarial linear bandits setting. We give a brief introduction in Appendix F.3. Readers who are interested in this topic can refer to Part VI of [17]. In our case the action matrix is constrained to  $\{\mathbf{A} \in \mathbb{R}^{\widetilde{d} \times \widetilde{d}} : \mathbf{A} \succeq 0, \operatorname{Trace}(\mathbf{A}) = 1\}$  and is chosen by Eq. (17); the loss matrix is constrained to the set of the transformed Fisher information matrices  $\{\widetilde{\mathbf{H}}(x_i)\}_{i=1}^m$  and is chosen by minimizing Eq. (23).
Algorithm complexity. Our algorithm has two steps: convex relaxation (line 2 in Algorithm 1) and sparsification (lines 3-11). Let  $T_{\mathrm{eigen}}(\widetilde{d})$  be the complexity of eigendecomposition of a  $d$ -dimensional symmetric positive definite matrix. Given an unlabeled point pool  $U$  with  $m = |U|$ , the complexity of solving the convex relaxation problem by mirror descent (Algorithm 2) is  $\mathcal{O}\big(m\widetilde{d}^2\log m + T_{\mathrm{eigen}}(\widetilde{d})\log m\big)$ , where we assume that the number of iterations is  $\mathcal{O}(\log m)$  according to Theorem 42. Given sample budget  $b$ , the complexity of solving the sparsification problem is  $\mathcal{O}\big(T_{\mathrm{eigen}}(\widetilde{d})b + T_{\mathrm{eigen}}(c - 1)bm\big)$ .
4.4 Performance guarantee
We intend to lower bound  $\lambda_{\mathrm{min}}\Big(\sum_{t\in [b]}\widetilde{\mathbf{H}} (x_{i_t})\Big)$  through lower bounding the right hand side of (18). First, since our point selection algorithm selects point  $x_{i}$  at each step to maximize Trace  $[\mathbf{A}_t^{1 / 2} - (\mathbf{A}_t^{-1 / 2} + \eta \widetilde{\mathbf{H}} (x_i))^{-1}]$ , we establish a lower bound for this term at each step, as demonstrated in Proposition 9.
Proposition 9. At each step  $t \in [b]$ , we have
$$
\max  _ {i \in [ m ]} \frac {1}{\eta} \operatorname {T r a c e} \left[ \mathbf {A} _ {t} ^ {1 / 2} - \left(\mathbf {A} _ {t} ^ {- 1 / 2} + \eta \widetilde {\mathbf {H}} \left(x _ {i}\right)\right) ^ {- 1} \right] \geq \frac {1 - \frac {\eta}{2 b}}{b + \eta \sqrt {\tilde {d}}}. \tag {24}
$$
The derivation is elaborated in Appendix F.4. We remark that there is a similar lower bound derived for the optimal design setting in [14] (Lemma 3.2), where a rank-1 matrix  $\tilde{x}_{i_t}\tilde{x}_{i_t}^\top$  ( $i_t\in [m]$  and  $\tilde{x}_{i_t}\in \mathbb{R}^d$ ) is selected at each step. In contrast, in our active learning setting, the selected matrix  $\widetilde{\mathbf{H}} (x_{i_t})$  possesses a minimum rank of  $c - 1$  and can even be a full-rank matrix, contingent upon
Figure 2: Synthetic experiments: excess risk of  $p(x)$  as a function of the FIR  $(\mathbf{H}_q^{-1} \cdot \mathbf{H}_p)$  in dilation and translation tests.
the labeled points from prior rounds. The distinction between the characteristics of the matrices significantly complicates the derivation of such a general lower bound.
By connecting the observations obtained in this section, we can show that our algorithm can achieve  $(1 + \epsilon)$ -approximation of the optimal objective with sample size  $\mathcal{O}(\widetilde{d}/\epsilon^2)$ . We conclude our results in Theorem [10].
Theorem 10. Given  $\epsilon \in (0,1)$ , let  $\eta = 8\sqrt{\tilde{d}}/\epsilon$ , whenever  $b \geq 32\tilde{d}/\epsilon^2 + 16\sqrt{\tilde{d}}/\epsilon^2$ , denote the instance index selected by Algorithm [1] at step  $t$  by  $i_t \in [m]$ , then the algorithm is near-optimal:  $f\left(\sum_{t \in [b]} \mathbf{H}(x_{i_t})\right) \leq (1 + \epsilon)f_*$ , where  $f$  is the objective function defined in Eq. (13) and  $f_*$  is its optimal value.
The excess risk upper bound for unlabeled points can be obtained using our algorithm in Theorem 4 by combining Theorem 3 and Theorem 10 while considering the impact of using  $\theta_0$  as an approximation for  $\theta_*$ . We present the proof in Appendix F.6. Comparing Eq. (9) to Eq. (7), we observe a factor of  $2(1 + \epsilon)e^{2\alpha_0}$  degradation in the upper bound. The  $(1 + \epsilon)$ -term comes from our algorithm, while the  $2e^{2\alpha_0}$ -term comes from the use of  $\theta_0$  instead of  $\theta_*$ . This observation suggests that, given a total budget of points to label  $b$  we should consider an iterative approach consisting of  $r$  active learning rounds. At each round  $k$  we label a new batch of size  $b/r$  points and we obtain a new estimate  $\theta_k$  that can be used to approximate  $\theta_*$ . The prefactor containing  $\alpha_0$  will become  $\alpha_k$  and reduces  $\theta_k$  converges to  $\theta_*$ . The simplest solution would be to use  $r = b$  but this can be computationally expensive. In our tests, we use this batched approach and choose  $b/r$  to be a small multiple of  $c$ .
Synthetic datasets. We use synthetic datasets to demonstrate the excess risk bounds Eq. (7) derived in Theorem 3. We choose  $p(x) \sim \mathcal{N}(\mathbf{0}, \mathbf{V}_p)$ , where  $\mathbf{V}_p = 100\mathbf{I}_d$  and  $d = 8$ . We explore different numbers of classes denoted by  $c \in \{2, 3, 5, 8\}$ . We define the ground truth parameter  $\theta_*$  in such a way that the points generated from  $p(x)$  are nearly equally distributed across the  $c$  classes. In Fig. 4 (Appendix G.1), we plot the first two coordinates of the points draw from  $p(x)$ , where each point is colored by its class id.
We conduct tests using two different types of  $q(x)$  based on operations applied to  $p(x)$ : dilation and translation. For the dilation,  $q(x) \sim \mathcal{N}(\mathbf{0}, \nu_q\mathbf{V}_p)$ , where  $\nu_{q} \in \mathbb{R}^{+}$ . We vary  $\nu_{q}$  within so that FIR  $(\mathbf{H}_q^{-1} \cdot \mathbf{H}_p)$  is in  $[0.2\widetilde{d}, 10\widetilde{d}]$ , where  $\widetilde{d} = d(c - 1)$ . For translation,  $q(x) \sim \mathcal{N}(\tau_q\mathbf{a}, \mathbf{V}_p)$ , where  $\mathbf{a} = (1/\sqrt{2}, 1/\sqrt{2}, 0, \dots, 0)$  and  $\tau_{q} \in \mathbb{R}^{+}$ . We examine various  $\tau_{q}$  values that ensures  $\mathbf{H}_q^{-1} \cdot \mathbf{H}_p \in [\widetilde{d}, 10\widetilde{d}]$ . For each  $c \in \{2, 3, 5, 8\}$ ,  $q(x)$  and  $n \in \{1600, 3200\}$ , we i.i.d. draw  $n$  samples from  $\pi_q(x)$  and obtain  $\theta_n$  defined by Eq. (4) using these samples. We estimate excess risk  $L_{p}(\theta_{n}) - L_{p}(\theta_{*})$  by averaging the log-likelihood error on  $5 \times 10^{4}$  i.i.d. points sampled from  $p(x)$ .
Fig. 2 displays the excess risk plotted against FIR for both dilation tests (left two plots) and translation tests (right two plots). It is evident that FIR plays a crucial role in controlling the excess risk. In the case of dilation tests, we observe an almost linear convergence rate with respect to FIR. In the translation tests, we observe a faster-than-linear convergence rate, which can be explained by examining the upper bound of Eq. (7). As FIR decreases,  $\sigma$  also decreases according to the right plot of Fig. 6. By Proposition 37, in our scenario, we have  $K_{1,q} \lesssim (100 + \tau_q)^{3/4}$ . In Appendix C
Figure 3: Active learning results for MNIST (left), CIFAR-10 (center) and ImageNet-50 (right). Black dashed lines in the upper row plots are the classification accuracy using all points in  $U$  and their labels. The lower row shows 50 images that are selected in the first round of the active learning process for the ImageNet-50 dataset.
it is stated that  $C_3 = \mathcal{O}(K_{0,p}K_{1,q}K_{2,p})$ . As a result, as FIR decreases, both  $C_3$  and  $\sigma$  decrease, leading to a decrease in  $\alpha$  (as indicated by Eq. (63)). Referring to Fig. [1], the prefactor of the FIR term in the upper bound decreases as  $\alpha$  decreases. Consequently, the upper bound of Eq. [7] indicates a faster-than-linear convergence rate with respect to the FIR term in the case of translation. We perform similar tests on multivariate Laplace distribution and t-distribution, and the results are consistent with our observations on Gaussian tests. Further details of synthetic experiments are given in Appendix [G.1].
Real-world datasets. We demonstrate the effectiveness of our active learning algorithm using three real-world datasets: MNIST [18], CIFAR-10 [19], and ImageNet [20]. In the case of ImageNet, we randomly choose 50 classes for our experiments. First we use unsupervised learning to extract features and then apply active learning to the feature space, that is, we do not use any label information in our pre-processing. For MNIST, we calculate the normalized Laplacian of the training data and use the spectral subspace of the 20 smallest eigenvalues. For CIFAR-10 and ImageNet-50, we use a contrastive learning SimCLR model [21]; then we compute the normalized nearest-neighbor Laplacian and select the subspace of the 20 smallest eigenvalues; For ImageNet-50 we select the subspace of the 40 smallest eigenvalues. For each dataset, we initialize the labeled data  $S_0$  by randomly selecting one sample from each class. Further details about tuning hyperparameter  $\eta$  and data pre-processing are given in Appendix G.2
We compare our algorithm FIRAL with five methods: (1) Random selection, (2) K-means where  $k = b$ , (3) Entropy: select top- $b$  points that minimize  $\sum_{c} p(y = c|x) \log p(y = c|x)$  (where  $c$  is the class with the highest probability), (4) Var Ratios: select top- $b$  points that minimize  $p(y = c|x)$  (where  $c$  is the class with the highest probability), (5) BAIT [13]: solving the same objective as our method, select  $2b$  points and then delete  $b$  points, both in greedy way. Random and K-means are non-deterministic, we run each test 20 times. The other methods are deterministic and the only randomness is related to  $S_0$ . We performed several runs varying  $S_0$  randomly and there is no significant variability in the results, so for clarity we only present one representative run. We present the classification accuracy on  $U$  in the upper row of Fig. 3. We can observe that our method consistently outperforms other methods across all experiments. K-means, one of the most popular methods due to each simplicity significantly underperformed FIRAL. It is worth noting that the random selection method serves as a strong baseline in the experiments of ImageNet-50, where our method initially outperforms Random but shows only a marginal improvement in later rounds. But random selection underperforms in CIFAR-10. In the lower row of Fig. 3, we show the images selected in the first round on ImageNet-50 for BAIT and FIRAL. Images selected in other methods
and other datasets can be found in Appendix G.2. One way to qualitatively compare the two methods is to check the diversity of the samples: in the 50-sample example BAIT samples only 21/50 classes; FIRAL samples 37/50 classes. This could explain the significant loss of performance of BAIT in the small sample size regime.
We presented FIRAL, a new algorithm designed for the pool-based active learning problem in the context of multinomial logistic regression. We provide the performance guarantee of our algorithm by deriving a excess risk bound for the unlabeled data. We validate the effectiveness of our analysis and algorithm using experiments on synthetic and real-world datasets. The algorithm scales linearly in the size of the pool and cubically on the dimensionality and number of classes—due to the eigenvalue solves. The experiments show clear benefits, especially in terms of robustness of performance across datasets, in the low-sample regime (a few examples per class).
One limitation of our algorithm is the reliance of a hyperparameter,  $\eta$ , derived from the learning rate in the FTRL algorithm. There are a large body of work in online optimization about the adaptive FTRL algorithm (e.g., [16]), which eliminates the need for such hyperparameter. In our future work, we will investigate the integration of adaptive FTRL and evaluate its impact on the overall performance of FIRAL. By exploring this avenue, we aim to enhance the flexibility and efficiency of our algorithm. Another parameter is the number of rounds to use in batch mode, but this we have just set to a small multiple of the number of classes. Other extensions include more complex classifiers and combination with semi-supervised learning techniques.
This material is based upon work supported by NSF award OAC 2204226; by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, Applied Mathematics program, Mathematical Multifaceted Integrated Capability Centers (MMICCS) program, under award number DE-SC0023171; and by the U.S. National Institute on Aging under award number R21AG074276-01. Any opinions, findings, and conclusions or recommendations expressed herein are those of the authors and do not necessarily reflect the views of the DOE, NIH, and NSF. Computing time on the Texas Advanced Computing Centers Stampede system was provided by an allocation from TACC and the NSF.
[1] Ajay J. Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos. Multi-class active learning for image classification. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 2372-2379, 2009.
[2] Xin Li and Yuhong Guo. Adaptive active learning for image classification. In 2013 IEEE Conference on Computer Vision and Pattern Recognition, pages 859-866, 2013.
[3] Burr Settles. Active learning literature survey. 2009.
[4] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489, 2017.
[5] Daniel Gissin and Shai Shalev-Shwartz. Discriminative active learning. arXiv preprint arXiv:1907.06347, 2019.
[6] Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International conference on machine learning, pages 1183-1192. PMLR, 2017.
[7] Robert Pinsler, Jonathan Gordon, Eric Nalisnick, and José Miguel Hernández-Lobato. Bayesian batch active learning as sparse subset approximation. Advances in neural information processing systems, 32, 2019.
[28] Roy Frostig, Rong Ge, Sham M. Kakade, and Aaron Sidford. Competing with the empirical risk minimizer in a single pass, 2015.
[29] Amir Beck and Marc Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31(3):167-175, 2003.
[30] Axel Ruhe. Perturbation bounds for means of eigenvalues and invariant subspaces. BIT Numerical Mathematics, 10(3):343-354, 1970.