Title: FIRAL: An Active Learning Algorithm for Multinomial Logistic Regression
Venue: NeurIPS 2023
Accepted: yes
Keywords: statistical learning

active learning

logistic regression

regret minimization
Abstract: We investigate theory and algorithms for pool-based active learning for multiclass classification using multinomial logistic regression.  Using finite sample analysis, we prove that the Fisher Information Ratio (FIR)  lower and upper bounds  the excess risk. Based on our theoretical analysis, we propose an active learning algorithm that  employs regret minimization to minimize the FIR. To verify our derived excess risk bounds, we conduct experiments on synthetic datasets. Furthermore, we compare FIRAL with five other methods and found that our scheme  outperforms them: it consistently produces the smallest classification error in the multiclass logistic regression setting, as demonstrated through experiments on MNIST, CIFAR-10, and 50-class ImageNet.

FIRAL: An Active Learning Algorithm for Multinomial Logistic Regression

Abstract

We investigate theory and algorithms for pool-based active learning for multiclass classification using multinomial logistic regression. Using finite sample analysis, we prove that the Fisher Information Ratio (FIR) lower and upper bounds the excess risk. Based on our theoretical analysis, we propose an active learning algorithm that employs regret minimization to minimize the FIR. To verify our derived excess risk bounds, we conduct experiments on synthetic datasets. Furthermore, we compare FIRAL with five other methods and found that our scheme outperforms them: it consistently produces the smallest classification error in the multiclass logistic regression setting, as demonstrated through experiments on MNIST, CIFAR-10, and 50-class ImageNet.

1 Introduction

Active learning is of interest in applications with large pools of unlabeled data for which labeling is expensive. In pool active learning, we're given a set of unlabeled points U , an initial set of labeled points S 0 , and a budget of new points b , our goal is to algorithmically select b new points to label in order to minimize the log-likelihood error over the unlabeled points. Equivalently instead of selecting points directly, we seek to find a probability density function that we can use to sample the b points. Informally (precise formulation in § 2), let x denote a data point and p ( x ) denote the distribution density of unlabeled points. Let q ( x ) be the sampling distribution we will use to select the new b points for labeling. We will choose q ( x ) in order to minimize the generalization error (or excess risk) of the classifier over p ( x ) . Our theory is classifier specific: it assumes multinomial logistic regression with parameters θ . The expectations of the Hessian-with respect to θ -of the classifier loss function over p ( x ) and q ( x ) distributions are denoted by H p and H q respectively. Using finite sample analysis, our first result (Theorem 3 in § 3) is to show that the unlabeled data excess risk is bounded below and above by the Fisher information ratio Trace ( H q -1 H p ) , subject to the assumption of p being a sub-Gaussian distribution. Our second result (Theorem 4 in § 4) is to propose and analyze a point selection algorithm based on regret minimization that allows us to bound the generalization error.

There is a large body of work on various active learning methods based on uncertainty estimation ([1, 2, 3]), sample diversity ([4, 5]), Bayesian inference ( [6, 7]), and many others ([8]). Here we just discuss the papers closest to our scheme. Zhang and Oles [9] claimed without proof that FIR is asymptotically proportional to the log-likelihood error of unlabeled data. Sourati et al. [10] proved that FIR is an upper bound of the expected variance of the asymptotic distribution of the log-likelihood error. Chaudhuri et al. [11] proved non-asymptotic results indicating that FIR is closely related to the expected log-likelihood error of an Maximum Likelihood Estimation-based classifier in bounded domain. In this work, we use finite sample analysis to establish FIR-based bounds for the excess risk in the case of multinomial logistic regression with sub-Gaussian assumption for the point distributions.

Algorithmically finding points to minimize FIR is an NP-hard combinatorial optimization problem. There have been several approximate algorithms proposed for this problem. Hoi et al. [12] studied the binary classification problem and approximated the FIR using a submodular function and then used a greedy optimization algorithm. Chaudhuri et al. [11] proposed an algorithm that first solves a relaxed continuous convex optimization problem, followed by randomly sampling from the weights. Although they derived a performance guarantee for their approach, it needs a substantial number of samples to approach near-optimal performance solely through random sampling from the weights, and no numerical experiments results were provided using such approach. Ash et al. [13] adopted a forward greedy algorithm to initially select an excess of points and then utilized a backward greedy algorithm to remove surplus points. But such approach has no performance guarantee. Hence, there is still a need for computationally efficient algorithms that can optimize FIR in a multi-class classification context while providing provable guarantees.

Our proposed algorithm, FIRAL, offers a locally near-optimal performance guarantee in terms of selecting points to optimize FIR. In our algorithm we have two steps: first we solve a continuous convex relaxation of the original problem in which we find selection weights for all points in U . Then given these weights, we select b points for labeling by a regret minimization approach. This two-step scheme is inspired by Allen-Zhu et al. [14] where a similar approach was used selecting points for linear regression. Extending this approach to active learning for multinomial logistic regression has two main challenges. Firstly, we need to incorporate the information from previously selected points in each new round of active learning. Additionally, while the original approach selects features of individual points, in logistic regression, we need to select a Fisher information matrix ( H q ), which complicates the computation and derivation of theoretical performance guarantees. In Section 4, we present our approach in addressing these challenges.

Our Contributions. ❶ In § 3 we prove that FIR is a lower and upper bound of the excess risk for multinomial logistic regression under sub-Gaussian assumptions. ❷ In § 4 we detail our FIR Active Learning algorithm (FIRAL) and prove it selects b points that lead to a bound to the excess risk. ❸ In § 5 we evaluate our analysis empirically on synthetic and real world datasets: MNIST, CIFAR-10, and ImageNet using a subset of 50 classes. We compare FIRAL with several other methods for pool-based active learning.

2 Problem Formulation

We denote a labeled sample as a pair ( x, y ) , where x ∈ R d is a data point, y ∈ { 1 , 2 , · · · , c } is its label, and c is the number of classes. Let θ ∈ R ( c -1) × d be the parameters of a c -class logistic regression classifier. Given x and θ , the likelihood of the label y is defined by

We use the negative log-likelihood as the loss function: /lscript ( x,y ) ( θ ) /defines -log p ( y | x, θ ) . To simplify notation we define ˜ d = d ( c -1) . We derive standard expressions for the gradient ∇ /lscript ( x,y ) ( θ ) ∈ R ( c -1) × d and Hessian ∇ 2 /lscript ( x,y ) ( θ ) ∈ R ˜ d × ˜ d in the Appendix B.1 (Proposition 23). We assume there exists θ ∗ such that p ( y | x ) = p ( y | x, θ ∗ ) . Then, given p ( x ) , the joint ( x, y ) distribution is given by

Then, the expected loss at θ is defined by

The excess risk of p ( x ) at θ is defined as R p ( θ ) = L p ( θ ) -L p ( θ ∗ ) . Note that R p ( θ ) ≥ 0 .

Notation: The inner product between two matrices is A · B = Trace ( A /latticetop B ) . For a matrix A ∈ R m × n , let ‖ A ‖ be the spectral norm of A , let vec( A ) ∈ R mn be the vectorization of A by stacking all rows together, i.e. vec( A ) = ( A /latticetop 1 , · · · , A /latticetop m ) /latticetop where A i is i -th row of A . Given a positive definite matrix A ∈ R d × d , we define norm ‖ · ‖ A for x ∈ R d by ‖ x ‖ A = √ x /latticetop A x . For

integer k ≥ 1 , we denote by I k the k -byk identity matrix. For any point distribution p ( x ) we define V p /defines E x ∼ p ( x ) [ xx /latticetop ] to be its covariance matrix, H p ( θ ) /defines ∇ 2 L p ( θ ) be the Hessian matrix of L p ( θ ) , define H p /defines H p ( θ ∗ ) .

Active learning. Let U = { x i } i m =1 , be the set of unlabeled points and S 0 be the set of n 0 initially labeled samples. In particular, we denote the set of points in S 0 as X 0 . Let θ 0 be the solution of training a classifier with S 0 , i.e., θ 0 ∈ arg min θ 1 n 0 ∑ ( x,y ) ∈ S 0 /lscript ( x,y ) ( θ ) . We select a set of b points X ⊂ U , query their labels y ∼ p ( y | x, θ ∗ ) , ∀ x ∈ X , and train a new classifier θ n ∈ arg min θ 1 n ∑ ( x,y ) ∈ S /lscript ( x,y ) ( θ ) , where S is the set of S 0 with new labeled points and n = n 0 + b .

Our goal is to optimize the selection of X so that we can minimize the excess risk on the original unlabeled set U , i.e. L p ( θ n ) -L p ( θ ∗ ) . In this context, we define two problems:

Problem 1: Given X or equivalently q ( x ) , can we bound L ( θ ) L ( θ )

Problem 2: Can we construct an efficient algorithm for finding X that minimizes L ( θ ) L ( θ ) ?

3 Excess Risk Bounds

In this section, we develop our theory to address Problem 1. Our plan is to endow p ( x ) and q ( x ) with certain properties (sub-Gaussianity or finite support) and derive FIR bounds for L p ( θ n ) -L p ( θ ∗ ) . Let θ n be the empirical risk minimizer (ERM) obtained from n i.i.d. samples drawn from π q ( x, y ) :

We assume that both p ( x ) and q ( x ) are sub-Gaussian distributions. Appendix A.1 gives a brief review of definitions and basic properties of sub-Gaussian random variables (vector). We define the ψ 2 -norm of a sub-Gaussian random variable x ∈ R as ‖ x ‖ ψ 2 /defines inf { t > 0 : E exp( x 2 /t 2 ) ≤ 2 } . For a sub-Gaussian random vector x ∈ R d , ‖ x ‖ ψ 2 = sup { ‖ u /latticetop x ‖ ψ 2 : ‖ u ‖ 2 ≤ 1 } . We formalize our assumption for p ( x ) and q ( x ) in Assumption 1. Based on this assumption, we can derive some properties for the gradient and Hessian of /lscript ( x,y ) ( θ ) shown inLemma 2 (proof can be found in Appendix D). We present the results for q (thus the subscript in the K constants); exactly the same results, with different constants hold for p .

Assumption 1. Let q ( x ) be a sub-Gaussian distribution for x ∈ R b , we assume that V q is positive definite. We assume that there exists r /greaterorsimilar 1 such that for any θ ∈ B r ( θ ∗ ) = { θ : ‖ θ -θ ∗ ‖ 2 , ∞ ≤ r } , H q ( θ ) is positive definite, where ‖ · ‖ 2 , ∞ denotes the maximum row norm of a matrix.

Lemma 2. If Assumption 1 holds for q ( x ) , then for ( x, y ) ∼ π q ( x, y ) :

where S ˜ d -1 is the unit sphere in R ˜ d , norm ‖ · ‖ ψ 1 is the norm defined for a sub-exponential random variable z ∈ R by ‖ z ‖ ψ 1 = inf { t > 0 : E exp( | z | /t ) ≤ 2 } .

Our main result of this section is Theorem 3. Under the sample bound given by Eq. (6), we derive high probability bounds for the excess risk in Eq. (7). Details and the proof of Theorem 3 can be found in Appendix C.

Theorem 3. Suppose Assumption 1 holds for both p ( x ) and q ( x ) . Let σ and ρ > 0 be constants such that H p /precedesequal σ H q and I c -1 ⊗ V p /precedesequal ρ H p ( θ ∗ ) hold. There exit constants C 1 , C 2 and C 3 > 0 , such that for any δ ∈ (0 , 1) , whenever

p n -p ∗ ? p n -p ∗

where ˜ d /defines d ( c -1) , we have with probability at least 1 -δ ,

Here H p = H p ( θ ∗ ) and H q = H q ( θ ∗ ) ; and E is the expectation over { y i ∼ p ( y i | x i , θ ∗ ) } n i =1 . Furthermore,

From Eq. (7), we observe that FIR ( H q -1 · H p ) appears in both the lower and upper bounds for R ( θ n ) . In other words, it is essential for controlling the excess risk. To the right we show how the prefactors e α -α -1 α 2 and e -α + α -1 α 2 change as a function of α . Constants C 1 , C 2 and C 3 depend on constants defined in Lemma 2 for both p ( x ) and q ( x ) . In Appendix D, we derive bounds for K 1 ,p and K 2 ,p ( r ) in Proposition 35. For a Gaussian design x ∼ N (0 , V p ) , we derive bounds for ρ , K 0 ,p , K 1 ,p and K 2 ,p ( r ) in Proposition 37.

0

2

4

6

8

10

α

10

-

1

10 0

10 1

10 2

(

e

α

-

α

-

1

)

/

α

2

(

e

-

α

+

α

-

1

)

/

α

2

Bounded domain. If the domain of x is bounded, Chaudhuri et al.[11] provided lower and upper bounds for the excess risk of p ( x ) (Lemma 1 in [11]). Their conclusion is similar to ours, namely that FIR is crucial in controlling the excess risk of p ( x ) . It is worth noting that when the domain is bounded, both p ( x ) and q ( x ) are always sub-Gaussian. Thus, our assumption is more general. For the sake of completeness, we provide a detailed derivation of the excess risk bounds for p ( x ) in Theorem 40 when x is bounded with Assumption 38.

4 Active Learning via Minimizing Fisher Information Ratio

We now discuss the FIRAL algorithm that addresses Problem 2. We can use the theoretical analysis derived in the previous section to guide us for the point selection. Let p ( x ) be the empirical distribution on unlabeled pool U with | U | = m , and q the distribution for the n = n 0 + b labeled points. Eq. (7) inspires us to select points to label such that we can minimize the FIR H q -1 · H p , where H q = H q ( θ ∗ ) , H p = H p ( θ ∗ ) . However, we cannot directly use this as θ ∗ is unknown. Instead, we will use θ 0 , the solution by training the classifier with the initial labeled set S 0 . 1 That is, we will find q by minimizing H q ( θ 0 ) -1 · H p ( θ 0 ) .

In § 4.1, we formalize our optimization objective in Eq. (13). Solving Eq. (13) exactly is NP-hard [15]. Inspired by [14], we approximate the solution in two steps: (1) we solve a continuous convex optimization problem in Eq. (14) (§ 4.2), (2) and use the results in a regret minimization algorithm to select points by Eq. (19) (§ 4.3). In Algorithm 1 we summarize the scheme.

We state theoretical guarantees for the algorithm in § 4.4, where we prove that it achieves (1 + /epsilon1 ) -approximation of the optimal objective value in Eq.(13) with sample complexity b = O ( ˜ d/ /epsilon1 2 ) , as stated in Theorem 10. Finally, we obtain the excess risk bound for unlabeled points U by accounting for the fact that we use θ 0 instead of θ ∗ in the objective function. The overall result is summarized in the following theorem.

Theorem 4. Suppose that Assumption 1 holds. Let /epsilon1 ∈ (0 , 1) , δ ∈ (0 , 1) , and b the number of points to label. Then with probability at least 1 -δ , the θ n -computed by fitting a multinomial logistic regression classifier on the labeled points selected using FIRAL (Algorithm 1) with learning rate η = 8 √ ˜ d/ /epsilon1 , b ≥ 32 d/ /epsilon1 2 +16 √ ˜ d/ /epsilon1 2 , and n = n 0 + b satisfying Eq. (6) -results in

Here OPT is the minimal H q -1 · H p , attained by selecting the best b points from U ; E is the expectation over { y i ∼ p ( y i | x i , θ ∗ ) } n i =1 ; and α 0 and α n are constants.

4.1 Optimization objective

First we define the precise expression for H q ( θ 0 ) -1 · H p ( θ 0 ) . Wedefine the Fisher information matrix H ( x, θ ) = ∇ 2 /lscript ( x,y ) ( θ ) . By Eq. (40) in Proposition 23 (Appendix B), we find that for multinomial logistic regression

where ⊗ represents the matrix Kronecker product, h ( x, θ ) is a ( c -1) -dimensional vector whose k -th component is h k ( x, θ ) = p ( y = k | x, θ ) . In Eq. (10) we can see that the Hessian of /lscript ( x,y ) ( θ ) does not depend on the class label y . Following our previous definitions, H p ( θ 0 ) = ∇ 2 L p ( θ 0 ) = 1 m ∑ x ∈ U H ( x, θ 0 ) and H q ( θ 0 ) = ∇ 2 L q ( θ 0 ) = 1 n ∑ x ∈ X 0 ∪ X H ( x, θ 0 ) . For notational simplicity we also define

Then minimizing H q ( θ 0 ) -1 · H p ( θ 0 ) is equivalent to

We define z ∗ be the optimal solution of Eq. (13) and f ∗ /defines f ( z ∗ ) . In the following, with some abuse of notation, we will consider f being a function of either a vector z or a positive semidefinite matrix f ( Σ ) , depending on the context. Lemma 5 lists key properties of f when viewed as a matrix function; we use them in § 4.2 to prove the optimality of FIRAL.

Lemma 5. f : { A ∈ R ˜ d × ˜ d : A /followsequal 0 } → R defined in Eq. (13) satisfies the following properties:

convex: f ( λ A +(1 -λ ) B ) ≤ λ f ( A ) + (1 -λ ) f ( B ) for all λ ∈ [0 , 1] and A , B ∈ S ˜ d ++

reciprocally linear: if t > 0 then f ( t A )) = t -1 f ( A ) .

monotonically non-increasing: if A /precedesequal B then f ( A ) ≥ f ( B ) ,

4.2 Relaxed problem

As a first step in solving Eq. (13) we relax the constraint z ∈ { 0 , 1 } m to z ∈ [0 , 1] m . Then we obtain the following convex programming problem:

Since both the objective function and the constraint set are convex, conventional convex programming algorithm can be used to solve Eq. (14). We choose to use a mirror descent algorithm in our implementation (outlined in the Appendix, Algorithm 2). Since the integrality constraint is a subset of the relaxed constraint we obtain the following result.

Proposition 6. f ( z /diamondmath ) ≤ f ∗ .

In what follows, we use matrices Σ /diamondmath and H ( x i ) ( i ∈ [ m ] ) defined by

4.3 Solving Sparsification problem via Regret Minimization

Goal of sparsification. Now we introduce our method of sparsifying z /diamondmath (optimal solution to Eq. (14)) into a valid integer solution to Eq. (13). To do so, we use an online optimization algorithm in which we select one point at a time in sequence until we have b points. Notice that alternative techniques like thresholding z /diamondmath could be used but it was unclear to us how to provide error estimates for such a scheme. Instead, we use an alternative scheme that we describe below.

Let i t ∈ [ m ] be the point index selected at step t ∈ [ b ] . We can observe that if λ min ( ∑ t ∈ [ b ] ˜ H ( x i t ) ) ≥ τ for some τ > 0 , then ∑ t ∈ [ b ] H ( x i t ) /followsequal τ Σ /diamondmath . By Lemma 5-Item (3) and Proposition 6, we obtain the following result.

Proposition 7. Given τ ∈ (0 , 1) , we have

From Eq. (16), a larger τ value indicates that f is closer to f ∗ . Therefore, our objective is to choose points in such a way that λ min ( ∑ t ∈ [ b ] ˜ H ( x i t ) ) is maximized.

Lower bound minimum eigenvalue via Follow-The-Regularized-Leader (FTRL). We apply FTRL, which is a popular method for online optimization [16], to our problem because it can yield a lower bound for λ min ( ∑ t ∈ [ b ] ˜ H ( x i t ) ) in our setting. FTRL takes b steps to finish. At each step t ∈ [ b ] , for a fixed learning rate η > 0 , we generate a matrix A t defined by

Here ν t is the unique constant such that Trace( A t ) = 1 . Using Eq. (17) we can guarantee a lower bound for λ min ( ∑ s ∈ [ t ] ˜ H ( x i t ) ) , which is formalized below: Proposition 8. Given A l , l ∈ [ b ] , defined by Eq. (17) and for all t ∈ [ b ]

Point selection via maximizing the lower bound in Eq. (18). Now we discuss our choice of point selection at each time step based on Eq. (18). Recall that our sparsification goal is to make λ min ( ∑ t s =1 ˜ H ( x i s ) as large as possible. Since Eq. (18) provides a lower bound for such minimum eigenvalue, we can choose i t ∈ [ m ] to maximize the lower bound, which is equivalent to choose

Solving Eq. (19) directly can become computationally expensive when the dimension d , number of classes c , and the pool size n are large. This is due to the fact that the matrix A -1 / 2 t + η ˜ H ( x i ) ∈ R ˜ d × ˜ d (where ˜ d = d ( c -1) ), requiring n eigendecompositions of a ˜ d × ˜ d matrix to obtain the solution. Fortunately, we can reduce this complexity without losing accuracy . First, by Eq. (10) and Eq. (11), we have for any i ∈ [ m ] ,

where V i Λ i V /latticetop i is the eigendecomposition of diag ( h ( x i , θ 0 )) -h ( x i , θ 0 ) h ( x i , θ 0 ) /latticetop . Define matrix Q i /defines V i Λ 1 / 2 i , then ˜ H ( x i ) = D +( Q i Q /latticetop i ) ⊗ ( x i x /latticetop i ) . Substitute this into Eq. (15), we have a new expression for transformed Fisher information matrix H ( x i ) :

Now define B t ∈ R ˜ d × ˜ d s.t. B -1 / 2 t = A -1 / 2 t + η ˜ D . By Eq. (21), we have A -1 / 2 t + η ˜ H ( x i ) = B -1 / 2 t + η P i P /latticetop i . Applying Woodbury's matrix identity, we have

Algorithm 1 FIRAL( b , U , S 0 , θ 0 )

Input: sample budget b , unlabeled pool U = { x i } i ∈ [ m ] , labeled set S 0 , initial ERM θ 0 Output: selected points X

1:

2:

3:

4:

5:

6:

7:

8:

9:

10:

11:

solution of Eq. (14),

←∅

Λ

i

X

z

/diamondmath

V

i

P

i

D

˜

n

Σ

V

/latticetop

i

=1

z

eigendecomposition of diag

-

i

1

,i

(

∑

(

1

/

2

/

2

Σ

for

/diamondmath

⊗

i

i

x

V

Λ

)

,

defined in Eq. (21),

t

(

= 1

i

t

i

A

∀

1

1

-

/diamondmath

)

to

V

do

b

solution of Eq. (23),

Λ

find

X

X

V

/latticetop

ν

-

t

1

/

A

t

+1

end for

]

d

∈

j

[

+1

I

d

˜

Now our point selection objective Eq. (19) is equivalent to

Since ( I c -1 + η ˜ P /latticetop i B 1 / 2 t ˜ P i ) ∈ R ( c -1) × ( c -1) , solving Eq. (23) is faster than solving Eq. (19). We summarize FIRAL for selecting b points in Algorithm 1.

Connection to regret minimization. Our algorithm is derived as the solution of a regret minimization problem in the adversarial linear bandits setting. We give a brief introduction in Appendix F.3. Readers who are interested in this topic can refer to Part VI of [17]. In our case the action matrix is constrained to { A ∈ R ˜ d × ˜ d : A /followsequal 0 , Trace( A ) = 1 } and is chosen by Eq. (17); the loss matrix is constrained to the set of the transformed Fisher information matrices { ˜ H ( x i ) } i m =1 and is chosen by minimizing Eq. (23).

Algorithm complexity. Our algorithm has two steps: convex relaxation (line 2 in Algorithm 1) and sparsification (lines 3-11). Let T eigen ( ˜ d ) be the complexity of eigendecomposition of a ˜ d -dimensional symmetric positive definite matrix. Given an unlabeled point pool U with m = | U | , the complexity of solving the convex relaxation problem by mirror descent (Algorithm 2) is O ( m ˜ d 2 log m + T eigen ( ˜ d ) log m ) , where we assume that the number of iterations is O (log m ) according to Theorem 42. Given sample budget b , the complexity of solving the sparsification problem is O ( T eigen ( ˜ d ) b + T eigen ( c -1) bm ) .

4.4 Performance guarantee

We intend to lower bound λ min ( ∑ t ∈ [ b ] ˜ H ( x i t ) ) through lower bounding the right hand side of (18). First, since our point selection algorithm selects point x i at each step to maximize Trace[ A 1 / 2 t -( A -1 / 2 t + η ˜ H ( x i )) -1 ] , we establish a lower bound for this term at each step, as demonstrated in Proposition 9.

Proposition 9. At each step t ∈ [ b ] , we have

The derivation is elaborated in Appendix F.4. We remark that there is a similar lower bound derived for the optimal design setting in [14] (Lemma 3.2), where a rank-1 matrix ˜ x i t ˜ x /latticetop i t ( i t ∈ [ m ] and ˜ x i t ∈ R d ) is selected at each step. In contrast, in our active learning setting, the selected matrix ˜ H ( x i t ) possesses a minimum rank of c -1 and can even be a full-rank matrix, contingent upon eigendecomposition of

m

]

d

I

∪

d

∗

h

H

(

,

{

˜

η

t

+1

j

(

+

)

ν

λ

-

(

+

)

/latticetop

,

2

1

B

(

x

i

x

,

1

B

x

1

i

t

}

t

s

= 1

∑

/

2

t

+1

=1

H

(

)

i

θ

0

/

2

))

(

-

x

i

1

˜

A

/

t

2

+1

˜

+1

2

∑

/

2

i

∈

[

˜

V

ν

t

Λ

V

-

h

(

(

-

x

i

1

A

1

s

) =

+

η

/

η

D

˜

,

θ

0

2

+

∑

t

s

)

-

1

# continuous convex relaxation

)

h

∀

∈

0

/latticetop

)

[

]

,

i

m

η

(

x

D

˜

=1

i

)

,

-

(

D

˜

θ

1

+

P

˜

i

s

P

˜

/latticetop

i

s

)

10

1

10

2

H

-

1

q

·

H

p

10

-

3

10

-

2

10

-

1

Lp

(

θ

n

)

-

Lp

(

θ

∗

)

c

= 2

c

= 3

c

= 5

c

= 8

10

1

10

2

H

-

1

q

·

H

p

10

-

4

10

-

3

10

-

2

10

-

1

Lp

(

θ

n

)

-

Lp

(

θ

∗

)

c

= 2

c

= 3

c

= 5

c

= 8

10

1

10

2

H

-

1

q

·

H

p

10

-

3

10

-

2

10

-

1

10

0

10

1

Lp

(

θ

n

)

-

Lp

(

θ

∗

)

c

= 2

c

= 3

c

= 5

c

= 8

10

1

10

2

H

-

1

q

·

H

p

10

-

3

10

-

2

10

-

1

10

0

10

1

Lp

(

θ

n

)

-

Lp

(

θ

∗

)

c

= 2

c

= 3

c

= 5

c

= 8

Dilation

(

:

increasing

ν

q

)

n

=1600

n

=3200

Translation

(

:

decreasing

τ

q

)

n

=1600

n

=3200

the labeled points from prior rounds. The distinction between the characteristics of the matrices significantly complicates the derivation of such a general lower bound.

By connecting the observations obtained in this section, we can show that our algorithm can achieve (1 + /epsilon1 ) -approximation of the optimal objective with sample size O ( ˜ d/ /epsilon1 2 ) . We conclude our results in Theorem 10.

Theorem 10. Given /epsilon1 ∈ (0 , 1) , let η = 8 √ ˜ d/ /epsilon1 , whenever b ≥ 32 ˜ d/ /epsilon1 2 + 16 √ ˜ d/ /epsilon1 2 , denote the instance index selected by Algorithm 1 at step t by i t ∈ [ m ] , then the algorithm is near-optimal: f ( ∑ t ∈ [ b ] H ( x i t ) ) ≤ (1 + /epsilon1 ) f ∗ , where f is the objective function defined in Eq. (13) and f ∗ is its optimal value.

The excess risk upper bound for unlabeled points can be obtained using our algorithm in Theorem 4 by combining Theorem 3 and Theorem 10 while considering the impact of using θ 0 as an approximation for θ ∗ . We present the proof in Appendix F.6. Comparing Eq.(9) to Eq.(7), we observe a factor of 2(1 + /epsilon1 ) e 2 α 0 degradation in the upper bound. The (1 + /epsilon1 ) -term comes from our algorithm, while the 2 e 2 α 0 -term comes from the use of θ 0 instead of θ ∗ . This observation suggests that, given a total budget of points to label b we should consider an iterative approach consisting of r active learning rounds. At each round k we label a new batch of size b/r points and we obtain a new estimate θ k that can be used to approximate θ ∗ . The prefactor containing α 0 will becomes α k and reduces θ k converges to θ ∗ . The simplest solution would be to use r = b but this can be computationally expensive. In our tests, we use this batched approach and choose b/r to be a small multiple of c .

5 Numerical Experiments

Synthetic datasets. We use synthetic datasets to demonstrate the excess risk bounds Eq. (7) derived in Theorem 3. We choose p ( x ) ∼ N ( 0 , V p ) , where V p = 100 I d and d = 8 . We explore different numbers of classes denoted by c ∈ { 2 , 3 , 5 , 8 } . We define the ground truth parameter θ ∗ in such a way that the points generated from p ( x ) are nearly equally distributed across the c classes. In Fig. 4 (Appendix G.1), we plot the first two coordinates of the points draw from p ( x ) , where each point is colored by its class id.

We conduct tests using two different types of q ( x ) based on operations applied to p ( x ) : dilation and translation . For the dilation, q ( x ) ∼ N ( 0 , ν q V p ) , where ν q ∈ R + . We vary ν q within so that FIR ( H q -1 · H p ) is in [0 . 2 ˜ d, 10 ˜ d ] , where ˜ d = d ( c -1) . For translation, q ( x ) ∼ N ( τ q a , V p ) , where a = (1 / √ 2 , 1 / √ 2 , 0 , · · · , 0) and τ q ∈ R + . We examine various τ q values that ensures H q -1 · H p ∈ [ ˜ d, 10 ˜ d ] . For each c ∈ { 2 , 3 , 5 , 8 } , q ( x ) and n ∈ { 1600 , 3200 } , we i.i.d. draw n samples from π q ( x ) and obtain θ n defined by Eq. (4) using these samples. We estimate excess risk L p ( θ n ) -L p ( θ ∗ ) by averaging the log-likelihood error on 5 × 10 4 i.i.d. points sampled from p ( x ) .

Fig. 2 displays the excess risk plotted against FIR for both dilation tests (left two plots) and translation tests (right two plots). It is evident that FIR plays a crucial role in controlling the excess risk. In the case of dilation tests, we observe an almost linear convergence rate with respect to FIR. In the translation tests, we observe a faster-than-linear convergence rate, which can be explained by examining the upper bound of Eq. (7). As FIR decreases, σ also decreases according to the right plot of Fig. 6). By Proposition 37, in our scenario, we have K 1 ,q /lessorsimilar (100 + τ q ) 3 / 4 . In Appendix C,

Random

K-Means

Entropy

BAIT

FIRAL

10

60

110

160

Number of Labeled Samples

70

80

90

100

Classification Accuracy (%)

MNIST

10

20

40

100

240

440

Number of Labeled Samples

70

75

80

85

90

95

CIFAR-10

50

100

200

400

700

1200

2200

Number of Labeled Samples

65

70

75

80

85

90

ImageNet-50

0

2

2

2

3

3

3

3

3

3

6

6

6

7

7

7

7

7

7

7

7

11

11

13

13

13

15

17

18

19

22

22

22

27

29

29

29

29

31

31

35

35

38

38

38

43

43

44

46

48

0

2

3

3

4

7

7

8

9

10

11

12

13

13

14

14

15

15

16

17

17

20

21

22

23

23

24

25

25

26

26

27

31

31

32

32

34

37

38

38

39

42

43

43

44

45

46

47

48

49

BAIT

FIRAL

C

74

it is stated that C 3 = O ( K 0 ,p K 1 ,q K 2 ,p ) . As a result, as FIR decreases, both C 3 and σ decrease, leading to a decrease in α (as indicated by Eq. (63)). Referring to Fig. 1, the prefactor of the FIR term in the upper bound decreases as α decreases. Consequently, the upper bound of Eq. (7) indicates a faster-than-linear convergence rate with respect to the FIR term in the case of translation. We perform similar tests on multivariate Laplace distribution and t-distribution, and the results are consistent with our observations on Gaussian tests. Further details of synthetic experiments are given in Appendix G.1.

Real-world datasets. We demonstrate the effectiveness of our active learning algorithm using three real-world datasets: MNIST [18], CIFAR-10 [19], and ImageNet [20]. In the case of ImageNet, we randomly choose 50 classes for our experiments. First we use unsupervised learning to extract features and then apply active learning to the feature space, that is, we do not use any label information in our pre-processing. For MNIST, we calculate the normalized Laplacian of the training data and use the spectral subspace of the 20 smallest eigenvalues. For CIFAR-10 and ImageNet-50, we use a contrastive learning SimCLR model [21]; then we compute the normalized nearest-neighbor Laplacian and select the subspace of the 20 smallest eigenvalues; For ImageNet-50 we select the subspace of the 40 smallest eigenvalues. For each dataset, we initialize the labeled data S 0 by randomly selecting one sample from each class. Further details about tuning hyperparameter η and data pre-processing are given in Appendix G.2.

We compare our algorithm FIRAL with five methods: (1) Random selection, (2) K-means where k = b , (3) Entropy: select topb points that minimize ∑ c p ( y = c | x ) log p ( y = c | x ) (where c is the class with the highest probability), (4) Var Ratios: select topb points that minimize p ( y = c | x ) (where c is the class with the highest probability), (5) BAIT [13]: solving the same objective as our method, select 2 b points and then delete b points, both in greedy way. Random and K-means are non-deterministic, we run each test 20 times. The other methods are deterministic and the only randomness is related to S 0 . We performed several runs varying S 0 randomly and there is no significant variability in the results, so for clarity we only present one representative run. We present the classification accuracy on U in the upper row of Fig. 3. We can observe that our method consistently outperforms other methods across all experiments. K-means, one of the most popular methods due to each simplicity significantly underperformed FIRAL. It is worth noting that the random selection method serves as a strong baseline in the experiments of ImageNet-50, where our method initially outperforms Random but shows only a marginal improvement in later rounds. But random selection underperforms in CIFAR-10. In the lower row of Fig. 3, we show the images selected in the first round on ImageNet-50 for BAIT and FIRAL. Images selected in other methods

and other datasets can be found in Appendix G.2. One way to qualitatively compare the two methods is to check the diversity of the samples: in the 50-sample example BAIT samples only 21/50 classes; FIRAL samples 37/50 classes. This could explain the significant loss of performance of BAIT in the small sample size regime.

6 Conclusions

We presented FIRAL, a new algorithm designed for the pool-based active learning problem in the context of multinomial logistic regression. We provide the performance guarantee of our algorithm by deriving a excess risk bound for the unlabeled data. We validate the effectiveness of our analysis and algorithm using experiments on synthetic and real-world datasets. The algorithm scales linearly in the size of the pool and cubically on the dimensionality and number of classes-due to the eigenvalue solves. The experiments show clear benefits, especially in terms of robustness of performance across datasets, in the low-sample regime (a few examples per class).

One limitation of our algorithm is the reliance of a hyperparameter, η , derived from the learning rate in the FTRL algorithm. There are a large body of work in online optimization about the adaptive FTRL algorithm (e.g., [16]), which eliminates the need for such hyperparameter. In our future work, we will investigate the integration of adaptive FTRL and evaluate its impact on the overall performance of FIRAL. By exploring this avenue, we aim to enhance the flexibility and efficiency of our algorithm. Another parameter is the number of rounds to use in batch mode, but this we have just set to a small multiple of the number of classes. Other extensions include more complex classifiers and combination with semi-supervised learning techniques.

References

Ajay J. Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos. Multi-class active learning for image classification. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 2372-2379, 2009.

Xin Li and Yuhong Guo. Adaptive active learning for image classification. In 2013 IEEE Conference on Computer Vision and Pattern Recognition , pages 859-866, 2013.

Burr Settles. Active learning literature survey. 2009.

Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489 , 2017.

Daniel Gissin and Shai Shalev-Shwartz. Discriminative active learning. arXiv preprint arXiv:1907.06347 , 2019.

Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International conference on machine learning , pages 1183-1192. PMLR, 2017.

Robert Pinsler, Jonathan Gordon, Eric Nalisnick, and José Miguel Hernández-Lobato. Bayesian batch active learning as sparse subset approximation. Advances in neural information processing systems , 32, 2019.

Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Brij B Gupta, Xiaojiang Chen, and Xin Wang. A survey of deep active learning. ACM computing surveys (CSUR) , 54(9):1-40, 2021.

Tong Zhang and F Oles. The value of unlabeled data for classification problems. In Proceedings of the Seventeenth International Conference on Machine Learning,(Langley, P ., ed.) , volume 20, page 0. Citeseer, 2000.

Jamshid Sourati, Murat Akcakaya, Todd K Leen, Deniz Erdogmus, and Jennifer G Dy. Asymptotic analysis of objectives based on fisher information in active learning. The Journal of Machine Learning Research , 18(1):1123-1163, 2017.

Kamalika Chaudhuri, Sham M Kakade, Praneeth Netrapalli, and Sujay Sanghavi. Convergence rates of active learning for maximum likelihood estimation. Advances in Neural Information Processing Systems , 28, 2015.

Steven CH Hoi, Rong Jin, Jianke Zhu, and Michael R Lyu. Batch mode active learning and its application to medical image classification. In Proceedings of the 23rd international conference on Machine learning , pages 417-424, 2006.

Jordan Ash, Surbhi Goel, Akshay Krishnamurthy, and Sham Kakade. Gone fishing: Neural active learning with fisher embeddings. Advances in Neural Information Processing Systems , 34:8927-8939, 2021.

Zeyuan Allen-Zhu, Yuanzhi Li, Aarti Singh, and Yining Wang. Near-optimal design of experiments via regret minimization. In International Conference on Machine Learning , pages 126-135. PMLR, 2017.

Michal ˇ Cern` y and Milan Hladík. Two complexity results on c-optimality in experimental design. Computational Optimization and Applications , 51(3):1397-1408, 2012.

H Brendan McMahan. A survey of algorithms and analysis for adaptive online learning. The Journal of Machine Learning Research , 18(1):3117-3166, 2017.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine , 29(6):141-142, 2012.

Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research).

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Everest Hinton. A simple framework for contrastive learning of visual representations. 2020.

Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices. 2010.

Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

Dmitrii Ostrovskii and Francis Bach. Finite-sample analysis of m-estimators using selfconcordance, 2018.

Daniel Hsu, Sham Kakade, and Tong Zhang. A tail inequality for quadratic forms of subgaussian random vectors. Electronic Communications in Probability , 17(none):1 - 6, 2012.

Jonas Moritz Kohler and Aurelien Lucchi. Sub-sampled cubic regularization for non-convex optimization, 2017.

Francis Bach. Self-concordant analysis for logistic regression. Electronic Journal of Statistics , 4(none):384 - 414, 2010.

Roy Frostig, Rong Ge, Sham M. Kakade, and Aaron Sidford. Competing with the empirical risk minimizer in a single pass, 2015.

Amir Beck and Marc Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters , 31(3):167-175, 2003.

Axel Ruhe. Perturbation bounds for means of eigenvalues and invariant subspaces. BIT Numerical Mathematics , 10(3):343-354, 1970.

Peer reviews:
comments: SUMMARY:
The authors first prove that the excess risk of multinomial logistic regression with under subgaussian data distribution is lower and upper bounded by terms involving the ratio of the Fisher information of the unlabeled data and the Fisher information of the labeled data.
The authors then propose an algorithm for active learning, based in FIR minimization, and prove a performance guarantee for it.
Experimental results on simulated and real world datasets demonstrate well the performance of the proposed approach.

STRENGTHS:
1. The proposed algorithm stands on a solid mathematical background.
2. The experimental results demonstrate the performance benefits over competitive methods.

WEAKNESSES:
1. The manuscript is very technical and not easy to read.
2. Comparison to competitive methods in terms of computation is missing.

QUESTIONS:
1. Can the authors refer to the capabilities of their algorithm in terms of scale?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies active learning when the underlying data distribution follows the multinomial logistic model. This paper considers the pool-based setting and designs an algorithm to select the sample to query in a batch fashion. There are two main contributions: (1) The paper shows that the excess risk is lower and upper bounded by the Fisher Information Ratio. (2) propose an algorithm to select the samples to optimize the Fisher Information Ratio and thus minimize the expected risk. Theoretical analysis and experiments are conducted to validate the proposed approach.\n\nSTRENGTHS:\nThe strengths of the paper are as follows:\n+ This paper shows the relationship between expected excess risk and the fisher information ratio for the multinomial logistic regression model\n+ This paper proposes an algorithm to maximize the fisher information ratio\n+ The technical part of this paper is well-written and easy to follow\n\nWEAKNESSES:\nThe weaknesses of the paper are as follows:\n\n- unclear comparison with existing work and novelty: my main concern about the paper is the novelty compared with existing literature. \n   -  comparison with [11]: as mentioned by the authors, the non-asymptotic relationship between the FIR and excess risk has been provided in [11] for the generalized linear model. However, I am a little bit confused by the claim in lines 43-44 that \"... the assumption that the loss function is strictly self-concordant (assumption 2 in [11]), which does not hold for the logistic regression case.\" Although the logistic regression loss is not strictly self-concordant, it is a general self-concordant function [27]. Based on the general self-concordance, it seems to me one can also bound the gap between the Hessians of two points as Assumption 2 in [11]. The authors also use the general self-concordant property (Proposition 31) to prove Theorem 3. I think it would be better if the authors could highlight that logistic loss is a general self-concordant function and highlight the novelty compared with [11].\n   - comparison with [14]: it seems to me the method used to optimize the FIR is largely established on [14]. Proposition 7 is similar to a combination of Eq (8) and Proposition 3.1 in [14]. Both papers use the regret analysis of FTRL to show the lower bound in Proposition 8.  I think the paper would have a more significant influence if the author could highlight the main challenge of applying the analysis of [14] to the logistic regression model.\n\n- about assumption in Theorem 3: Theorem 3 relies on the condition that $H_p(\\theta_*)\\leq H_q(\\theta_*)$. It is unclear to me how large this constant would be and how to verify the condition in practice since $\\theta_*$ is generally unknown.\n\n- about the labeled dataset: the algorithm requires a labeled dataset $S_0$ to estimate the unknown parameter $\\theta_*$. However, it is unclear to me how to obtain such a labeled dataset and when it is sufficient to obtain a good approximation of $\\theta_*$ for the following active learning. As shown in Theorem 4, the number of labeled data will affect the coefficient $\\alpha_0$ appearing in the exponential term that can be potentially large. I think it would be nice if the author could provide a more detailed discussion on the effect of the initial labeled data on the proposed algorithm.\n\n- about the constants in Theorem 4: since $\\alpha_0$ and $\\alpha_n$ are related to $n_0$ and $n$, I am not sure they are appropriate to be considered constants in the proposed bound.\n\nQUESTIONS:\n- could you highlight the main technical challenges and contributions of the paper beyond the existing work? (please refer to the first point of weaknesses for more details)\n- when the assumption on the Hessian $H_p$ and $H_q$ required by Theorem 3 is satisfied? How large is the constant $\\sigma$?\n- could you provide more discussion on the influence of the labeled dataset on the proposed method? (please refer to the third point of weaknesses for more details)\n\nLIMITATIONS:\nThere are still some limitations in the paper:\n- realizable assumption: the theoretical analysis is established on the realizable assumptions such that the data distribution follows the logistic regression model. \n- tightness of the excess risk: the tightness of the proposed theoretical bound is not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel active learning algorithm called FIRAL for pool-based active learning in multinomial logistic regression. The paper investigates the theory and algorithms for pool-based active learning and compares FIRAL to other active learning methods in terms of classification error. The authors use finite sample analysis to establish FIRAL-based bounds for the excess risk in the case of multinomial logistic regression with sub-Gaussian assumption for the point distributions. The experiments conducted on various datasets show that FIRAL outperforms other active learning algorithms in terms of classification error.\n\nSTRENGTHS:\nThe equation and proof make sense. Good introduction on this part. \n\nConvinced theory.\n\nWEAKNESSES:\nNot enough introduction on related work.\n\nThe target task is logistic regression classifier. I think it is simple and has been researched for many years. If this method can be applied on more complex classifiers or segmentation, it would be better.\n\nNo ablation study on hyperparameter, such as \\eta.\n\nQUESTIONS:\nIn Line 67, the logistic regression classifier here is very simple, is it possible to apply this method on more complex classifiers, such as ResNet, Transformer, which need more data to train?\n\nWhat is the meaning of \\theta_*? The optimal parameters to predict y?\n\nCould you compare with uncertainty-based active learning method, such as \"Towards better uncertainty sampling: Active learning\nwith multiple views for deep convolutional neural network\"?\n\nCould you do experiments on the whole ImageNet data set, not just ImageNet-50?\n\nLIMITATIONS:\nThe notation is not constant. For example, the subscripts of \\theta_0 and \\theta_n have different meaning.\n\nSome writing errors. Line 17 \"we use the sample the b points\". Line 20 \"choose q in order minimize\"... Please check them.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops a pool-based active learning method for multinomial logistic regression, following a long line of work using the Fisher Information Ratio as a criterion for active set selection. The paper establishes FIR to tightly (within constants) characterize excess risk (Theorem 3), so that this can be used to select a set of points for active learning. This method is guaranteed pretty well (Theorem 4).\n\nSTRENGTHS:\nIt's nice that the evaluations include strong baselines like BAIT; this is a valuable contribution because of the computational efficiency of the proposed method. \\tilde{d} is relatively small (hundreds) in useful regimes, where the proposed method would be quite efficient. I would not be entirely convinced that it performs better than BAIT without the empirical evaluations - but since they are present and look promising, this seems like a useful method. \n\nThe setting of multinomial logistic regression is at once specific enough to be useful here, and general enough to be interesting (could be any discrete label). So the extent to which theory meets practice here is uncommon and laudable - fantastic problem selection, and approach (with the FIR being the \"right\" quantity for other exponential families).\n\nWEAKNESSES:\nThe sparsification problem in (14) in Section 4.3 is where a lot of looseness in the theory creeps in; optimizing the eigenvalue for sparsification is a roundabout way of selecting points. I am a little puzzled by the relaxation of Section 4.3; the natural relaxation of the convex program (13) is a randomized rounding procedure optimizing over z \\in [0,1]^{m} , rather than optimizing over the positive orthant as in (14). I am also curious about the choice of FTRL for sparsification. It works well enough, but initially was quite confusing in the exposition.\n\nIn Figure 2, dashed constant-slope lines would help readability a lot. \n\nThe main theoretical results are stated in an unnecessarily complex way, I believe. \n- The complex dependence on \\alpha is not really necessary to fully write out in the theorem statements. Fig. 1 handles this well, but in order to interpret it in context, you should stress that n needs to be O(\\tilde{d}) for \\alpha to be low and the bounds to be tight. The complex interplay between these variables could overall use better explanation. \n- Theorem 4's use of the \\lesssim makes it difficult to interpret the (1+\\epsilon) prefactor on the right-hand side, and I had to rummage in the appendix and proofs to understand what the result is trying to express.\n\nQUESTIONS:\n- The parameter settings for the real-world experiments are unclear right now (e.g. for \\eta); the included code has \\eta = 200, but is this ok for all datasets?\n- I'd like to see the results with multinomial logistic regression on pretrained frozen embeddings, i.e. \"linear probing.\" The theory here might actually reflect practice closely enough to make the results noteworthy. This type of approach would strengthen the paper a lot, and opportunities to do so with theory in non-toy settings are not so common.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present theory and algorithms for training multinomial logistic regression models in the pool-based active learning setting; how we should choose $b$ extra points to label from a pool of unlabeled ones, so that when we train a model including the newly acquired labeled points the excess risk of the classifier is minimized. The provide with novel asymptotic lower and upper bounds for the excess risk under sub-gaussian assumptions on the point distribution, and they show that excess risk is in $\\Theta$ of the Fisher Information Ratio between the proposal and point distributions. This justifies algorithms which select points based on the minimization of Fisher Information Ratio. They further devise an algorithm for $(1 + \\epsilon)$-optimal minimization of the Fisher Information Ratio objective. The algorithm depends on approximating the oracle predictive model $p(y|x, \\theta^*)$ by a pretrained classifier, a continuous convex relaxation of the NP-hard discrete problem of point selection, and a greedy regret minimization procedure for sparsifying the continuous solutions, while maintaining bounds for the optimality of the final solution. Finally, they perform extensive experimentation on synthetic and image classification benchmarks to demonstrate that 1) their bounds are numerically valid, 2) their method outperforms chosen baselines.\n\nSTRENGTHS:\n1.  They provide asymptotic lower and upper bounds for the excess risk **under sub-gaussian assumptions** on the point distribution, and they show that excess risk is in $\\Theta$ of the Fisher Information Ratio between the proposal and point distributions. This weakens bounded domain assumptions found for similar bounds in the literature.\n2.  The employed sparsification methodology in the provided algorithm seems to be novel in the context of pool-based active learning, while maintaining the ability to have performance guarantees.\n3.  Numerical experiments are extensive and demonstrate that the algorithm outperforms the compared baselines. Even in the larger scale ImageNet-50 case, with pretrained feature extractors, their method provides with some benefits over random sampling.\n\nWEAKNESSES:\n1.   Portion of the effort and presentation goes into reducing the requirement of performing eigendecompositions of a $\\tilde{d} \\times \\tilde{d}$ matrix for each unlabeled point to performing eigendecompositions of $(c - 1) \\times (c - 1)$. However, there is still need to perform $b$ eigendecompositions of  $\\tilde{d} \\times \\tilde{d}$ matrices in order to use the FTRL algorithm. As a result, complexity is still $O(\\tilde{d}^3)$.\n2.   It seems that in more demanding cases (like ImageNet-50 presented in paper), there are diminishing returns from using an active learning setting like FIRAL over randomly sampling from unlabeled samples. This may discourage practitioners from implementing a pool-based active learning approach over random sampling from unlabeled samples, in order to avoid computational complexity issues and implementation overheads. This seems to be significant as the authors chose to further embed features extracted from pretrained networks with SimCLR to an even lower dimensional space using spectral embeddings. Evaluating the proposed algorithm under more practical considerations, it would be interesting to see the baseline of random sampling (and even K-means) from the unlabeled pool using instead the full $D$ dimensional extracted features from the pretrained network (in the cases of CIFAR10 and ImageNet-50); baselines which are certainly computationally possible to execute.\n\nQUESTIONS:\n### Questions\n*  On a more general note, active learning might prove itself more advantageous when unlabeled points U come from a different distribution than the actual distribution $p$ that we want to evaluate the excess risk under. Such example is the setting of training with class-imbalanced sets (or more generally datasets which have been collected under biased processes), which is a common issue with uncurated large-scale datasets collected in-the-wild. Changing the setting slightly, e.g. by assuming access to a small set S of curated class-balanced samples, might incur benchmarks that are even more favorable to using approaches like FIRAL. What are the authors thoughts about such baselines and problems?\n\n### Typos and Edits\n* Line 17: “we use **the** sample the b points” -> “we use to sample these b points”\n* Line 25: “is bounded above and below **by** FIR”. Also, there should be a citation about FIR at that place.\n* Line 36: “sub-Guassian”\n* Line 38: NP-hard problem needs a citation there\n* Line 80: $V_p$ definition has a $q$, which should be $p$\n\nLIMITATIONS:\nSee Weaknesses and Questions.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-31"}