GAGA: Deciphering Age-path of Generalized Self-paced Regularizer
Authors: Xingyu Qu
Diyang Li
Xiaohan Zhao
Bin Gu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Self-paced Learning
Solution Path
Biconvex Optimization
Partial Optimum

Abstract: Nowadays self-paced learning (SPL) is an important machine learning paradigm that mimics the cognitive process of humans and animals. The SPL regime involves a self-paced regularizer and a gradually increasing age parameter, which plays a key role in SPL but where to optimally terminate this process is still non-trivial to determine. A natural idea is to compute the solution path w.r.t. age parameter (i.e., age-path). However, current age-path algorithms are either limited to the simplest regularizer, or lack solid theoretical understanding as well as computational efficiency. To address this challenge, we propose a novel Generalized Age-path Algorithm (GAGA) for SPL with various self-paced regularizers based on ordinary differential equations (ODEs) and sets control, which can learn the entire solution spectrum w.r.t. a range of age parameters. To the best of our knowledge, GAGA is the first exact path-following algorithm tackling the age-path for general self-paced regularizer. Finally the algorithmic steps of classic SVM and Lasso are described in detail. We demonstrate the performance of GAGA on real-world datasets, and find considerable speedup between our algorithm and competing baselines. 

GAGA: Deciphering Age-path of Generalized Self-paced Regularizer
Xingyu Qu Diyang Li Xiaohan Zhao² Bin 1.2t
Mohamed bin Zayed University of Artificial Intelligence
2 Nanjing University of Information Science & Technology
{Xingyu. Qu, bin. Baaa.ggu@mmern gu)ambzai. ae, Di Diyounglee@gmail.c com, xiaohan. 4aaao4200xxmaii com
Abstract
Nowadays self-paced learning (SPL) is an important machine learning paradigm that mimics the cognitive process of humans and animals. The regime involves a self-paced regularizer and a gradually increasing age parameter, which plays a key role in SPL. but where to optimally terminate this process is still non-trivial to determine. A natural idea is to compute the solution path w.r.t. age parameter (i.e., age-path). However, current age-path algorithms are either limited to the sim- plest regularizer, or lack solid theoretical understanding well as computational efficiency. To address this challenge, we propose a novel Generalized Age-path Algorithm (GAGA) for SPL with various self-paces regularizers based on ordinary differential equations (ODEs) and sets control, which can learn the entire solution spectrum w.r.t. a range of age parameters. To the best of our knowledge, GAGA is the first exact path-following algorithm tackling the age-path for general self- self-paced regularizer. Finally the algorithmic steps of classic SVM and Lasso described in detail. We demonstrate the performance of GAGA on real-world datasets, and find considerable speedup between our algorithm and competing baselines.
Introduction
The SPL. Self-paced learning (SPL) D is classical learning paradigm and has attracted increasing attention in the communities of machine learning 2 4 6) data mining [7] and computer vision 10 [T]. The philosophy under this paradigm is simulating the strategy that how human-beings learn new knowledge. In other words, SPL starts learning from easy tasks and gradually levels up the difficulty while training samples are fed to the model sequentially.
its core, SPL can be viewed as an automatic variant of curriculum learning (CL) 122 [3], which uses prior knowledge to discriminate between simple instances and hard ones along the training process. Different from CL. the SPL assigns a real-valued "easiness weight" to each sample implicitly by adding self- self-paced regularizer (SP-regularizer briefly) to the primal learning problem and optimizes the original model parameters as well as these weights. Considering this setting, the SPL is reported to alleviate the problem of getting stuck bad local minima and provides better generalization as well as robustness for the models, especially in hard condition of heavy noises a high outlier rate 14
There are critical aspects in SPL. namely the SP-regularizer and a gradually increasing age parameter. Different SP-regularizers can be designed for different kinds of training tasks. At
the primary stage of SPI., only the hard SP-regularizer is utilized and leads binary variable for weighting samples I Going with the advancing of the diverse SP-regularizers [15) SPL
"These authors contributed equally. Corresponndig Author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

equipped with different types of SP-reuularizers has been successfully applied to various applications [6 177. As for the age parameter (a.k.a pace parameter), the users are expected increase its value continually under the SPL. paradigm, given that the age parameter represents the maturity of current model. lot of empirical practices have turned out that seeking out an appropriate age parameter is crucial to the SPL procedure [18]. The SPL tends to obtain a worse performance in the presence of noisy samples/outliers when the age parameter gets larger, or conversely, an insufficient age parameter makes the gained model immature (i.e. underfitting, See Figure I).
On the age-path of SPL. Although the SPL a classical and widespread learning paradigm, when to stop the increasing process of age parameter in implementation is subject to surprisingly few theoretical studies. In the majority of practices [1], the choice of the optimal model age has, for the time being, remained restricted to be made by experience or by using the trial-and-error approach, which is to adopt the alternate convex search (ACS) (00 multiple times at a predefined sequence of age parameters. This operation is time-consuming and could miss some significant events along the way of age parameter. In addition, the SPL regime is a successive training process, which makes existing hyperparameter tuning algorithms like parallelizing sequential search [21] and bilevel optimization [22] difficult to apply. Instead of training multiple subproblems at different age parameters, natural idea is to calculate the solution path about age parameter, namely age-path (e.g., see Figure 2). A solution path is a set of curves that demonstrate how the optimal solution of given optimization problem changes w.r.t. a hyperparameter. Several papers like 23 24 laid
the foundation of solution path algorithm in machine learning by demonstrating the rationale of path tracking, which is mainly built on the solution path algorithms involve generalized Lasso 226] semi-supervised support vector classification 27), general parametric quadratic programming (28), etc. However, none of the existing methods is
(KKT) theorem 255 Existing
available to SPL regime because they are limited to uni-convex optimization while the SPL objective a biconvex formulation. Assume we've got such an age-path, we can observe the whole self- self-paced evolution process clearly and recover useful intrinsic patterns from it.
31 is devoted to developing better algorithms for solving the SPL optimization with ideas similar to age-path. However, despite countless theoretical and empirical efforts, the understanding of age-path remains rather deficient. Based on techniques from incremental learning (32). (3 derived an exact age-path algorithm for mere hard -P-reuularizer, where the path remains piecewise constant. 229 30
State of the art. Yet, a rapidly growing literature 22 19
30
proposed a multi-objective self-paced learning (MOSPL) method to approximate the age-path by evolutionary algorithm. which not theoretically stable. Unlike previous studies, the difficulty of revealing the exact generalized age-path lies in the continuance of imposed weight and the alternate optimization procedure used to solve the minimization function. From this point of view, the technical difficulty inherent in the study of age- age-path with general SP-reuularizer is intrinsically more challenging.
Proposed Method. In order to tackle this issue, we establish novel Generalized Age-path Algorithm (GAGA) for various self-paced regularizers, which prevents a straightforward calcula- tion of every age parameter. Our analysis is based on the theorem of partial optimum while previous theoretical results are focused on the implicit SPL objective. In particular, we enhance the original objective to a vangle-variable analysis problem, and use different sets to partition samples and func- tions by their confidence level and differentiabiility. Afterward, we conduct our main theorem results based on the technique of ordinary differential equations (ODEs). In the process, the solution path hits, exits, and slides along the various constraint boundaries. The path itself is piecewise smooth with kinks at the times of boundary hitting and escaping. Moreover, from this perspective we are able to explain some shortcomings of conventional SPL practices and point out how we can improve them. We believe that the proposed method may be of independent interest beyond the particular problem studied here and might be adapted to similar biconvex schemes.
Contributions.
Therefore, the main contributions brought by this work are listed as follows.
We firstly connect SPL paradigm to the concept of partial optimum and emphasize its importance here that has been ignored before, which gives novel viewpoint to the robustness of SPI.. Theoretical studies are conducted to reveal that our result does exist some equivalence with previous literature, which makes our study more stable.
2

---

Generalization Error
0.24
GAGA
Solutions
Optimal model
High
0.22
0.20
-30
230
250
300
250
400
10-1
10a
I0a
Learning curve against age The curve is recorded when running linear regression on music dataset.
Figure 1:
Figure 2: An age-path visuali- sation. Different vertical lines represent different types of crit- ical points. The figure is plotted on random 60% features from diabetes dataset using Lasso with a = 0.01.
Figure 3: An example of set partition in 2-D space. Sample points of same colors belong to one set. The two dashed lines represent partition boundaries (smooth surfaces), which satis- fies 4 X.
A framework of computing the exact age-path for generalized SP-regularizer is derived using the technique of ODEs, which allows for the time-consuming ACS to be avoided. Concrete algorithmic steps of classic SVM 33 and Lasso (34] are given for implementation.
Simulations on real and synthetic data are provided to validate our theoretical findings and justify their impact on the designing future SPL algorithms of practical interest.
Notations, We write matrices in uppercase (e.g., X) and vectors in lowercase with bold font (e.g.. x). Given the index & (or D). Xg (or X&D) denotes the submatrix that taking rows with indexes in & (or rows/columns with indexes in E/D, respectively). Similarly notations lie on ve for vector v. (z) for vector functions ((x). For set of scalar functions {4(x)) we denote the vector function 2(x) where ((x) = (=)) without statement and vice versa. Moreover, we defer the full proofs as well as the algorithmic steps on applications the Appendix.
2 Preliminaries
2.1 Self-paced Learning
Suppose we have dataset containing the label vector y E IR" and X E Raxd, where n samples with features are included. The i-th row X, represents the i-th data sample x; (i.e., the i-th observation). In this paper, the following unconstrained learning problem is considered
min WERR
Σ (xi, w)
(g))
where (.).) the regularization item with a positive trade-off parameter oj, and (,( denotes loss function w.r.t. w.
Definition 1 (PC' Function). Let f f U + R be continuous function on the open set U E IR.. If (fovee, is set of C. (i.e., r-times continuously differentiable) functions such that f(x) (f(x)))er, holds for every x E U, then f is an r-times piecewise continuously differentiable function, namely PC" function. The (fi)ser is set of selection functions of f.
Assumption We assume that ((w)) and R2 (w) are convex PC functions each with a set of selection functions UkEte {Dt. and UKetn, (Dk,) }. respectively.
In self-paced learning, the goal is to jointly train the model parameter and the latent weight variable v by minimizing
Without ambiguity, (((w) the shorthand notations of (2s, 24; w).

---

m argmin ((w, = Σ Σ wE
RgR2(())
114 (w) + f (oz,
(2)
j=1
i=1
where f (v, represents the SP-reullarierr
2.2 SP-regularizer
Suppose that is a weight variable, E is the loss, and X is the age parameter. f(v, X) called a self paced regularizer, if (i)f(, X] is convex with respect
Definition 2 (SP-reuularizer 35
[0, 1]:
(ūye" (E. X) is monotonically decreasing w.r.f. l, and holds lime- 1" (f, = lime- v (l, = 0: (iit)"" (e, X) is monotonically increasing w.r.t. X, and holds lim v² (L, 1, 1, 1² (f,
where v² (e, A) = arg mitloe/o.11 vl +(U, A).
The Definition gives axiomatic definition of SP-reuularizer, Some frequently utilized SP-regularizers include f (u, 2) -uu, (e, A (2 v), fM (v. X, 7, eth/> and fL.0 X, a) = [KL(1 v), which represents hard, linear, mixture, LOG SP-regularizer, respectively,
2.3 Biconvex Optimization
Definition 3 (Biconvex Function), A function f: B + R a biconvex set B C X x y is called a biconvex function on B, if f=()) == f(x,-) B2 + is convex function on Ba for every fixed E X and fg(-) f(,,)) By - I is convex function on for every fixed E V.
Definition 4 (Partial Optimum). Let f: B - T. be a given biconvex function and let (x", E B. Then, 2 (x" """ called partial optimum of f B, if f (x", v²) (x, Vx By and f (x",)") S (x",y)) Vy E B...
Optimizing leads biconvex optimization problem and is generally non-convex with fixed X, in which number of local minima exist and previous convex optimization tools can't 't achieve a promising effect [36]. It's reasonably believed that algorithms taking advantage of the biconvex structure are more efficient in the corresponding setting. For frequently used one, ACS (c.f. Algorithm [T] is presented to optimize x and y in f (ar, y) alternately until termi- nating condition is met.
Algorithm 1 Alternate Convex Search (ACS)
Require: Dataset X and y, age parameter X.
1: Initialize w.
2: while not converged do
3: Update x* = argmin, L(w", ",0"",) v).
Update w argmin C(w, v²).
5: end while Ensure:
Remark The order of the optimization subproblems in line 4 in Algorithm Drarrtthhmaann be permuted.
Theorem 1. (37) Let X CR" and y C R be closed sets and let f X - R. be continuous. Let the sequence {2, (zi) generated by ACS converges to x" X x y. Then x is a partial optimum.
2.4 Theoretical Consistency
Researchers in earlier study B5 theoretically conducted the latent SPL loss (a.k.a., implicit objective) and further proved that the SPL paradigm converges to the stationary point of the latent objective under some mild assumptions, which gives explanation to the robustness of the SPL B5 38 In this paper, we focus on the partial optimum of original SPL objective and result is given in Theorem
Theorem Under the same assumptions Theorem 2 of [38], the partial optimum of SPL objective consists with the stationary point of implicit SPL objective Gy-
Factoring in both Theorem & 2 the ACS procedure (or its variations) used in SPL paradigm indeed finds the partial optimum SPL objective, which unifies the two proposed analysis frameworks and provides more in-depth understanding the intrinsic mechanism behind the SPL regime.

---

3
Age-Path Tracking
3.1 Objective Reformulation
For the convenience of derivation, we denote the set IR or IR to be the set of indexes j where R, is differentiable or non-differeniaalle at w, respectively. Similarly, we have It and It w.r.t. l.
Moreover, the training of the SPI. is essentially process adaptive sample selection, so we classify all the sample points in the training set into differ- ent sets P = { J......J ac-
X7 ) in mixture f (v, A))
1f < A (or ( E,
E
D,
i 2 (or X in mixture f (),
by X+7
M.
if
< X2 (only used in mixture f (u, A))
cording to their confidence (or loss Figure illustrates partition example when hard, linear or LOG SPrreuulari is used. Since subproblem in line 3 of Algorithm Talways gives closed-fone solutions in iterations to searching partial optimum of (2).
rewrite SPL optimization objective as 33. which is indeed equivalent
Compute i in E arg min wERd
Σ Ry
ΣΣ JEP iEJ
of (l((o) (((w).
(3)
3.2 Piecewise Smooth Age-path
The KKT theorem
25 states that
holds iff
0 E
Σ j=1
(w) ΣΣ o" (l((0), A)- -e,(w),
(4)
JEP
where Ø(-) denotes the subdifferential (set of all subgradients). In the TC" setting, subgradient can be expressed explicitly by essentially active functions (c.f. Lemma
Definition 5 (Essentially Active Set). Let f with a set of selection functions {filiery For x E U, we call [z)x)) = f(x) f((x)
R be a PC" function on the open set U e R"
the active set at æ. and 1ÿ(x) == {i 1y x E el ( int
f()== is the set.
essentially active set at æ. where el(.) and int(() denote the closure and interior of set.
open set U and User, {fi) is set of selec- tion functions of f, then (f(z) = conv conv(Usery(a) (f((z))) -[Eie1ç0)) 4 (V()) Enerç(a) 4 4 0}. Especially, f is differentiable at æ, 0f(x) (V]())).
Lemma 1. (39] Let f U R be IPC function
Assumption 2. We assume that R (x) IR, (x). 1g. (x) = 12 (z) holds for all æ considered and all R,, & in the following.
We adopt mild relaxation as shown in Assumption Investigation 440 confirmed that it can be easily established in most practical scenarios. Without loss of generality, we suppose the following Assumption also holds to further ease the notation burden.
Assumption 3. We assume that R2, l are non-differentiable at with multiple active selection functions, where j E (1,...,m) ; E (1.....n).
Therefore, the condition (4) can be rewritten in detail. Formally, there exists by and to such that
We only present the mainstream SP-regularizerrs here. The partition similar in other cases.
-l:/ if < X for linear f (u, A). More results are shown in 35
For example, we have v," =
0, if 2
5

---

m
Σ
as 26) (D)VD (w) + ΣΣ Σ JEP EE ke17, (w)
of (C(0O). ( (w) 0,
j=1 Ke (w)
DR, (w) D2 (w) = 0, Ak E 1 (w)
VjEl
D (w) D/ ()) = O, v I (w)((4)
Vie
(5)
E 122 (w) 1 = O, (0)) 20, 13jsm
RETR, (1)
Σ
(w) 0, ()) 20 0. isisn,
REIT (+)
where rj, 1 randomly selected from RR 12 and being fixed. The second and third equations in (5) describe the active sets while the last two equations describe the subgradients. When the partial optimum is on the smooth part, we denote the left side of equations (5) to be a C function F, thus revealing that the solution path lies on the smooth manifold F Fww, X, (e) 0. By the time comes across the kink we need to refresh the index partitions and update (5) to run next segment of path. WLOG, we postulate that the initial point is non-degenerate (i.e., the J w.tR..t. is invertible). By directly applying the implicit function theorem, the existence and uniqueness of a local C² solution path (wr, tR. tt) can be established over here. Drawing from the theory of differential geometry gives another intuitive understanding of age-path, which tells that the first equation in (5) indeed uses an analogue moving frame [1] to represent smooth curve that consists of the smooth structure.
Theorem Given a purtial optimum (io, v² (0o,)) at An, iR Le in 5 can be solved from F (wr, λo tR te) = 0. If the Jacobian J Jwtra d is invertible at (wo, tR, te), then in an open neighborhood of No. (w, tR. tt) is a C function w.r.t. X and fits the ODEs
i
d
tR
t:
JA:
(6)
d'
w.R.
which the explicit expressions of t Jx. are listed in Appendix A
Corollary 1. If all the functions smooth in neighborhood of the initial point, then 6 can be simplified as dis/d /d> = -J=1 Jx-
Remark Our supplement parts Appendix Apresent additional discussions.
3.3 Critical Points
By solving the initial value problem (6) numerically with ODE solvers, the solution path regarding to A can be computed swiftly before any of P, IR or changes. We denote such point where the set changes a critical point, which can be divided into turning point or jump point on the basis of path's 's property that point. To be more specific, the age-path discontinuous at a jump point, while being continuous but non-diffcrentiable at the turning point. This is also verified by Figure and large quantity of numerical experiments.
At turning points, the operation of the algorithm is to update P, IR. It according to index violator(s) and move on to the next segment. At jump points, path is no longer continuous and warm-star[] can be utilized to speed up the training procedure. The total number of critical points on the solution path estimated approximately O(I ((DUM) Consequently, we present heuristic trick to figure the type of a critical point with complexity (d). so as to avoid excessive restarts. As a matter of fact, the solutions returned by the numerical ODE solver continuous with the fixed set, despite it may actually passes jump point. In this circumstance, the solutions returned by ODEs have deviated
"Byx &y, hit the restriction bound in Lemma or Ie, P are violated that the entire structure changes. " 'Reuse previous solutions. The subsequent calls to the model will not re-initialise parameters. Prrecisely speaking, it's related to interval length of the nature of objective and the distribution of data.

---

Algorithm 2 Generalized Age-path Algorithm (GAGA)
Input: Initial solution w A X, y, Amin and Amaz.
Output: Age-Path tu (A) on [Xmin Amaz).
1: d + Amins set P, I according to u
2: while d₂ S Amer do
3: Solve (6) and examine partition P. TR lt simultaneously.
4: if Partition P, TR, It was met then
5: Update P.RR. Le according to index violator(s).
6: Solve (6) with updated P, IR, Le-
7: if TKT conditions are not met then
Warm start D (for small 0). 9: end if
8:
end if
11: end while
from the ground truth partial optimum. Hence it's convenient that we can detect KKT conditions to monitor this behavior. This approach enjoys higher efficiency than detecting the partition conditions themselves, especially when the set partition is extraordinarily complex.
3.4 GAGA Algorithm
There has been extensive research in applied mathematics numerical methods for solving ODEs, where the solver could automatically determine the step size of X when solving (6). In the tracking process, we compute the solutions with regard to A. After detecting a new critical point, we need to reset P, IR. It turning point while warm-start is required for jump point. The above procedure is repeated until we traverse the entire interval [Amin: Amaz]. We show the detailed procedure in Algorithm 2 The main computational burden occurs in solving J in (6) with an approximate complexity (I²²) in general, where p denotes the dimension of J. Further promotion can be made via decomposition or utilizing the sparse representation of J on specific learning problems.
4 Practical Guides
In this section, we provide practical guides of using the GAGA to solve two important learning problems, i.e., classic SVM and Lasso. The detailed steps of algorithms are displayed in Appendix C
4.1 Support Vector Machines
Support vector machine (SVM) 30] has attracted much attention from researchers in the areas of bioinformatics, computer vision and pattern recognition. Given the dataset and label y, we focus on the classic support vector classification
minl w.b
2
+ Σ
max {0, 1 - vi((ó(z)), w) + +b)
where H. is the reproducing kernel Hilbert space (RKHS) with the inner product (.) and corresponding kernel function d. Sceing that (5) still holds in infinite dimensional H. the above analyses can be directly applied here. We also utilize the kernel trick 42 avoid involving the explicit expression of o. consistent with the framework, we have L, = C' max {0, 9:} and 9, = y: (o()) w) + b). merely need to refine the division of & as En (i E
The It and P are determined by g. thus
< Ez = {i & & Afterwards, with some simplifications and denoting = Co" @ i, we can obtain a simplified version of 5) from where the age-path can be equivalently calculated w.r.t. optimal (á, b).
0} and &p = f e > 0}. which gives 1, = En U D(UMM)

---

Proposition 1. When ax b indicate partial optimun. the dynamics of optimal a, b in the linear and mixture SP-regularizer are shown as
w.E.. ffr
d
7 b
Yez Qeałz QEPÊz
yēp QEZEP A QÉrEr
-1
0
YEz À Ver
Ez
(8)
dx
IεPEP
C
Lep
OzE d
O.M b
VEz Qεzεz ca QMEZ
yrM QEEM
Yea Ca CRUUM UM
Ogz
(9)
dx
C QMM IMM
X 1M
respectively, where QM = Diagle JQMEz. QMM = Diagle JQM. = eM 2 Yy
Other components are constant OKEN 4N 0xD OD. Only for mixture regularizer, DEE 1SP
Critical Point. We track along the path. The critical point is sparked off by any set in EN, Ex, Ep, M( changes.
4.2 Lasso
Lasso (34] uses sparsity based regularization term that can produce sparse solutions. Given the dataset X and label y, the Lasso regression is stated
min WERS
Xww- yll² + a 1|
(10)
2n
We expand || || -ÿ |wy] and treat [wy] as R, in hence the LR (1 dd: 0 We denote the set of active or inactive functions (components) by A = IR. A TR. respectively. In view of the fact that 0/wy| removes LR, from the equations in (5 for j A, only pay attention
the A part w.r.t. the (w A). The l is defined as En (X - in the following.
Proposition 2. When (w, v" (w. A)) is a partial optimum, the dynamics of optimal w for the linear and mixture SP-regularizer are described as
10
w.t..
Aw dA X2
V2n
DiaDiag {
3 (eexee) xxx)
18 10-2e0
XTole
(11)
dw
V2ny
(Xheumdeuma) Xheum (
O0 )
(12)
dA
X²
em
respectively, where XSUMA = (
Xea and w =0
X.MA
Critical Point. The critical point is encountered when or P changes.
Experimental Evaluation
We present the empirical results of the proposed GAGA on two tasks: SVM for binary classification and Lasso for robust regression in the noisy environment. The results demonstrate that our approach outperforms existing SPL implementations on both tasks.
Baselines. GAGA is compared against three bascline methods: 1) Original learning model without SPL regime. 2) ACS (20) performs sequential search of X, which is the most commonly used algorithm in SPL implementations. 3) MOSPL [29] is a state-of-the-art age-path approach that using the multi objective optimization, in which the solution is derived with the age parameter À implicitly.
Notations such LM for vectors represent the element- wise operation in this section.

---

Dataset
Source
Samples
Dimensions
Tisk
0.25
ACS-40%
20%
míeaa pixel
UCT SS
2000
240
ACS
ACS-00% 80%
pendigits
UCT
3498
16
Generaiization
0.24
GAGA.20%
hiva agnostic
OpenML
4230
1620
40%A-40%
0.23
music
OpeaML A4
1060
117
cadala
UCI
20640
delta elevators
OpenMI.
9517
0.22
houses
OpenML
22600
R
atlerons
OpeaML
13750
41
101
OpenML
A
Table 1: Datasets description in our experiments. The C=Classification, R-Regression.
Figure 4: Robustness to noise. This fig- ure shows the learning curve under dif- ferent noise ratios, which confirms that the GAGA is more robust when in the set- ting of relatively high noise.
Purumeter
Competing Meethods
Ours
Dataset
Restarting times
C.
Original
AC3
MOSPL
GAGA
piselt
1.00.00
0:95-:00 037
0.976-:0000 015
0,978:2:0 021
0.986:t
mfcor-pieel
0.50
0.94
0:947:200031
0.00:20.027
0033--0.011
hive sgnosicf
1000 0.50
0 0:66-1:0.110 0.027
0.941 1:0.09
0046=0.0137
0.960.1.0
8
pendigiist
1000, 1.00
0.921200 069
0.960:200 005
1.962:00001
0 D46
10
pendigins
1.000.200
20.05
0.94::00008 089
0.940:00008$
0.944:20.0088
elevatort
2e
0,146:::0000 011
0.144:000012
0,144:2:0 020
0.143:::0 012
3
allemns?
60-3
0674-10.077
8492.0. 0.006
0.491 D41
0.489 0.009
16
0.325 O...0.00 009
0.211.0.001
0.215:1..01 0.012
0.286 10.013
0
0.679:1:0. 057
0.184
bouses?
0213::00013
0.30110.010
0.305.10.222
0.201
Table 2: Average results with the standard deviation in 20 runs on different datasets using the linear SP- re-reuularizer. The top results in each in boldface.
Parameter
Competing Methods
Restarting times
Citr
Original
ACS
MOSPL
GAGA
infrat-pisal
1,00,
0.959: 0.037
0:63:2000388
0968=0.033
(.973=00
infeat plselt
0.50
0.20, 1,00
0.945::0 025
0.62=0.021
0970= 027
Ry77= 015
10
lina agundic?
0.50
1,00, 1.00
0.868:2:0 022
0.06:000004
0940=0.011
RU57
0. 0007
10
peolisits?
0.50
2,00, 1.00
0.924::: 069
0.06:00.062
0957=0.077
IL962
(88
perdiging
0.20
1,00) 1.00)
00.055
0:40=00088
0992200.080
0.94::0 088
30
07822000022
0754
ailaeen
1..7:
0452:00.077
0.43=20.000
1422=00.000
muict
03.59
0218:0.000
0.162
IL213
delta elevatorst
3e-3
0.83::0.
0663200074
0::0=:0.000 0/29
0.395=0.
housest
0.50
3e-3
0.01:
0.14620.012
0.144:0.022
0.142=0.012
Table 3: Average results with the standard deviation in 20 runs on different datasets using the mixture (P-reuularizer. The top results in each row in boldface.
Datasets. The Table [ summarizes the datasets information. As universally known that SPL enjoys robustness in noisy environments, we impose 30% of noises into the real-world datasets. In particular, we generate noises by turning normal samples into poisoning ones by flipping their labels 45 46 for classification tasks. For regression problem, noises are generated by the similar distribution of the training data as performed in (77].
Experiment Setting. In experiments, we first verify the performance GAGA and traditional ACS algorithm under different noise intensity to reflect the robustness of GAGA. We further study the generalization performance of GAGA with competing methods, so as to show its ability select optimal model during the learning process. Meanwhile, we also evaluate the running efficiency between GAGA and existing SPL. implementations in different settings, which examines the speedup of GAGA as well as its practicability, Finally, count the number of restarts and different types of

---

2.207
X1.0
107
100
10-
0.5
1.0
1.5
2.0
0.5
1.0
1.5
2.0
10
35
20
100
15
Samole Size
1e3
Sample Size
163
Avar -Aas
Armu Apeo
101
107
1.0
107
107
10
10%
10
20
15
Samole Sec
le
Somple Size
Avar
Ame
Aew
Anto
ACS
MOSPI
GAGA
Figure 5: The study of efficiency comparison. y-axis denotes the average running time (in seconds) with 20 runs. The interval [Amins Amaz] refers to the predefined searching area.
critical points when using GAGA, to investigate its ability to address critical points. For SVM, use Gaussian kernel K (x1,, x2) = exp( -w / - x2ll²). More details can found in Appendix D
Results. Figure 4 illustrates that conventional ACS fails to reduce the generalization error due to heavy noises fed to the model at a large age (i.e., overfits the dataset), while GAGA makes up for this shortcoming by selecting the optimal model that merely learns the trust-worthy samples during the continuous learning process. Ta- and demonstrate demonnttattt an overall performance enhancing in GAGA than competing algorithms. The "t" in tables denotes 30% of tificial noise, while "f" represents 20%. Note that performances measured by accuracy and generalization error for classifica- tion and regression, respectively. The results guarantee that GAGA outperforms the state-of-the-att approaches in SPL under different circumstances. Figure 5) shows that GAGA also enjoys a high com- putational efficiency by changing the sample size well as the predefined age sequence, emphasizing the potentials of utilizing GAGA in practice. The number of different types of critical points some datasets is given in Figure 6 Corresponding restarting times can be found in Table and 3] hence indicate that GAGA capable of identifying different types of critical points and uses the heuristic trick to avoids restarts at massive turning points.
Tuming
1000
+ump points
750
500
250
cad.hou. all. mws.del. ele.
Figure Histogram illustrat- ing the number of different types of critical points. Names of datasets are shortened into the first letters.
We further demonstrate the ability of GAGA address the relatively large sample size and present more histograms. In addition, we apply GAGA to the logistic regression 33 for classification. We also verify that conventional ACS indeed tracks an approximation path of partial optimum in experiments, which provides more in-depth understanding towards SPL and the performance promotion brought by GAGA. We also conduct a comparative study to the state-of-the-are robust model for SVMs 48 49
Additional Experiments in Appendix D
50
and Lasso [51 besides the SPL domain.
6 Conclusion
In this paper, we connect the SPL paradigm the partial optimum for the first time. Using this idea, we propose the first exact age-path algorithm able to tackle general SP-regularizer, namely GAGA. Experimental results demonstrate GAGA outperforms traditional SPL paradigm and the state-of-the-a age-path approach in many aspects, especially in the highly noisy environment. We further build the relationship between our framework and existing theoretical analyses on SPL regime, which provides more in in-depth understanding towards the principle behind SPL.
10

---

References
[1] M. Kumar, Ben Packer, and Daphne Koller. Self-paced learning for latent variable models. pages 1189-1197, 01 2010.
[2] Lu Jiang, Deyu Meng, Shoou-1 Yu, Zhenzhong Lan, Shiguang Shan, and Alexander Hauptmann. Self-paced learning with diversity. Advances neural information processing systems, 27, 2014.
[3] Qian Zhao, Deyu Meng, Lu Jiang, Qi Xie, Zongben Xu, and Alexander G Hauptmann. Self- paced learning for matrix factorization. In Twenty-ninth AAAI conference artificial intelli- gence, 2015.
[4] Fan Ma, Deyu Meng, Qi Xie, Zina Li, and Xuanyi Dong. Self- co-training. In International Conference on Machine Learning, pages 2275-2284. PMLR, 2017.
[5] Liang Lin, Keze Wang, Deyu Meng. Wangmeng Zuo, and Lei Zhang. Active self- seef-paced learning for cost-effective and progressive face identification. IEEE transactions on pattern analysis and machine intelligence, 40(1):7-19, 2017.
[6] Kamran Ghasedi, Xiaoqian Wang, Cheng Deng, and Heng Huang. Balanced self-paced learning for generative adversarial clustering network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4391-4400, 2019.
[7] Hongchang Gao and Heng Huang. Self-paced network embedding. In Proceedings of the 24th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 1415, 2018.
[8] Jiangzhang Gan, Guoqiu Wen, Hao Yu, Wei Zheng, and Cong Lei. Supervised feature selection by self-paced learning regression. Pattern Recognition Letters, 132:30-37, 2020.
[9] James S Supancic and Deva Ramanan. Self-paced learning for long-term tracking. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 2379-2386 2013.
[10] Lili Pan, Shijie Ai, Yazhou Ren, and Zenglin Xu. Self-paced deep regression forests with consideration on underrepresented examples. In European Conference on Computer Vision, pages 271-287. Springer, 2020.
[11] Xiaoping Wu, Jianlong Chang, Yu-Kun Lai, Jufeng Yang, and Qi Tian. Bispl: Bidirectional self-paced learning for recognition from web data. IEEE Transactions on Image Processing, 30:6512 1016512-657,220 2021.
[12] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. Proceedings of the 26th annual international conference on machine learning, pages 41 41-48, 2009.
[13] Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. Curriculum learning: A survey. International Journal of Computer Vision, pages 1-40, 2022.
[14] Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander G Hauptmann. Self-paced curriculum learning. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
[15] Deyu Meng, Qian Zhao, and Lu Jiang. A theoretical understanding of self-paced learning. Information Sciences, 414:319-328. 2017.
[16] Lu Jiang, Deyu Meng, Teruko Mitamura, and Alexander G Hauptmann. Easy samples first: Self-paced reranking for zero-example multimedia search. In Proceedings of the 22nd ACM international conference Multimedia, pages 547-556, 2014.
[17] Fan Ma. Deyu Meng, Xuanyi Dong, and Yi Yang. Self-paced nie co-training. Journal of Machine Learning Research, 2020.

---

[18] Jiangtao Peng, Weiwei Sun, and Qian Du. Self joint sparse representation for the classification of hyperspectral images. IEEE Transactions on Geoscience and Remote Sensing. ((()::18331194, 2018.
[19] Maoguo Gong, Hao Li, Deyu Meng, Qiguang Miao, and Jia Liu. Decomposition- evolu- tionary multiobjective optimization to self-paced learning. IEEE Transactions on Evolutionary Computation, 23(2):288-300 -02, 2018.
[20] Richard E Wendell and Arthur P Hurter Jr. Minimization of non-separable objective function subject to disjoint constraints. Operations Research, 24(4):643-657, 1976.
[21] James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper- parameter optimization. Advances in neural information processing systems, 24, 2011.
[22] Jonathan F Bard. Practical bilevel optimization: algorithms and applications, volume 30.
Springer Science & Business Media, 2013.
[23] Bradley Efron, Trevor Hastie, lain Johnstone, and Robert Tibshirani. Least angle regression. The Annals of statistics, 32(2):407-499 2004.
[24] Trevor Hastie, Saharon Rosset, Robert Tibshirani, and Zhu. The entire regularization path for the support vector machine. Journal of Machine Learning Research, 1311-11)13111111, 2004.
[25] William Karush. Minima of functions several variables with inequalities as side constraints. M. Sc. Dissertation. Dept. of Mathematics, Univ. of Chicago, 1939.
[26] Ryan Tibshirani and Jonathan Taylor. The solution path of the generalized lasso. The annals of statistics, 9933):133--1371, 2011.
[27] Kohei Ogawa, Motoki Imamura, Ichiro Takeuchi, and Masashi Sugiyama. Infinitesimal an- nealing for training semi-supervised support vector machines. In International Conference Machine Learning, pages 897-905. PMLR, 2013.
[28] Bin Gu and Victor S Sheng. A solution path algorithm for general parametric quadratic programming problem. IEEE transactions on neural networks and learning systems, 29(9):4462- 4472, 2017.
[29] Hao Li, Maoguo Gong, Deyu Meng, and Qiguang Miao. Multi-objective self- paced learning. In Thirtieth AAAI conference on artificial intelligence, 2016.
[30] Hao Li, Maoguo Gong, Congcong Wang, and Qiguang Miao. Pareto self-paced learning based on differential evolution. IEEE Transactions on Cybernetics, 2019.
[31] Bin Gu, Zhou Zhai, Xiang Li, and Heng Huang. Finding age path of self-paced learning. In 2021 IEEE International Conference on Data Mining (ICDM), pages 151-160. IEEE, 2021.
[32] Bin Gu, Xiao- Tong Yuan. Songean Chen, and Heng Huang. New incremental learning algo- rithm for semi- supervised support vector machine. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1475-1484, 2018.
[33] Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 1999.
[34] Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), (8(1)2677-288 1996.
[35] Deyu Meng, Qian Zhao, and Lu Jiang. What objective does self self-paced learning indeed optimize? arXiv preprint arXiv: 15/ 1.0644,, 2015.
[36] Guoqi Li, Changyun Wen, Wei Xing Zheng, and Guangshe Zhao. Iterative identification of block-oriented nonlinear systems based on biconvex optimization. Systems Control Letters, 79:68-75, 2015.
12

---

[37] Jochen Gorski, Frank Pfeuffer, and Kathrin Klamroth. Biconvex sets and optimization with biconvex functions: a survey and extensions. Mathematical methods of operations research, (((3))373-407, 2007.
[38] Zilu Ma, Shiqi Liu, Deyu Meng, Yong Zhang, SioLong Lo, and Zhi Han. On convergence properties of implicit self-paced objective. Information Sciences, 462: 132-140, 2018.
[39] Michael Held, Philip Wolfe, and Harlan P Crowder. Validation of subgradient optimization. Mathematical programming. 6(1):62-88) 88, 1974.
[40] Bennet Gebken, Katharina Bieker, and Sebastian Peitz. On the structure of regularization paths for piecewise differentiable regularization terms. arXiv preprint arXiv:2111.06775, 2021.
[41] Barrett O'neill. Elementary differential geometry. Elsevier, 2006.
[42] Bernhard Schölkopf. The kernel trick for distances. Advances in neural information processing systems, 13, 2000.
[43] Arthur Asuncion and David Newman. Uci machine learning repository. 2007.
[44] Joaquin Vanschoren, Jan N Van Rijn, Bernd Bischl, and Luis Torgo Openml: networked science
in machine learning. ACM SIGKDD Explorations Newsletter, 15(2):99-60, 2014.
[45] Benoît Frénay and Michel Verleysen. Classification the presence of label noise: survey, Neural Networks and Learning Systems, IEEE Transactions on, 25:845-869, 05 2014.
[46] Aritra Ghosh, Himanshu Kumar, and P. Sastry. Robust loss functions under label noise for deep neural networks. 12 2017.
[47] Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina Nita-Rotaru, and Bo Li. Manipulating machine learning: Poisoning attacks and countermeasures for regression learning. pages 19-35, 05 2018.
[48] Chuanfa Chen, Changqing Yan, and Yanyan Li. A robust weighted least squares support vector regression based on least trimmed squares. Neurocomputing, 168:941-966, 2015.
[49] Xiaowei Yang. Liangjun Tan, and Lifang He. A robust least squares support vector machine for regression and classification with noise. Nearocomputing, 140:41-52, 2014.
[50] Battista Biggio, Blaine Nelson, and Pavel Laskov. Support vector machines under adversarial label noise. In Asian conference machine learming, pages 97-112. PMLR, 2011.
[51] Nasser Nasrabadi, Trac Tran, and Nam Nguyen. Robust lasso with missing and grossly corrupted observations. Advances in Neural Information Processing Systems, 24, 2011.
[52] Adi Ben-Israel and T Greville. Generalized Inverse: Theory and Application. 01 1974.
[53] Roger A. Horn and Charles R. Johnson. Matrix Analysis. Cambridge University Press, 1990.
[54] Krishnan Radhakrishnan and Alan Hindmarsh. Description and use of Isode, the livermore solver for ordinary differential equations. 01 1994.
[55] J. Watt. Numerical initial value problems in ordinary differential equations. The Computer Journal, 155, 05 1972.
[56] William Adkins and Mark Davidson. Ordinary Differential Equations. 01 2012.
[57] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Prettenhofer. R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830 2011.
[58] ct.al. Jazzbin. geatpy: The genetic and evolutionary algorithm toolbox with high performance in python, 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Appendix
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix [.2
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work existing assets, did you cite the creators? [Yes] See Table
(b) Did you mention the license of the assets? [Yes] Licenses are available referring the provided links.
(c) Did you include any assets either in the supplemental material or URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you'le 'u using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] This paper does not use crowdsourcing.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] This paper does
crowdsourcing. (c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A] This paper does not use crowdsourcing.
14


Reviews:
[{"review_id": "vln9cMiBdH", "reviewer": "Reviewer_oJry", "summary": "This paper proposes a general age-path algorithm for various self-paced regularizers based on solving ordinal differential equations (ODE). Prior methods in self-paced learning (SPL) are prone to be underfitting and sensitive to noise and outliers, mostly due to the difficulty of choosing the optimal model age. Moreover, some existing solution path algorithms such as Lasso and SVM can not apply to SPL due to the optimization difficulty. To tackle these issues, this paper establishes a theoretical analysis based on the KKT condition and some fair assumptions and proposes a novel algorithm based on solving ODE. The methodology seems solid and the authors also introduce some guidance in practice.", "questions": "1. Please compare the proposed method with recent SOTA such as [1].   \n2. Please compare the training time of the proposed method with the baselines.  \n3. What is the meaning of 0 belonging to a scalar in Eqn.4?  \n4. Why are some results of the proposed method in bold in Table 2 and 3, even if they underperform the baselines?  \n[1]. Finding age path of self-paced learning. ICDM 2021.    \n\n  \nAfter Rebuttal:  \nThanks for the responses, which have addressed my concerns. Thus, I have raised my score from 4 to 6.", "limitations": "Not applicable.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:  \nThis paper proposes an in-depth theoretical analysis of age-tracking in SPL. The proposed method incorporates KKT condition and ODE to advance the empirical bilevel solution of the prior method, leading to more stable, sample-efficient, and robust optimization.   \n\nWeaknesses:  \nThe main concerns are clarity and experimental evaluation. In fact, I am confused by some notations and formulations. For example, 0 can not belong to another scalar in Eqn.4 (the right-hand side of Eqn.4 is a scalar). Secondly, the proposed method needs to solve an ODE in each iteration, as shown in Algorithm 2. It is important to discuss the training efficiency as solving an ODE in each iteration may be time-consuming. Moreover, the experimental results in Table 2 and 3 are quite misleading. The proposed method actually underperforms the baseline methods on many datasets but its results are in bold. Meanwhile, a comparison with the recent SOTA is also needed.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "i3IiimeBRlP", "reviewer": "Reviewer_sP8f", "summary": "This paper proposes GAGA, an exact age-path algorithm to tackle general SP-regularizer for the self-paced learning (SPL) paradigm. The authors build the relationship between the GAGA framework and the existing theoretical analysis on SPL to provide a more in-depth understanding of the principle behind SPL.", "questions": "Major concerns about missing literature review and related works:\n- Relation to curriculum learning (CL): according to [1], it seems the SPL is a sub-category of CL. However, there is no mention of the CL domain in this paper. The authors are recommended to conduct a comprehensive survey of the related works in CL in general, as well as other sub-categories of CL to discuss the relationships among them.\n- Relation to the general robustness model under label noise: The paper claims the robustness of the proposed GAGA model, but did not conduct any comparative study to the SOTA robust model besides the SPL domain. For instance, robust SVM models such as [2, 3, 4], and robust lasso models such as [5] should also be considered as comparison methods in the experimental studies. In addition, a comprehensive literature review of the existing works about the general robustness models designed for handling noisy environments needs to be included as related works. \n\nMajor concerns about experimental validation:\n- Concerns about the choices of the backbone model's parameters: how the backbone model parameters (e.g. alpha) are chosen in the experimental section for each dataset as shown in Table 2 and Table 3? Are they based on the baseline or GAGA? It will be unfair to compare the performance using the best parameter specifically selected based on GAGA on comparison methods. \n- Concerns about the performance gain as compared to the existing works: the performance gain of the proposed GAGA is very marginal as compared to existing methods, as shown in Table 2 and Table 3. More specifically, less than 1% of improvement is observed in many datasets as compared with ACS and MOSPL, such as in pendigits, elevator, ailerons, and mfeat-pixel datasets, leading to no statistically significant conclusions of the results. \n- Concerns about the choices of the noise ratios studied: In Table 2 and Table 3, only 20% and 30% noise ratios are selected and shown among different datasets without clear justification of the choices.  What is the rationale for choosing these specific numbers instead of applying a more comprehensive and well-spread range of the ratios (e.g. 20%, 40%, 60%, etc.)?\n\nReference:\n[1] Soviany, P., Ionescu, R. T., Rota, P., & Sebe, N. (2022). Curriculum learning: A survey. International Journal of Computer Vision, 1-40.\n[2] Biggio, Battista, Blaine Nelson, and Pavel Laskov. \"Support vector machines under adversarial label noise.\" Asian conference on machine learning. PMLR, 2011.\n[3] Yang, X., Tan, L., & He, L. (2014). A robust least squares support vector machine for regression and classification with noise. Neurocomputing, 140, 41-52.\n[4] Chen, C., Yan, C., & Li, Y. (2015). A robust weighted least squares support vector regression based on least trimmed squares. Neurocomputing, 168, 941-946.\n[5] Nasrabadi, N., Tran, T., & Nguyen, N. (2011). Robust lasso with missing and grossly corrupted observations. Advances in Neural Information Processing Systems, 24.\n", "limitations": "Not applicable. The authors did not provide any limitations or potential negative societal impact of their work.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The proposed GAGA is the first exact age-path algorithm to tackle the general SP-regularizer for the self-paced learning (SPL) paradigm.\n+ The authors build the relationship between the proposed GAGA framework and the existing theoretical analysis on SPL to provide a more in-depth understanding of the principle behind SPL.\n+ The proposed GAGA is more computationally efficient than existing methods.\n- Missing related works about the relation of the work to the general curriculum learning models \n- Missing related works and experimental comparison to the general robustness model under label noise\n- Missing justifications about many specific parameter choices in the experimental validation.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Zd1SfQXioz0", "reviewer": "Reviewer_Qt26", "summary": "This paper proposes an age-path algorithm (named GAGA) to tackle general SP-regularizer in self-paced learning.\nDuring the optimization process, GAGA detects critical points via solving ODE.\nExperimental results further verify the superiorty of GAGA over traditional SPL paradigms in multiple small-scale datasets.", "questions": "I hope the author could explain the motivation behind the proposed method more clearly.\n\nBesides, using deep learning methods besides SVM and Lasso on some large-scale datasets to verify the effectiveness of the proposed method is more persuasive.\n\n-----------\nAfter reading the authors' response, I would increase the score from 5 to 6.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. this studied problem is interesting and would attract attention from different machine learning topics\n\n\nWeaknesses:\n\n1. this paper is not easy to follow especially for the method section, many variables are not well defined ($m$ in Eq.(1), and $I_R$ is not mentioned in Section 3.1). Different sets in Line 160 are also explained with the right equation.\n\n2. Overall, the motivation of the proposed method is not clear.\n\n3. the experiment part is not enough to validate the effectiveness of the proposed method since most of these datasets in Table 1 are small and easy. Large-scale datasets like Dog-120 in [a] are suggested to be used in the experiment.\n\n[a]. Wu, Xiaoping, et al. \"BiSPL: Bidirectional self-paced learning for recognition from web data.\" IEEE Transactions on Image Processing 30 (2021): 6512-6527.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "Ho6hWcufzeQ", "reviewer": "Reviewer_ZV4b", "summary": "This paper tackles the critical problem of how to optimally choose the age parameter and stop the increasingly learning process in self-paced learning (SPL). Grounded on the bi-convexity of the problem, the authors observe and prove that previous SPL methodologies are closely connected to the partial optimum of the SPL objective function. Thus, they reformulate the SPL paradigm as searching the partial optimum with optimal age parameter which is merely related to a single group of variables. Then, a generalized age-path algorithm that exactly traces the path of the partial optimum is proposed with the technique of ordinary differential equations (ODEs). Despite the general framework, two detailed algorithms on SVM and Lasso are discussed in the paper. Numerical results demonstrate the correctness and efficiency of the algorithm.", "questions": "1. It is expected to include [2] in the paper since it also provides insights to reveal the intrinsic mechanism behind SPL. Could the authors compare the differences between [2] and the proposed method in this paper?\n2. In Section 4.1, the authors restrict the problem into the subspace generated by the basis $\\{y_{i}\\phi(x_{i})\\}_{i=1}^{n}$, while the proof in Appendix B.4 adopts another way to convert the problem. It is a little confusing. Could the authors clarify more about these two statements? Is it possible to unify them?\n3. What is the difference between Eq.(5) in the main body and Eq.(17) in the Appendix, so that the author placed them separately? In my understanding, they are both the detailed expansion of the KKT conditions of the original objective.\n4. I noticed that $Gen$, $N_p$, and polynomial order $t$ are fixed for MOSPL. How did the authors choose these parameters in their experiments? Is it because they have been searched in advance, which is the best practice? If they are not fully studied, it is likely to damage the comparative fairness of the baseline.\n5. Could the authors briefly summarize the limitations of this work? \n***************\nDuring the response period, the authors provided additional empirical results to support their experiments, and more discussions were supplemented in the Appendix. My concerns have been addressed. So I update my rating.", "limitations": "Please refer to the above discussions and questions.\n******************\nDuring the authors' responses period, more discussions have been provided in the Appendix part.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n1. The problem that this paper tries to address is significant in the SPL but lacks sufficient investigations both in the theoretical and practical aspects before. Unlike the existing studies, the framework derived in this paper is able to address most currently used self-paced regularizers (having explicit closed forms and following definitions in [1]) while owning theoretical guarantees.\n2. The paper proposes a rigorous theoretical framework. All detailed proofs of all the results are given in the paper (except Theorem 5 in Appendix D). The experimental results are also fairly abundant in different views.\n3. This paper provides a novel and interesting perspective to reconsider the previous SPL regime w.r.t the concept of partial optimum. The authors provide various discussions which are supported by both intuitions and theoretical results and dive much deeper above the concept itself.\n4. The technique that utilizes the closed-form in alternative optimization to derive the age path is interesting. Since the problems of optimizing two groups of variables attract more and more focus (e.g., bi-level problems), I think this idea is of great potential to be studied further.    \n\n\n### Weakness:\n1. It seems that the limitations of the proposed method are not explicitly discussed in the paper. \n2. The theoretical framework and its derivation look quite complicated (to be rigorous and solid) and thus somewhat difficult to understand since there are many definitions, assumptions as well as reformulations. But the specific algorithms on the SVM and Lasso make up for this part in some sense.\n3. The proposed framework is based on the earliest SPL regime, which involves only one hyperparameter. Hence the application scenarios of GAGA are restricted. The author should consider the proposed algorithm in more recent SPL formulations with more hyperparameters.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept