Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a general age-path algorithm for various self-paced regularizers based on solving ordinal differential equations (ODE). Prior methods in self-paced learning (SPL) are prone to be underfitting and sensitive to noise and outliers, mostly due to the difficulty of choosing the optimal model age. Moreover, some existing solution path algorithms such as Lasso and SVM can not apply to SPL due to the optimization difficulty. To tackle these issues, this paper establishes a theoretical analysis based on the KKT condition and some fair assumptions and proposes a novel algorithm based on solving ODE. The methodology seems solid and the authors also introduce some guidance in practice.
strengths_and_weaknesses: Strengths:  
This paper proposes an in-depth theoretical analysis of age-tracking in SPL. The proposed method incorporates KKT condition and ODE to advance the empirical bilevel solution of the prior method, leading to more stable, sample-efficient, and robust optimization.   

Weaknesses:  
The main concerns are clarity and experimental evaluation. In fact, I am confused by some notations and formulations. For example, 0 can not belong to another scalar in Eqn.4 (the right-hand side of Eqn.4 is a scalar). Secondly, the proposed method needs to solve an ODE in each iteration, as shown in Algorithm 2. It is important to discuss the training efficiency as solving an ODE in each iteration may be time-consuming. Moreover, the experimental results in Table 2 and 3 are quite misleading. The proposed method actually underperforms the baseline methods on many datasets but its results are in bold. Meanwhile, a comparison with the recent SOTA is also needed.

questions: 1. Please compare the proposed method with recent SOTA such as [1].   
2. Please compare the training time of the proposed method with the baselines.  
3. What is the meaning of 0 belonging to a scalar in Eqn.4?  
4. Why are some results of the proposed method in bold in Table 2 and 3, even if they underperform the baselines?  
[1]. Finding age path of self-paced learning. ICDM 2021.    

  
After Rebuttal:  
Thanks for the responses, which have addressed my concerns. Thus, I have raised my score from 4 to 6.
limitations: Not applicable.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes an age-path algorithm (named GAGA) to tackle general SP-regularizer in self-paced learning.
During the optimization process, GAGA detects critical points via solving ODE.
Experimental results further verify the superiorty of GAGA over traditional SPL paradigms in multiple small-scale datasets.
strengths_and_weaknesses: Strengths:

1. this studied problem is interesting and would attract attention from different machine learning topics


Weaknesses:

1. this paper is not easy to follow especially for the method section, many variables are not well defined ($m$ in Eq.(1), and $I_R$ is not mentioned in Section 3.1). Different sets in Line 160 are also explained with the right equation.

2. Overall, the motivation of the proposed method is not clear.

3. the experiment part is not enough to validate the effectiveness of the proposed method since most of these datasets in Table 1 are small and easy. Large-scale datasets like Dog-120 in [a] are suggested to be used in the experiment.

[a]. Wu, Xiaoping, et al. "BiSPL: Bidirectional self-paced learning for recognition from web data." IEEE Transactions on Image Processing 30 (2021): 6512-6527.

questions: I hope the author could explain the motivation behind the proposed method more clearly.

Besides, using deep learning methods besides SVM and Lasso on some large-scale datasets to verify the effectiveness of the proposed method is more persuasive.

-----------
After reading the authors' response, I would increase the score from 5 to 6.
limitations: N/A
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper tackles the critical problem of how to optimally choose the age parameter and stop the increasingly learning process in self-paced learning (SPL). Grounded on the bi-convexity of the problem, the authors observe and prove that previous SPL methodologies are closely connected to the partial optimum of the SPL objective function. Thus, they reformulate the SPL paradigm as searching the partial optimum with optimal age parameter which is merely related to a single group of variables. Then, a generalized age-path algorithm that exactly traces the path of the partial optimum is proposed with the technique of ordinary differential equations (ODEs). Despite the general framework, two detailed algorithms on SVM and Lasso are discussed in the paper. Numerical results demonstrate the correctness and efficiency of the algorithm.
strengths_and_weaknesses: ### Strengths:
1. The problem that this paper tries to address is significant in the SPL but lacks sufficient investigations both in the theoretical and practical aspects before. Unlike the existing studies, the framework derived in this paper is able to address most currently used self-paced regularizers (having explicit closed forms and following definitions in [1]) while owning theoretical guarantees.
2. The paper proposes a rigorous theoretical framework. All detailed proofs of all the results are given in the paper (except Theorem 5 in Appendix D). The experimental results are also fairly abundant in different views.
3. This paper provides a novel and interesting perspective to reconsider the previous SPL regime w.r.t the concept of partial optimum. The authors provide various discussions which are supported by both intuitions and theoretical results and dive much deeper above the concept itself.
4. The technique that utilizes the closed-form in alternative optimization to derive the age path is interesting. Since the problems of optimizing two groups of variables attract more and more focus (e.g., bi-level problems), I think this idea is of great potential to be studied further.    


### Weakness:
1. It seems that the limitations of the proposed method are not explicitly discussed in the paper. 
2. The theoretical framework and its derivation look quite complicated (to be rigorous and solid) and thus somewhat difficult to understand since there are many definitions, assumptions as well as reformulations. But the specific algorithms on the SVM and Lasso make up for this part in some sense.
3. The proposed framework is based on the earliest SPL regime, which involves only one hyperparameter. Hence the application scenarios of GAGA are restricted. The author should consider the proposed algorithm in more recent SPL formulations with more hyperparameters.
questions: 1. It is expected to include [2] in the paper since it also provides insights to reveal the intrinsic mechanism behind SPL. Could the authors compare the differences between [2] and the proposed method in this paper?
2. In Section 4.1, the authors restrict the problem into the subspace generated by the basis $\{y_{i}\phi(x_{i})\}_{i=1}^{n}$, while the proof in Appendix B.4 adopts another way to convert the problem. It is a little confusing. Could the authors clarify more about these two statements? Is it possible to unify them?
3. What is the difference between Eq.(5) in the main body and Eq.(17) in the Appendix, so that the author placed them separately? In my understanding, they are both the detailed expansion of the KKT conditions of the original objective.
4. I noticed that $Gen$, $N_p$, and polynomial order $t$ are fixed for MOSPL. How did the authors choose these parameters in their experiments? Is it because they have been searched in advance, which is the best practice? If they are not fully studied, it is likely to damage the comparative fairness of the baseline.
5. Could the authors briefly summarize the limitations of this work? 
***************
During the response period, the authors provided additional empirical results to support their experiments, and more discussions were supplemented in the Appendix. My concerns have been addressed. So I update my rating.
limitations: Please refer to the above discussions and questions.
******************
During the authors' responses period, more discussions have been provided in the Appendix part.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes GAGA, an exact age-path algorithm to tackle general SP-regularizer for the self-paced learning (SPL) paradigm. The authors build the relationship between the GAGA framework and the existing theoretical analysis on SPL to provide a more in-depth understanding of the principle behind SPL.
strengths_and_weaknesses: + The proposed GAGA is the first exact age-path algorithm to tackle the general SP-regularizer for the self-paced learning (SPL) paradigm.
+ The authors build the relationship between the proposed GAGA framework and the existing theoretical analysis on SPL to provide a more in-depth understanding of the principle behind SPL.
+ The proposed GAGA is more computationally efficient than existing methods.
- Missing related works about the relation of the work to the general curriculum learning models 
- Missing related works and experimental comparison to the general robustness model under label noise
- Missing justifications about many specific parameter choices in the experimental validation.

questions: Major concerns about missing literature review and related works:
- Relation to curriculum learning (CL): according to [1], it seems the SPL is a sub-category of CL. However, there is no mention of the CL domain in this paper. The authors are recommended to conduct a comprehensive survey of the related works in CL in general, as well as other sub-categories of CL to discuss the relationships among them.
- Relation to the general robustness model under label noise: The paper claims the robustness of the proposed GAGA model, but did not conduct any comparative study to the SOTA robust model besides the SPL domain. For instance, robust SVM models such as [2, 3, 4], and robust lasso models such as [5] should also be considered as comparison methods in the experimental studies. In addition, a comprehensive literature review of the existing works about the general robustness models designed for handling noisy environments needs to be included as related works. 

Major concerns about experimental validation:
- Concerns about the choices of the backbone model's parameters: how the backbone model parameters (e.g. alpha) are chosen in the experimental section for each dataset as shown in Table 2 and Table 3? Are they based on the baseline or GAGA? It will be unfair to compare the performance using the best parameter specifically selected based on GAGA on comparison methods. 
- Concerns about the performance gain as compared to the existing works: the performance gain of the proposed GAGA is very marginal as compared to existing methods, as shown in Table 2 and Table 3. More specifically, less than 1% of improvement is observed in many datasets as compared with ACS and MOSPL, such as in pendigits, elevator, ailerons, and mfeat-pixel datasets, leading to no statistically significant conclusions of the results. 
- Concerns about the choices of the noise ratios studied: In Table 2 and Table 3, only 20% and 30% noise ratios are selected and shown among different datasets without clear justification of the choices.  What is the rationale for choosing these specific numbers instead of applying a more comprehensive and well-spread range of the ratios (e.g. 20%, 40%, 60%, etc.)?

Reference:
[1] Soviany, P., Ionescu, R. T., Rota, P., & Sebe, N. (2022). Curriculum learning: A survey. International Journal of Computer Vision, 1-40.
[2] Biggio, Battista, Blaine Nelson, and Pavel Laskov. "Support vector machines under adversarial label noise." Asian conference on machine learning. PMLR, 2011.
[3] Yang, X., Tan, L., & He, L. (2014). A robust least squares support vector machine for regression and classification with noise. Neurocomputing, 140, 41-52.
[4] Chen, C., Yan, C., & Li, Y. (2015). A robust weighted least squares support vector regression based on least trimmed squares. Neurocomputing, 168, 941-946.
[5] Nasrabadi, N., Tran, T., & Nguyen, N. (2011). Robust lasso with missing and grossly corrupted observations. Advances in Neural Information Processing Systems, 24.

limitations: Not applicable. The authors did not provide any limitations or potential negative societal impact of their work.
review_title: null

Meta-review: metareview: The paper received 4 positive reviews after the rebuttal. The technical concerns raised by the reviewers were addressed properly. Overall this work introduces a challenging and realistic setting that can be of large interest to the community working on self-paced learning.
confidence: Certain
Decision: Accept