Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a consolidated crossed validation for Support Vector Mahines (ccvsvm), based on the recent leave-one-out lemma and magicsvm, having as objective to reduce the training time via data reduction. Specifically, this two-stage ccvsvm deals with the SVM bias term, which is not handled by some existing data reduction methods. The authors demonstrate with numerical experiments that the proposed algorithm is about an order of magnitude faster than the two mainstream SVM solvers, kernlab and  LIBSVM, with almost the same accuracy.
strengths_and_weaknesses: •	Strengths: The core idea of the work is original in the sense of thinking at the same time data reduction (as part of CV), the QP optimization process and the LOO at the same time. The quality of the research work is solid. The work is clear and simple in explaining the state of the art, their contribution, the definition of their methods, and their general experimental setting.
•	Weaknesses: It will be great to have the code available to (at least partially) reproduce the experiments. Section 4.1 is of particular interest and table 1 underline how the proposed ccvsvm algorithm results efficient, but, again, reproducibility is a key. This comment also applies to section 4.2.

questions: •	The manuscript section organization is missing from the introduction, it should be included.
•	One particular characteristic of this proposed solution is the sparsity on the solution of SVM. Is it possible to use the solution of the k-1 fold to improve the initial solution on the k-fold (finding the initial support vectors and non-zero alpha values)?

limitations: The authors address some of the method’s limitations and provide some future lines to generalize to other K-fold CV or other computationally-expensive ML methods
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: In this paper, an integrated Cross-Validation procedure is introduced for one of the leading classifiers, kernel SVM, and an algorithm called ccvsvm is developed. This algorithm is based on the recently proposed leave-one-out lemma and magicsvm algorithms. Numerical experiments demonstrate that the algorithm is more than an order of magnitude faster than the two SVM solvers but with almost the same accuracy compared to existing data reduction methods.
strengths_and_weaknesses: Strengths:
- The proposed method can perform leave-one-out cross-validation very fast, as shown by theoretical aspects and numerical experiments.

- Demonstrations are carried out using kernlab and LIVSVM, the main packages of existing support vector machines.

Weaknesses:
- Due to its reliance on support vector machine mechanisms, it is currently impossible to demonstrate its effectiveness with other machine learning algorithms.

- Numerical experiments have been carried out on relatively small benchmark data sets. For example, the performance has not been tested when the number of cases exceeds 10 000.
questions: Q1: Numerical experiments demonstrating the effectiveness of the proposed method have been carried out with a maximum number of cases of around 3,000. Would the proposed method still be effective if, for example, the number of instances exceeds 10,000?

Q2: Do you already have ideas for extending the proposed method to machine learning algorithms other than SVM? If so, it would improve the value of the paper if you could present them in the Discussion.
limitations: This paper is not applicable as there are no problems in terms of limitations or potential societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposed a computationally efficient algorithm to perform leave-one-out cross-validation using the support vector machine with Gaussian kernel. The authors proposed a two-stage cross-validation that is applicable to the support vector machine with the bias term.
strengths_and_weaknesses: Strengths:
* Nice and detailed theoretical analysis.

Weaknesses:
* Empirical results on small data sets.
questions: Major questions:
* For a support vector machine classifier, while training if we remove a data point that is not a support vector and add a data point that is not a support vector from the support vector machine trained using the full data, the solution does not change. Is this property used in the implementation?
* Can the proposed algorithm be applied to support vector regression algorithm with the $\epsilon$-insensitive error function?
* Can the proposed algorithm be applied to arbitrary kernels with unbounded values such as polynomial kernels? 

Minor questions:
* Is there a typo in the equation just after line 109?
limitations: Yes.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a data reduction strategy and an algorithm that learns an SVM and a leave-one-out-cross-validated version much faster than the standard scheme which would require training once for every fold. The paper exploits the sparsity within the solution process to reduce the amount of computation required. The paper describes the theory around this and several experiments showing greatly superior running times from the scheme.
strengths_and_weaknesses: Strengths:
1. The development is clearly an important development for SVMs, which are an important class of algorithms. It represents an original and clever development.
2. The paper has convincing experimental results, showing that the new algorithm achieves nearly the same objective values but in much lower running time.

Weaknesses:
1. The paper should provide the algorithm in pseudocode form to ease reproducibility due to the multiple parts of the algorithm. If space is a concern, then the key optimization problems or equations can be put in the pseudocode and then cited in the text.
2. There should be more analysis of how effective the data reduction strategy suggested by equation 5 is. Clearly it is effective as demonstrated by the running times. However, I think the authors should still show what fraction of computations is eliminated due to the strategy and give a sense of the factors in the dataset that are most responsible for reduction in computations.
questions: 1. Do the authors have some sense of how much computation the data reduction strategy saves and the characteristics of the problem that relate the most with computation reduction? This is in line with what I listed as the second weakness of the paper.
limitations: The authors clearly addressed the limitations of their work. There is no potential negative societal impact of their work that they can account for and affect.
review_title: null

Meta-review: metareview: The paper proposes a new algorithm for training kernel support vector machines (SVM), that exploits a leave one out lemma to develop a significantly faster algorithm. The key advances are a data reduction approach, a consolidated cross validation method, and a warm start. Empirical comparisons to magicsvm, kernlab, and LIBSVM demonstrate an order of magnitude speedup. The paper proposes a major advance to kernel SVMs, without resorting to approximations or linearization. The combination of theoretical insight, exploitation of the structure of the method, as well as careful reuse of computation are all excellent contributions.

Four reviewers considered the submission, and three reviewers were very pleased by the novel approach for a classic algorithm. One reviewer found several issues with the submission, and did not further engage during the rebuttal period. A different reviewer actually was positively influenced by the author response to the negative review.

It gives me great pleasure to recommend this paper for acceptance to NeurIPS 2022. Congratulations!
confidence: Certain
Decision: Accept