Title: Linear Label Ranking with Bounded Noise
Authors: Dimitris Fotakis
Alkis Kalavasis
Vasilis Kontonis
Christos Tzamos
Venue: NeurIPS 2022
Keywords: Label Ranking
Noise
Gaussian
Linear Sorting Function

Abstract: Label Ranking (LR) is the supervised task of learning a sorting function that maps feature vectors $x \in \mathbb{R}^d$ to rankings $\sigma(x) \in \mathbb S_k$ over a finite set of $k$ labels. We focus on the fundamental case of learning linear sorting functions (LSFs) under Gaussian marginals: $x$ is sampled from the $d$-dimensional standard normal and  the ground truth ranking $\sigma^\star(x)$ is the ordering induced by  sorting the coordinates of the vector $W^\star x$, where  $W^\star \in \mathbb{R}^{k \times d}$ is unknown. We consider learning LSFs in the presence of bounded noise: assuming that a noiseless example is of the form $(x, \sigma^\star(x))$, we observe $(x, \pi)$, where for any pair of elements $i \neq j$, the probability that the order of $i, j$ is different in $\pi$ than in  $\sigma^\star(x)$ is at most $\eta < 1/2$. We design efficient non-proper and proper learning algorithms that  learn hypotheses within normalized Kendall's Tau distance $\epsilon$ from the ground truth  with $N= \widetilde{O}(d\log(k)/\epsilon)$ labeled examples and runtime $\mathrm{poly}(N, k)$. For the more challenging top-$r$ disagreement loss, we give an efficient proper learning algorithm that achieves $\epsilon$ top-$r$ disagreement with the ground truth with $N = \widetilde{O}(d k r /\epsilon)$ samples and $\mathrm{poly}(N)$ runtime.

TL;DR: We study the problem of learning linear sorting functions in the presence of bounded noise.

Review discussion:
Strengths:
The paper studies a natural problem in learning rankings. The problems seem like natural extensions of learning halfspaces with Massart noise, and a good model for learning rankings with noise. From a technical perspective, the approach is natural and the geometric lemmas interesting. The paper is also well-written.
Weaknesses:
I don't really see any strong weaknesses in the paper.
I don't have any pressing questions.
The work is purely theoretical at this point, and seems to have no potential negative societal impact.
No
We would like to thank the reviewer for carefully reading our manuscript and for the useful and positive feedback!
The work makes a significant contribution by proposing the first efficient algorithm for learning of LSFs with bounded noise. The basic algorithmic ingredient is an efficient learner ([ZSA20]) for the class of halfspaces (for the special case of k=2).
However, the algorithm is generalized to any k (improperly), and is further used to obtain a proper learner using the ellipsoid method.
When the error is measured by top-r disagreement loss, the proper learner also achieves improved sample complexity comparing to a naive invocation of the improper learner. The paper is very well-written with technical highlights appropriately placed and the analysis is sound.
The current work studies bounded noise type. Is there any possibility or evidence of extending it to more challenging noise types, e.g. adversarial noise?
Minor issue: Some notations are not well defined.
For example, the sample complexity N could be distinct for different algorithms. r was used both for the top-r problem and for the radius in Claim 2.
The work does not have negative social impacts.
No
We thank the reviewer for carefully reading our manuscript and providing insightful feedback (and also some interesting questions!).
"The current work studies bounded noise type. Is there any possibility or evidence of extending it to more challenging noise types, e.g. adversarial noise?"
That is indeed a very interesting question for direct future research.
While in general adversarial noise models have strong lower bounds, in the presence of Gaussian marginals or more generally structured distributions, approximate learnability is known to be achievable for binary classifiers with adversarial label noise.  It is very interesting to investigate whether such approximate learning results can be obtained for multiclass/label ranking settings under structured distributions.  We believe that the techniques developed in this work can serve as a starting point towards this goal.
Minor issue: Some notations are not well defined.
For example, the sample complexity N could be distinct for different algorithms. r was used both for the top-r problem and for the radius in Claim 2.
We will fix this (and potential similar issues) in the first revision of our work.
Originality: I am not an expert in this area so I am not entirely sure about other related work. The proposed algorithms and getting them to work (as in proving guarantees for them) are quite non-trivial and so the paper is quite original in my opinion.
Quality: The submission is technically sound. All claims are well-supported with proofs.
Clarity: The submission is clearly written and well-organized.
Significance: The paper is the first to study a very natural problem and so I think it is quite significant. Ranking functions have many applications and developing robust algorithms for learning ranking functions can have good practical impact. On the theoretical front, these problems are also clearly of interest to the NeurIPS community. As mentioned on page 2 of the paper, the case of $k = 2$ captures the problem of learning halfspaces with Massart noise - the best paper award winner of NeurIPS 2019 was on this topic.
Not any as of now.
This is primarily a theoretical paper and so the authors have mentioned that it doesn't have any negative social impact.
No
We would like to thank the reviewer for carefully reading our manuscript and for appreciating our work and results!
Strengths:
- Presentation: the problem is well introduced and the main results are clearly presented
- Impact: the results established seem to be of general interest in addition to solve the label ranking problem
- The paper is technically sound.
Weaknesses:
- No experimental results limit the impact of the work.
- Clarity: although the first two sections are very clear, the second half of the paper feels harder to follow. It does not feel clear to me whether the stated algorithms are solutions to the problem with KT Distance or with top-r Disagreement, or both.
- What is the intuition behind the power 6 in theorem 1 and 2?
- Is Algorithm 2 used for both Theorem 1 and 2?
The theoretical limitations are adequately addressed. The authors state that the potential negative societal impacts of their work is N/A due to its theoretical nature. It might still be valuable to mention what could go wrong if the suggested algorithms were actually deployed.
No
We thank the reviewer for recognizing the importance of the noisy label ranking problem and the fact that the work is clearly presented and written. We would like to respond to the reviewer's questions and comments inline, as follows:
"No experimental results limit the impact of the work."
The main goal of this work is to provide the first efficient algorithms for learning noisy LSFs with provable guarantees.  We agree with the reviewer that experimental evaluation is interesting and we plan to further investigate and implement our algorithms in a future work.
"Clarity: although the first two sections are very clear, the second half of the paper feels harder to follow. It does not feel clear to me whether the stated algorithms are solutions to the problem with KT Distance or with top-r Disagreement, or both.
Is Algorithm 2 used for both Theorem 1 and 2?"
Yes, our proper learning Algorithm 2 is used for both Theorem 1 and Theorem 2. Let us be more specific.
In the KT distance learning, our goal is to control the expected KT distance between the true LSF and our output ranking hypothesis. Algorithm 2 attains this result by first obtaining a collection of $\binom{k}{2}$ linear classifiers (each one associated with the pair $1 \leq i < j \leq k$ and obtained by solving an instance of the Massart halfspace problem) and then by compressing this collection of vectors $v_{ij}$ into a single matrix $W \in \mathbb{R}^{k \times d}$ using a convex program. This algorithm uses $d \log k / \epsilon$ samples to efficiently compute the parameter matrix $W$ that gets $\epsilon$-close to $\Delta_{KT}$ distance. The key lemma that makes this idea work is Lemma 2 in Line 254 which relates the expected KT distance with the angle metric of the two matrices (see also Equation (1) in Line 139). Our Algorithm 2 essentially gives an upper bound on this angle metric.
When we shift our objective and our goal is to control the top-r disagreement, we can still apply Algorithm 2 which essentially controls the angle metric (see Line 139). The crucial ingredient that is missing is the relation between the loss we have to control, i.e., the expected top-r disagreement and the angle metric of Equation (1) in Line 139. This relation is one of the main technical contributions of this work and is presented in Lemma 1, which essentially says that the expected top-r disagreement is at most $O(kr)$ times this angle metric.
Hence, in order to get top-r disagreement of order $\epsilon$, it suffices to apply our Algorithm 2 with $\epsilon’ = O(\epsilon / (kr) )$. This is essentially how the sample complexity of $\widetilde{O}(dkr/\epsilon)$ is obtained.
We will make this more clear in the first revision of our work.
"What is the intuition behind the power 6 in theorem 1 and 2?"
This result comes from an application of an algorithm that learns halfspaces with Massart noise from [ZSA20]. Since we are reducing the problem of learning LSFs to learning binary linear classifiers with noise, any improvement on this bound for binary classification with Massart noise would also improve the corresponding dependence on $1-2 \eta$ in our setting. Intuitively the reason that there is a power on the $(1-2\eta)$ is that all algorithms for learning halfspaces with Massart noise essentially rely on localizing (conditioning) on an area close to the decision boundary of probability inverse proportional to $\mathrm{poly}(1 - 2 \eta)$.

Meta-review:
role: AC
data: metareview: The reviewers are unanimous in their strong positive opinion on this paper.  The authors have given the first efficient algorithms for learning noisy linear sorting functions with theoretical guarantees a relevant and useful problems setup for the NeuRIPS community.  The reviewers consider the paper clear and well-presented and thus this is a natural accept.

role: PC

Decision: Accept