Title: GRAPH CONVOLUTIONAL NETWORKS FOR LEARNING WITH FEW CLEAN AND MANY NOISY LABELS

Abstract: In this work we consider the problem of learning a classifier from noisy labels when a few clean labeled examples are given. The structure of clean and noisy data is modeled by a graph per class and Graph Convolutional Networks (GCN) are used to predict class relevance of noisy examples. For each class, the GCN is treated as a binary classifier learning to discriminate clean from noisy examples using a weighted binary cross-entropy loss function, and then the GCN-inferred "clean" probability is exploited as a relevance measure. Each noisy example is weighted by its relevance when learning a classifier for the end task. We evaluate our method on an extended version of a few-shot learning problem, where the few clean examples of novel classes are supplemented with additional noisy data. Experimental results show that our GCN-based cleaning process significantly improves the classification accuracy over not cleaning the noisy data and standard few-shot classification where only few clean examples are used. The proposed GCN-based method outperforms the transductive approach (Douze et al., 2018) that is using the same additional data without labels.

1. INTRODUCTION:

State-of-the-art deep learning methods require a large amount of manually labeled data. The need for supervision may be reduced by decoupling representation learning from the end task and/or using additional training data that are unlabeled, weakly labeled (with noisy labels), or belong to different domains or classes. Example approaches are transfer learning (Wang & Gupta, 2015) , unsupervised representation learning (Wang & Gupta, 2015) , semi-supervised learning (Weston et al., 2008) , learning from noisy labels (Joulin et al., 2016) and few-shot learning (Snell et al., 2017) .

Learning from noisy labels allows using large-scale data and labels from the web without human annotation effort. Most work focuses on learning the representation jointly with the end task, assuming there is still a considerable amount of clean labeled data (Patrini et al., 2017; Lee et al., 2018; Li et al., 2017) . However, for a number of classes only very few or even no clean labeled examples might be available at the representation learning stage. Few-shot learning limits the labeled data to very few on the end task, while the representation is learned on a large training set of different classes (Hariharan & Girshick, 2017; Snell et al., 2017; Vinyals et al., 2016) . Nevertheless, in many situations, more data with noisy labels are available or can be acquired for the end task.

One interesting mix of few-shot learning with additional large-scale data is the work of Douze et al. (2018) , where labels are propagated from few clean labeled examples to a large-scale collection. This collection is unlabeled and actually contains data of many more classes than the end task. Their method overall improves the classification accuracy, but at an additional computational cost; it is a transductive method, i.e., instead of learning a parametric classifier, the large-scale collection is still necessary at inference.

In this work, we learn a classifier from few clean labeled examples and additional weakly labeled data, while the representation is learned on different classes, as in few-shot learning. We assume the class names are known, and we use them to search an existing large collection of images with textual description. The result is a set of images with potentially relevant, but noisy labels. As shown in Figure 1 , we clean this data using a graph convolutional network (GCN) (Kipf & Welling, 2017) , which learns to predict a class relevance score per image based on the source (clean vs. noisy) of its connections in the graph. Both the clean and the noisy images are then used to learn a classifier, We make the following contributions:

• We learn a classifier on a large-scale weakly-labeled collection jointly with only few clean labeled examples. • To our knowledge, we are the first to use a GCN to clean noisy data: we cast a GCN as a binary classifier learning to discriminate clean from noisy data, and we use its inferred probabilities for the "clean" class as a relevance score per example. • We apply our method to a few-shot learning benchmark and show significant improvement in accuracy, while outperforming the method by Douze et al. (2018) using the same large-scale collection of data.

2. RELATED WORK:

Learning with noisy labels is often concerned with estimating or learning a transition matrix (Natarajan et al., 2013; Patrini et al., 2017; Sukhbaatar et al., 2014) or knowledge graph (Li et al., 2017) between labels and correcting the loss function, which does not apply in our case since the classes in the noisy data are unknown. Most recent work on learning from large-scale weakly-labeled data focuses on learning the representation e.g. by metric learning (Lee et al., 2018; Wang et al., 2018a) , bootstrapping (Reed et al., 2015) , or distillation (Li et al., 2017) . In our case however, since the clean labeled examples are few, we need to keep the representation mostly fixed.

Dealing with the noise, e.g. by thresholding (Lee et al., 2018) , outlier detection (Wang et al., 2018a) or reweighting (Liu & Tao, 2015) , is applicable while the representation is learned, based e.g. on the gradient of the loss (Ren et al., 2018b) . In contrast, the relatively-shallow GCN that we propose effectively decouples reweighting from both representation learning and classifier learning. Learning to clean the noisy labels (Veit et al., 2017) typically assumes adequate human verified labels for training, which again is not the case in this work.

Few-shot learning. Meta-learning (Vilalta & Drissi, 2002) refers to learning at two levels, where generic knowledge is acquired before adapting to more specific tasks. In few-shot learning, this translates to learning on a set of base classes how to learn from few examples on a distinct set of novel classes without overfitting. For instance, optimization meta-learning (Finn et al., 2017; 2018; Ravi & Larochelle, 2017) amounts to learning a model that is easy to fine-tune in few steps. In our work, we study an extension of few-shot learning where more data are available on novel classes, reducing the risk of overfitting when fine-tuning the model. Metric learning approaches learn how to compare queries for instance to few examples (Vinyals et al., 2016) or to the corresponding class prototypes (Snell et al., 2017) . Hariharan & Girshick (2017) and Wang et al. (2018b) (Chen et al., 2019) confirms the superiority of the cosine classifier to previous work including meta-learning (Finn et al., 2017) . We use the cosine classifier in this work, both for base and novel classes. All of the above use only the few labeled examples of the novel classes.

Making use of unlabeled data has been little explored in few-shot learning until recently. Ren et al. (2018a) introduce a semi-supervised few-shot classification task, where some labels are unknown. Liu et al. (2019) follow the same semi-supervised setup, but use graph-based label propagation (LP) (Zhou et al., 2003a) for classification and consider jointly all test images. These methods assume a meta-learning scenario, where only small-scale data is available at each training episode; arguably, such a small amount of data limits the representation adaptation and generalization to unseen data. Similarly, Rohrbach et al. (2013) use label propagation in a transductive setting, but at a larger scale assuming that all examples come from a set of known classes. Douze et al. (2018) extend to even larger scale, leveraging 100M unlabeled images in a graph without using additional text information.

We focus on the latter large-scale scenario using the same 100M dataset. However, we filter by text to obtain noisy labels and follow an inductive approach by training a classifier for novel classes, such that the 100M collection is not needed at inference.

Graph neural networks are generalizations of convolutional networks to non-Euclidean spaces (Bronstein et al., 2017) . Early spectral methods (Bruna et al., 2014; Henaff et al., 2015) have been succeeded by Chebyshev polynomial approximations (Defferrard et al., 2016) , which avoid the high computational cost of computing eigenvectors. Graph convolutional networks (GCN) (Kipf & Welling, 2017) provide a further simplification by a first-order approximation of graph filtering and are applied to semi-supervised (Kipf & Welling, 2017) and subsequently to few-shot learning (Garcia & Bruna, 2018) . In Kipf & Welling (2017), the loss function is applied to labeled examples to make predictions on unlabeled ones. Similarly in Garcia & Bruna (2018) , GCNs make predictions on novel class examples. Gidaris & Komodakis (2019) use Graph Neural Networks as denoising autoencoders to generate class weights for novel classes. In contrast, we cast GCNs as binary classifiers discriminating clean from noisy examples: we apply a loss function to all examples, and then use the inferred probabilities as a class relevance measure, effectively cleaning the data.

Our counter-intuitive objective of treating all noisy examples as negative can be compared to treating each example as a different class in instance-level discrimination (Wu et al., 2018) . In fact, our loss function is similar to noise-contrastive estimation (NCE) (Gutmann & Hyvärinen, 2010) used in that work. According to our experiments, our GCN-based classifier outperforms classical LP (Zhou et al., 2003a) used for a similar purpose by Rohrbach et al. (2013) .

3. PROBLEM FORMULATION:

We consider a space X of examples. We are given a set

X L ⊂ X of examples, each having a clean (manually verified) label in a set C L of classes with |C L | = K L .

For any set X ⊂ X , we denote by X c its subset of examples having a label in class c. We assume that the number

|X c L | of examples labeled in each class c ∈ C L is only k, typically in {1, 2, 5, 10, 20}. We are also given an additional set X c Z of examples, each with a set of noisy labels in C L . The extended set of examples for class c is now X c E = X c L ∪ X c Z .

Examples or sets of examples having clean (noisy) labels are referred to as clean (noisy) as well. The goal is to train a classifier, using the additional noisy set in order to improve the accuracy compared to only using the small clean set.

We assume that we are given a feature extractor g θ : X → R d , mapping an example to a d-dimensional vector. For instance, when examples are images, the feature extractor is typically a convolutional neural network (CNN) and θ are the parameters of all layers.

In this work, we assume that the noisy set X Z is collected via web crawling with examples that are images accompanied with free-form text description and/or user tags originating from community photo collections. To make use of text data, we assume that the names of classes in C L are given. An example in X Z is given a label in class c ∈ C L if its textual information contains the name of class c; it may then have none, one or more labels. In this way, we automatically infer labels for X Z without human effort, which are however noisy.

4. CLEANING WITH GRAPH CONVOLUTIONAL NETWORKS:

We perform cleaning by predicting a class relevance measure for each noisy example in X c Z , independently per class c ∈ C L . To simplify notation, we drop superscript c where possible in this subsection and we denote X c E by {x 1 , . . . , x k , x k+1 , . . . , x N }, where X c L = {x 1 , . . . , x k } and X c Z = {x k+1 , . . . , x N }. The features of these examples are similarly represented by matrix

V = [v 1 , . . . , v k , v k+1 , . . . , v N ] ∈ R d×N , where v i = g θ (x i ) for i = 1, . . . , N .

We construct an affinity matrix A ∈ R N ×N with elements

a ij = [v i v j ] + if examples v i and v j are reciprocal nearest neighbors in X c

E and 0 otherwise. Matrix A has zero diagonal, but self-connections are added and then A is normalized as Ã = D -1 (A + I) with D = diag((A + I)1) being the degree matrix of A + I and 1 the all-ones vector.

Graph convolutional networks (GCNs) (Kipf & Welling, 2017) are formed by a sequence of layers. Each layer is a function

EQUATION

where Z ∈ R l×N represents the input features, Θ ∈ R l×n holds the parameters of the layer to be learned, and h is a nonlinear activation function. Function f Θ maps l-dimensional input features to n-dimensional output features.

In this work we consider a two layer GCN with a scalar output per example. This network is a function

EQUATION

where

Θ = {Θ 1 , Θ 2 }, Θ 1 ∈ R d×m , Θ 2 ∈ R m×1 , [•]

+ is the positive part or ReLU function (Nair & Hinton, 2010) and σ(x) = (1 + e -x ) -1 for x ∈ R is the sigmoid function. Function F Θ performs feature propagation through the affinity matrix in an analogy to classical graph-based propagation methods for classification (Zhou et al., 2003a) or search (Zhou et al., 2003b) .

The output F Θ ( Ã, V ) is a vector of length N , with element F Θ ( Ã, V ) i in [0, 1] representing a relevance value of example x i for class c. To learn the parameters Θ, we treat the GCN as a binary classifier where target output 1 corresponds to clean examples and 0 to noisy. In particular, we minimize the loss function

L G (V, Ã; Θ) = - 1 k k i=1 log F Θ ( Ã, V ) i - λ N -k N i=k+1 log 1 -F Θ ( Ã, V ) i .

(3) Test accuracy without and with additional data using class prototypes ( 6) is shown next to class names.

example into account independently of the others, while the GCN considers the collection of examples as a whole. MLP training is performed identically to GCN by minimizing (3). We compare the two alternatives in our experiments.

5. LEARNING A CLASSIFIER WITH FEW CLEAN AND MANY NOISY EXAMPLES:

Our cleaning process applies when the clean labeled examples are few, but assumes a feature extractor g θ . That is, representation learning, label cleaning and classifier learning are decoupled. We follow few-shot learning in that we learn the representation by supervised classification on a set of base classes, obtaining g θ , and then solving new classification tasks on a distinct set of novel classes.

In these new tasks, we assume few clean and many noisy labels as specified in Section 3, perform GCN-based cleaning as described in Section 4, and learn a classifier by weighing examples according to class relevance. Representation and classifier learning are described below.

5.1. COSINE-SIMILARITY BASED CLASSIFIER:

We use a cosine-similarity based classifier (Gidaris & Komodakis, 2018; Qi et al., 2018) , or cosine classifier for short. Given classes C with |C| = K, each class c ∈ C is represented by a learnable parameter w c ∈ R d . The prediction of example x ∈ X is the class c of maximum cosine similarity

EQUATION

where W = [w 1 , . . . , w K ] ∈ R d×K . Prior to learning classifiers for novel classes, training examples x i ∈ X c Z are weighted by their relevance r(x i ) to class c. For a noisy example x i ∈ X c E , we define r(x i ) = F Θ ( Ã, V ) i where F Θ ( Ã, V ) is the output vector of the GCN, while for a clean example

5.2. REPRESENTATION LEARNING:

EQUATION

x i ∈ X c L we fix r(x i ) = 1. Note that optimizing (3) does not guarantee F Θ ( Ã, V ) i = 1 for clean examples x i ∈ X c

L . We define r(X) = x∈X r(x) for any set X ⊂ X .

We first assume that we no longer have access to examples of base classes in new classification tasks and consider two different classifiers, class prototypes and cosine-similarity based classifier. Then, this assumption is dropped and the classifier and feature representation are learned jointly by fine-tuning the entire network.

Class prototypes. For each novel class c ∈ C L , we define prototype w c by

EQUATION

Prototypes are fixed vectors, not learnable parameters. Collecting them into matrix 

W L = [w 1 , . . . , w K L ] ∈ R d×K L , K L -

EQUATION

while the parameters θ of the feature extractor are fixed. The scale parameter s is also fixed to the value obtained during base class learning. Prediction on novel only or all classes is then made as in the previous case.

Deep network fine-tuning. We now drop the assumption that base class examples are not accessible and, given all examples X A = X B ∪ X E , we jointly learn the parameters θ of the feature extractor and W A = (W B , W L ) of the K A -way cosine classifier for all classes by minimizing loss function

EQUATION

)

Note that, due to overfitting on the few available examples, such learning is avoided in a few-shot learning setup. In a few cases, it takes the form of fine-tuning including all base class data (Qi et al., 2018) , or only lasts for a few iterations when the base class data is not accessible (Finn et al., 2017) . We use the YFCC100M dataset (Thomee et al., 2016) as a source for additional data with noisy labels. It contains approximatively 100M images collected from Flickr. Each image comes with a text description obtained from the user title and caption. We use the text description to obtain images with noisy labels. as discussed in Section 3. This process results in very different numbers of additional examples per class, with a minimum of zero for classes maillot and missile, and a maximum of 620,142 for class church/church building.

Representation and classifier learning. In most experiments, we use ResNet-10 (He et al., 2016) as feature extractor as in Gidaris & Komodakis (2018) . Classification for novel classes is performed with class prototypes (6), cosine classifier learning (7) or deep network fine-tuning (8). Hyper-parameters such as batch size and number of epochs, are tuned on the validation set. Possible values are 2048, 4096, and 8192 for batchsize and 10, 30 and 50 for number of epochs. The learning rate starts from 0.1 and is reduced to 0.001 at the end of training with cosine annealing (Loshchilov & Hutter, 2017) .

We handle the imbalance of the noisy set by normalizing by r(X c ) in ( 7). Prototypes ( 6) are used to initialize W L of cosine classifier in (7), and the learned W L is used to initialize the corresponding part of W A when fine-tuning the network by ( 8). In the latter case, we train all layers for 10 epochs with learning rate 0.01. We ignore examples x i with relevance r(x i ) < 0.1 to reduce the complexity when fine-tuning the network.

We also report results with ResNet-50 as feature extractor, using the model trained on base classes by Hariharan & Girshick (2017) . Following Douze et al. (2018) , we apply PCA to the features to reduce their dimensionality to 256. Base classes are represented by class prototypes (6) in this case.

GCN training is performed with Adam optimizer and a learning rate of 0.1 for 100 iterations. We use dropout with probability 0.5. The dimensionality of the input descriptors is d = 512 for ResNet-10 and d = 256 for ResNet-50 (after PCA). Dimensionality of the internal representation in ( 1) is m = 16. The affinity matrix is constructed with reciprocal top-50 nearest neighbors.

Baselines. We implement and evaluate several baseline methods. β-cleaning assigns r(x i ) = β to all additional examples. We report results for β = 1.0 (unit relevance score) and β * , the optimal β for all k obtained on the validation set. MLP, discussed in Section 4, learns a nonlinear mapping to assign relevance, but does not propagate over the graph. Label Propagation (LP) (Zhou et al., 2003a) propagates information by a linear operation. It solves the linear system (I -αD -1/2 AD -1/2 )r c = y c (Iscen et al., 2017) for each class c, where D is the degree matrix of A, α = 0.9 and y c ∈ R N is a k-hot binary vector indicating the labeled examples of class c. Relevance r(x i ) is then the i-th element (r c ) i of the solution.

6.2. EXPERIMENTAL RESULTS:

The impact of importance weight λ is measured on the validation set and the best performing value is used on the test set for each value of k. Results are shown in Appendix A. The larger the value of λ, the more the loss encourages noisy examples to be classified as negatives. As a consequence, large (small) λ results in smaller (larger) relevance, on average, for noisy examples. The optimal λ per value of k suggests that the fewer the clean examples the larger the need for additional ones.

Comparison with baselines using additional data is presented in Table 1 . The use of additional data is mostly harmful for β-weighting except for 1 and 2-shot. MLP offers improvements in most cases, implying that it manages to appropriately downweigh irrelevant examples. The consistent improvement of our method compared to MLP, especially large for small k, suggests that it is beneficial to incorporate relations, with the affinity matrix A modeling the structure of the feature space. LP is a classic approach that also uses A but is a linear operation with no parameters, and is inferior to our method. The gain of cleaning (β = 1 vs. ours) ranges from 11% to 20%. Comparison with the state of the art is presented in Table 2 . We significantly improve the performance by using additional data and cleaning compared to a number of different approaches, including the work by Gidaris & Komodakis (2018) , which is our starting point. As expected, the gain is more pronounced for small k, reaching more than 20% improvement for 1-shot novel accuracy.

Closest to ours is the work of Douze et al. (2018) , who use the same experimental setup and the same additional data, but without filtering by text and using noisy labels. We outperform their approach in all cases, while requiring much less computation: offline, we construct a separate small graph per class rather than a single graph over the entire 100M collection; online, we perform inference by cosine similarity to one prototype per class or a learned classifier rather than iterative diffusion on the entire collection. Note that by ignoring examples that are not given any noisy label, we are only using a tiny fraction of the 100M collection: in particular, only 3,744,994 images for the 311-class test split of the Low-shot ImageNet benchmark. In contrast to Douze et al. (2018) , additional data brings improvement even at 20-shot with classifier learning or network fine-tuning. Most importantly, our approach does not require the entire 100M collection at inference.

7. CONCLUSIONS:

In this paper we have introduced a new method for assigning class relevance to noisy images obtained by textual queries with class names. Our approach leverages one or a few labeled images per class and relies on a graph convolutional network (GCN) to propagate visual information from the labeled images to the noisy ones. The GCN is a binary classifier discriminating clean from noisy examples using a weighted binary cross-entropy loss function and inferring "clean" probability as a relevance measure for that class. Experimental results show that using noisy images weighted by this relevance measure significantly improves the classification accuracy. 

:

We denote the

2-normalized counterpart of vector x by x. Similarly, if y = f (x), denote ŷ by f (x).

Figures and Tables:

Figure 1: Overview of our cleaning approach for 1-shot learning with noisy examples. We use the class name admiral to crawl noisy images from web and create an adjacency graph based on visual similarity. We then assign a relevance score to each noisy example with a graph convolutional network (GCN). Relevance scores are displayed next to the images.

Figure 2: Examples of clean images (left) for 1-shot classification, cumulative histogram of the predicted relevance for noisy images (middle), and representative noisy images (right), each having its position in the (descending) ranked list according to relevance and relevance value reported below.Test accuracy without and with additional data using class prototypes (6) is shown next to class names.

: BASE CLASSES We are given a set X B ⊂ X of examples, each having a clean label in a set of base classes C B with |C B | = K B . These data are used to learn a feature representation, i.e. a feature extractor g θ , by learning a K B -way base-class classifier for unseen data in X . The parameters θ of the feature extractor and W B of the classifier are jointly learned by minimizing the cross entropy loss

where σ : R K → R K is the softmax function with σ(a) c = e ac / j∈C e aj for a ∈ R K , s is a learnable scale parameter and ŴB = [ ŵ1 , . . . , ŵK B ] ∈ R d×K B . Learning and inference are performed on base classes by L B (C B , X B ; θ, W B ) (5) and π θ,W B (4), respectively. As a result, learned feature extractor parameters θ are used for base or novel classes, while the classifier parameters W B can be used for base class or all-class classification, as discussed below.5.3 NEW CLASSIFICATION TASKS: NOVEL OR ALL CLASSESEach new task is related to a set of novel classes C L , disjoint from C B . The goal is to learn a K L -way novel-class classifier or a K A -way classifier on all classes C A = C B ∪ C L for unseen data in X , where K A = K B + K L . Unlike the typical few-shot learning task, each novel class contains few clean and many noise examples.

way prediction on novel classes is made by classifier π θ,W L (4), while K A -way prediction on all (base and novel) classes by π θ,W A , where W A = [W B , W L ] and W B is learned according to L B (C B , X B ; θ, W B ) (5) and then kept fixed. Cosine classifier learning. Similarly to Section 5.2, given clean and noisy novel-class examples X E , we learn a parametric cosine classifier with parameters W L = [w 1 , . . . , w K L ] ∈ R d×K L by minimizing the weighted cross entropy loss L L (C L , X E ; θ, W L ) over W L , where

SETUPDatasets and task setup. We extend the Low-Shot ImageNet benchmark introduced byHariharan & Girshick (2017) by assuming many noisy examples for novel classes, in addition to the few clean ones. In this benchmark, the 1000 ImageNet classes(Russakovsky et al., 2015) are split into 389 base classes and 611 novel classes. The validation set contains 193 base and 300 novel classes, and the test set the remaining 196 base and 311 novel classes. The standard benchmark includes k-shot classification, i.e. classification on k clean examples per class, which we extend to k clean and many noisy examples per class, with k ∈ {1, 2, 5, 10, 20}. Similar to Hariharan & Girshick (2017) we perform 5 tasks, each drawing a subset of k clean examples per class. We report the average top-5 accuracy over the 5 tasks on novel or all classes of the test set.

Figure 3: (a) Number of additional images per class c sampled from YFCC-100M for all novel classes of Low-Shot ImageNet. (b) Number of classes per group, when |X c Z | is sampled logarithmically into groups. (c) Accuracy improvement ∆ Acc (difference of accuracy between our method with noisy examples and the baseline without noisy examples) for prototype classifier, for same groups as in (b).

learn how to generate novel-class examples, which is not needed when more data are actually available.

Table content: <table><tr><td>class training set. A recent survey</td></tr><tr><td>Gidaris &amp; Komodakis (2018) learn on base classes a simpler cosine similarity-based parametric</td></tr><tr><td>classifier, or simply cosine classifier, without meta-learning. The same classifier has been introduced</td></tr><tr><td>independently by Qi et al. (2018), who further fine-tune the network, assuming access to the base</td></tr></table>

Given the propagation on the nearest neighbor graph, and depending on the relative importance λ of the second term, noisy examples that are strongly connected to clean ones are still expected to receive high class relevance, while noisy examples that are not relevant to the current class are expected to get a class relevance near zero.The impact of parameter λ is validated in Section 6, where we show that the fewer the available clean images are (smaller k) the smaller the importance weight should be. As is standard practice for GCNs in classification(Kipf & Welling, 2017), training is performed in batches of size N , that is the entire set of features.Figure2shows examples of clean images, corresponding noisy ones and the predicted relevance. Thanks to the visual similarity to the clean image, we can use relevance to resolve cases of polysemy, e.g. black widow (spider) vs. black widow (superhero), or cases like pineapple vs. pineapple juice.

Table content: <table><tr><td/><td>830</td><td/><td/><td/><td/><td/><td/></tr><tr><td/><td/><td>(1, 1.00)</td><td>(2, 1.00)</td><td>(4, 1.00)</td><td>(5, 1.00)</td><td>(50, 1.00)</td><td>(100, 0.99)</td><td>(500, 0.12)</td></tr><tr><td>0</td><td>0.5</td><td>1</td><td/><td/><td/><td/><td/></tr><tr><td colspan="2">black widow: 61.6 → 85.6</td><td>(824, 0.06)</td><td>(825, 0.06)</td><td>(826, 0.06)</td><td>(827, 0.06)</td><td>(828, 0.06)</td><td>(829, 0.06)</td><td>(830, 0.06)</td></tr><tr><td/><td>8,297</td><td/><td/><td/><td/><td/><td/></tr><tr><td/><td/><td>(1, 1.00)</td><td>(2, 1.00)</td><td>(4, 1.00)</td><td>(5, 1.00)</td><td>(50, 1.00)</td><td>(100, 0.96)</td><td>(500, 0.54)</td></tr><tr><td>0</td><td>0.5</td><td>1</td><td/><td/><td/><td/><td/></tr><tr><td colspan="2">pineapple: 27.2 → 43.2</td><td colspan="7">(8291, 0.00) (8292, 0.00) (8293, 0.00) (8294, 0.00) (8295, 0.00) (8296, 0.00) (8297, 0.00)</td></tr><tr><td colspan="9">Discussion. Loss function (3) is similar to noise-contrastive estimation (NCE) (Gutmann &amp; Hyväri-</td></tr><tr><td colspan="9">nen, 2010) as used by Wu et al. (2018) for instance-level discrimination, whereas we discriminate</td></tr></table>

Comparison with baselines using noisy examples. We report top-5 accuracy on novel classes with classification by class prototypes (6).

Table content: <table><tr><td>Method</td><td/><td>k=1</td><td/><td>2</td><td>5</td><td/><td>10</td><td>20</td><td/><td/></tr><tr><td/><td/><td colspan="3">FEW CLEAN EXAMPLES</td><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="2">Class proto. Gidaris &amp; Komodakis (2018)</td><td colspan="2">45.3±0.65</td><td>57.1±0.37</td><td colspan="2">69.3±0.32</td><td>74.8±0.20</td><td colspan="2">77.8±0.24</td><td/></tr><tr><td/><td/><td colspan="6">FEW CLEAN &amp; MANY NOISY EXAMPLES</td><td/><td/><td/></tr><tr><td>β-weighting, β = 1</td><td/><td colspan="2">56.1±0.06</td><td>56.4±0.08</td><td colspan="2">57.1±0.05</td><td>57.7±0.08</td><td colspan="2">58.7±0.06</td><td/></tr><tr><td>β-weighting, β  *</td><td/><td colspan="2">55.6±0.24</td><td>58.3±0.14</td><td colspan="2">63.4±0.25</td><td>67.5±0.34</td><td colspan="2">71.0±0.22</td><td/></tr><tr><td colspan="2">Label Propagation Zhou et al. (2003a)</td><td colspan="2">62.6±0.35</td><td>67.0±0.41</td><td colspan="2">74.6±0.30</td><td>76.3±0.23</td><td colspan="2">77.7±0.18</td><td/></tr><tr><td>MLP</td><td/><td colspan="2">63.6±0.41</td><td>68.8±0.42</td><td colspan="2">73.9±0.25</td><td>75.6±0.21</td><td colspan="2">77.6±0.21</td><td/></tr><tr><td>Ours</td><td/><td colspan="2">67.8±0.10</td><td>70.9±0.30</td><td colspan="2">73.7±0.17</td><td>76.1±0.12</td><td colspan="2">78.2±0.14</td><td/></tr><tr><td>METHOD</td><td/><td colspan="3">NOVEL CLASSES</td><td/><td/><td colspan="2">ALL CLASSES</td><td/><td/></tr><tr><td/><td>k=1</td><td>2</td><td>5</td><td>10</td><td>20</td><td>k=1</td><td>2</td><td>5</td><td>10</td><td>20</td></tr><tr><td/><td colspan="7">RESNET-10 -FEW CLEAN &amp; MANY NOISY EXAMPLES</td><td/><td/><td/></tr><tr><td>Ours -class proto. (6)</td><td colspan="10">67.8±0.10 70.9±0.30 73.7±0.20 76.1±0.16 78.2±0.14 70.3±0.05 72.1±0.18 74.1±0.12 75.6±0.13 76.9±0.09</td></tr><tr><td>Ours -cosine (7)</td><td colspan="10">73.2±0.14 75.3±0.25 75.6±0.24 78.5±0.32 80.7±0.26 71.9±0.07 74.0±0.23 76.5±0.16 78.3±0.23 80.2±0.18</td></tr><tr><td>Ours -fine-tune (8)</td><td colspan="10">74.6±0.13 76.6±0.26 78.2±0.23 80.9±0.34 82.9±0.20 76.0±0.10 77.3±0.13 78.7±0.19 80.7±0.25 82.2±0.14</td></tr><tr><td/><td colspan="5">RESNET-50 -FEW CLEAN EXAMPLES</td><td/><td/><td/><td/><td/></tr><tr><td>Proto.-Nets (Snell et al., 2017)</td><td>49.6</td><td>64.0</td><td>74.4</td><td>78.1</td><td>80.0</td><td>61.4</td><td>71.4</td><td>78.0</td><td>80.0</td><td>81.1</td></tr><tr><td>PMN w/ H (Wang et al., 2018b)</td><td>54.7</td><td>66.8</td><td>77.4</td><td>81.4</td><td>83.8</td><td>65.7</td><td>73.5</td><td>80.2</td><td>82.8</td><td>84.5</td></tr><tr><td/><td colspan="7">RESNET-50 -FEW CLEAN &amp; MANY UNLABELED EXAMPLES</td><td/><td/><td/></tr><tr><td>Diffusion (Douze et al., 2018)</td><td colspan="6">63.6±0.61 69.5±0.60 75.2±0.40 78.5±0.34 80.8±0.18 -</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Diffusion -logistic (Douze et al., 2018)</td><td colspan="6">64.0±0.70 71.1±0.82 79.7±0.38 83.9±0.10 86.3±0.17 -</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td/><td colspan="7">RESNET-50 -FEW CLEAN &amp; MANY NOISY EXAMPLES</td><td/><td/><td/></tr><tr><td>Ours -class proto. (6)</td><td colspan="10">69.7±0.44 73.7±0.56 77.0±0.20 79.9±0.30 81.9±0.29 73.8±0.33 76.6±0.36 78.9±0.19 80.8±0.21 82.2±0.14</td></tr><tr><td>Ours -cosine (7)</td><td colspan="10">78.0±0.38 80.2±0.33 80.9±0.17 83.7±0.19 85.7±0.11 77.6±0.26 79.1±0.20 79.9±0.09 82.1±0.22 83.8±0.11</td></tr><tr><td>Ours -fine-tune (8)</td><td colspan="10">80.8±0.25 83.0±0.23 83.8±0.39 86.4±0.23 88.5±0.20 81.6±0.20 83.2±0.16 84.3±0.23 86.2±0.17 87.8±0.03</td></tr></table>

Comparison to the state of the art on the Low-shot ImageNet benchmark. We report top-5 accuracy on novel and all classes. We use class prototypes (6), cosine classifier learning (7) and deep network fine-tuning (8) for classification with our GCN-based data addition method.

Table content: <table/>

Peer review (iclr2020):
The paper makes a significant attempt at solving one of the practical problems in machine learning -- learning from many noisy and limited number of clean labels. This setting is presumably more practical than the setting of few-shot learning. Noisy labels are often abundantly available and investing in methods that can take the noise into account for building a discriminative model is quite timely. To be honest, the theoretical contribution of the paper is limited. The authors make use of the nearest neighbour graph obtained from a reduced-dimensional set of features to compute the weights of the noisy labels that must guide the predictive model. From this perspective, the paper seems like an application of existing tools (such as CNN, graph convolutional network and binary classification). However, that does not undermine the superior results the authors have received in the novel application they have targeted. I appreciate the effort that went validating these ideas with real-world datasets. In future, I would like to see a joint approach to such training, where the function g(), the nearest neighbour graph loss and the classification loss are all tied in the same objective function and are optimized jointly. The paper has few really minor grammatical errors and typos. Please fix those before uploading the final draft.