Graph convolutional networks for learning with few clean and many noisy labels
Authors: Ahmet Iscen
Giorgos Tolias
Yannis Avrithis
Ondrej Chum
Cordelia Schmid
Venue:  (2020)

Abstract: In this work we consider the problem of learning a classifier from noisy labels when a few clean labeled examples are given. The structure of clean and noisy data is modeled by a graph per class and Graph Convolutional Networks (GCN) are used to predict class relevance of noisy examples. For each class, the GCN is treated as a binary classifier learning to discriminate clean from noisy examples using a weighted binary cross-entropy loss function, and then the GCN-inferred "clean" probability is exploited as a relevance measure. Each noisy example is weighted by its relevance when learning a classifier for the end task. We evaluate our method on an extended version of a few-shot learning problem, where the few clean examples of novel classes are supplemented with additional noisy data. Experimental results show that our GCN-based cleaning process significantly improves the classification accuracy over not cleaning the noisy data and standard few-shot classification where only few clean examples are used. The proposed GCN-based method outperforms the transductive approach (Douze et al., 2018) that is using the same additional data without labels.

Under review as a conference paper at ICLR 2020
GRAPH CONVOLUTIONAL NETWORKS FOR LEARNING WITH FEW CLEAN AND MANY NOISY LABELS
Anonymous authors Paper under double-blind review
ABSTRACT
In this work consider the problem of learning classifier from noisy labels when a few clean labeled examples are given. The structure of clean and noisy data is modeled by graph per class and Graph Convolutional Networks (GCN) are used to predict class relevance of noisy examples. For each class, the GCN is treated as binary classifier learning to discriminate clean from noisy examples using a weighted binary cross cross-entrppy loss function, and then the GCN-inferred "clean" probability is exploited as a relevance measure. Each noisy example weighted by its relevance when learning a classifier for the end task. We evaluate our method on an extended version of few-shot learning problem, where the few clean examples of novel classes are supplemented with additional noisy data. Experimental results show that our GCN-based cleaning process significantly improves the classification accuracy over not cleaning the noisy data and standard few-shot classification where only few clean examples are used. The proposed GCN-based method outperforms the transductive approach (Douze al. 2018) that is using the same additional data without labels.
INTRODUCTION
State-of-the arrt deep learning methods require large amount of manually labeled data. The need for supervision may be reduced by decoupling representation learning from the end task and/or using additional training data that are unlabeled, weakly labeled (with noisy labels), or belong different domains classes. Example approaches are transfer learning (Wang & Gupta 2015), unsupervixed representation learning (Wang& & Gupta 2015), semi-supervised learning (Weston et al. 2008), learning from noisy labels (Joulin et al. 2016) and few-shot learning (Snell al. 2017).
Learning from noisy labels allows using large-scale data and labels from the web without human annotation effort. Most work focuses on learning the representation jointly with the end task, assuming there is still a considerable amount of clean labeled data (Patrini et al., 2017 Lee et al., 2018 Li clean labeled examples
et al. 2017). However, for number of classes only very few might be available at the representation learning stage. Few-shol learning limits the labeled data to very few on the end task, while the representation is learned a large training set of different classes (Hariharan & Girshick 2017 Snell et al. [0177] Vinyals et al. 2016). Nevertheless, in many situations, more data with noisy labels are available or can be acquired for the end task.
One interesting mix of few-shot learning with additional large-scale data is the work Douze et al. (2018), where labels are propagated from few clean labeled examples to large-scale collection. This collection is unlabeled and actually contains data of many more classes than the end task. Their method overall improves the classification accuracy, but at an additional computational cost; it is a transductive method, i.e., instead of learning a parametric classifier, the large-scale collection is still necessary at inference.
In this work, we learn a classifier from few clean labeled examples and additional weakly labeled data, while the representation is learned on different classes, as in few-shot learning. We assume the class names are known, and we use them to search an existing large collection of images with textual description. The result is a set of images with potentially relevant, but noisy labels. As shown in Figure Il we clean this data using a graph convolutional network (GCN) (Kipf & Welling 2017). which learns to predict a class relevance score per image based on the source (clean vs. noisy) of its connections in the graph. Both the clean and the noisy images are then used to learn a classifier,

---

Under review as a conference paper at ICLR 2020
Labeled example
1. Adjacency graph per class
Class relevance prediction with GCN
3. Relevance score output
Query admiral
E.7
Use for classifier training
2. Minimise
Loss clean
2. Loss noisy
clasvity
chaaily
Additional data
postitoes
negatives
Figure 1: Overview our cleaning approach for 1-shot learning with noisy examples. We use the class name admiral to crawl noisy images from web and create adjacency graph based on visual similarity. We then assign a relevance score to each noisy example with graph convolutional network (GCN). Relevance scores are displayed next the images.
where the noisy examples weighted by relevance. Unlike most existing work, our method operates independently per class and applies when clean labeled examples are few or even one per class.
We make the following contributions:
We learn a classifier on large-scale wealdy-labeled collection jointly with only few clean
labeled examples.
To our knowledge, we are the first to use a GCN to clean noisy data: we cast a GCN as binary classifier learning discriminate clean from noisy data, and we use its inferred probabilities for the "clean" class as a relevance score per example.
We apply our method to few-shot learning benchmark and show significant improvement in accuracy, while outperiorming the method by Douze et al. (2018) using the same large-scale collection of data.
RELATED WORK
Learning with noisy labels is often concerned with estimating or learning transition matrix Natara jan al., 2013; Patrini et al., 2017 Sukhbaatar et al. 2014) or knowledge graph (Li et al. 2017) between labels and correcting the loss function, which does not apply in our case since the classes
the noisy data are unknown. Most recent work on learning from large-scale weakly-labeled data focuses learning the representation e.g. by metric learning (Lee et al.) 2018 Wang et al. 2018a), bootstrapping (Reed et al. 2015), or distillation (Li et al. 2017). our case however, since the clean labeled examples are few, need to keep the representation mostly fixed.
Dealing with the noise, e.g. by thresholding (Lee et al. 2018), outlier detection (Wang et al.) 2018a) Sa) reweighting (Liu & Tao 2015). is applicable while the representation is learned, based e.g. the gradient of the loss (Ren et al. 2018b). In contrast, the relatively-shalle effectively decouples reweighting from both representation learning and classifier learning. Learning to clean the noisy labels (Veit et al. 2017) typically assumes adequate human verified labels for training, which again not the case this work.
GCN that we propose
Few-shot learning. Meta-learning (Vilalta & Drissi 2002) refers to learning at two levels, where generic knowledge acquired before adapting to more specific tasks. In few-shot learning. this translates to learning
set of base classes how to leam from few examples on a distinct set of novel classes without overfitting, For instance, optimization meaa-nnn (Finn al., 2017 2018 Ravi & Larochelle 2017) amounts to learning a model that is easy to fine-tune in few steps. In our work, we study an extension of few-shot learning where more data are available on novel classes, reducing the risk of overfitting when fine-tuning the model. Metric learning approaches learn how to compare queries for instance to few examples (Vinyals et al.) (016) or to the corresponding class prototypes (Snell al. 2017). Hariharan & Girshick (2017) and |aag et al. (2018b) learn how to generate novel-class examples, which is not needed when more data are actually ly available.
Gidaris & Komodakis (2018) learn base classes simpler cosine similarity based parametric classifier, or simply cosine classifier, without meta-learning. The same classifier has been introduced independently by al. (2018), who further fine-tune the network, assuming access the base
2

---

Under review as a conference paper at ICLR 2020
class training set. A recent survey (Chen et al., 2019) confirms the superiority of the cosine classifier to previous work including meta-learning (Finn et al., 2017). We use the cosine classifier in this work, both for base and novel classes. All of the above use only the few labeled examples of the novel classes.
Making use of unlabeled data has been little explored in few-shot learning until recently. Ren et al. (2018a) introduce a semi-supervised few-shot classification task, where some labels are unknown. Liu et al. (2019) follow the same semi-supervised setup, but use graph-based label propagation (LP) (Zhou et al. 2003a) for classification and consider jointly all test images. These methods assume meta-learning scenario, where only small-scale data is available at each training episode; arguably, such a small amount of data limits the representation adaptation and generalization to unseen data. Similarly, RRhhbacc et al. (2013) use label propagation in a transclctive setting. but at larger scale assuming that all examples come from a set of known classes. Douze et al. (2018) extend to even larger scale, leveraging 100M unlabeled images in a graph without using additional text information. We focus on the latter large-scale scenario using the same 100M dataset. However, we filter by text to obtain noisy labels and follow an inductive approach by training a classifier for novel classes, such that the 100M collection is not needed at inference.
Graph neural networks are generalizations of convolutional networks to non- non-Euclidean spaces (Bronstein et al., 2017). Early spectral methods (Bruna et al., 2014:) Henaff et al.) (0015) have been succeeded by Chebyshev polynomial approximations (Defferrard et al. 2016), which avoid the high computational cost of computing cigenvectors. Graph convolutional networks (GCN) (Kipf & Welling (017) provide a further simplification by a firrstorddrr approximation of graph filtering and applied to semi-supervised (Kipl & Welling 2017) and subscquently to few-shot learning (Garcia & Bruna 2018). In Kipf & Welling (2017). the loss function is applied to labeled examples to make predictions unlabeled ones. Similarly Garcia & Bruna (2018), GCNs make predictions on novel class examples. Gidaris & Komodakis (2019) use Graph Neural Networks as denoising autoencoders to generate class weights for novel classes, In contrast, we cast GCNs as binary classifiers discrimi- nating clean from noisy examples: we apply a loss function to all examples, and then use the inferred probabilities as a class relevance measure, effectively cleaning the data.
Our counter-intuitive objective of treating all noisy examples as negative can be compared to treating each example as a different class in instance-level discrimination (Wu et al. 2018). In fact, our loss function is similar to noise-contrastive estimation (NCE) (Gutmann & Hyvärinen (0100) used in that work. According to our experiments, our GCN-based classifier outperforms classical LP (Zhou et al.,
2003a) used for similar purpose by RRohrbach et al. (2013).
PROBLEM FORMULATION
We consider a space X' of examples. We are given a set Xx. C X' of examples, each having a clean (manually verified) label in a set C2 of classes with |Czl = Ke. For any set X C X, we denote by X its subset of examples having a label in class c. We assume that the number [X&] of examples labeled in each class E Cc is only k. typically in {1, 2, 5, 10, 20). We are also given an additional set X of examples, each with a set of noisy labels in Cc. The extended set of examples for class e now X X2 U X2. Examples or sets of examples having clean (noisy) labels are referred to as clean (noisy) as well. The goal is to train a classifier, using the additional noisy set in order to improve the accuracy compared to only using the small clean set.
We assume that we are given feature extractor go X - Rd, mapping an example to a d-dimensional vector. For instance, when examples are images, the feature extractor is typically a convolutional neural network (CNN) and 0 are the parameters of all layers.
In this work, we assume that the noisy set Xz is collected via web crawling with examples that are images accompanied with free-form text description and/or user tags originating from community photo collections. To make use of text data, we assume that the names of classes in Cc are given. An example in Xx is given a label in class E Cc if its textual information contains the name of class e; it may then have none, one or more labels. In this way, we automatically infer labels for Xz without human effort, which are however noisy.
3

---

Under review as a conference paper at ICLR 2020
4 CLEANING WITH GRAPH CONVOLUTIONAL NETWORKS
We perform cleaning by predicting a class relevance measure for each noisy example in X2. in- dependently per class Cc. To simplify notation, we drop superseript c where possible in this subsection and we denote X by fz1,..., Tk, ;x where X2 {zI..... Fz and X5 (UE+1),..,,NN] The features of these examples are similarly represented by matrix V [1 M.:.... Va, -e+1. VN] E RaxA, where V, go(x)) for = I....... N.
We construct an affinity matrix A E NxN with elements d; = [v vj]+ if examples v, and v, are reciprocal nearest neighbors in X and 0 otherwise. Matrix A has zero diagonal, but self-connections are added and then A is normalized as = D (A++)) with D diag((A I)1) being the degree matrix of A + and 1 the all-ones vector.
Graph convolutional networks (GCNs) (Kipf & Welling 2017) are formed by a sequence of layers. Each layer is function fe IR NxN x 1 T + R of the form
(A, 2) 1(@T ZA),
(1)
where Z E RIXN represents the input features, θ E R holds the parameters of the layer to be learned, and h is nonlinear activation function. Function fe maps I-dimensional input features to n-dimensional output features.
In this work we consider a two layer GCN with a scalar output per example. This network is a function Fg RNNN x R R given by
Fe V) = [0
where θ - (2). E R_xm O2 E Rm [+]+ is the positive part or ReLU function (Nair & Hinton 2010) and ((x) (1+ for x E I is the sigmoid function. Function Fe performs feature propagation through the affinity matrix in an analogy to classical graph-based propagation methods for classification (Zhou et al.] 2003a) or search (Zhou et al., 20036).
The output Fe (A, V) is vector of length N, with element (A, V); in [0. 1] representing a relevance value of example v; for class c. To learn the parameters O we treat the GCN binary classifier where target output 1 corresponds to clean examples and 0 to noisy. In particular, we minimize the loss function
k
Lo(V)A;0) = - log (F)(A,
Σ 1=1
Σ log (1- (0 (I- i=++1
(3)
K
This is binary cross-entroys loss function where noisy examples are given an importance weight Given the propagation on the nearest neighbor graph, and depending on the relative importance À of the second term, noisy examples that are strongly connected clean ones are still expected receive high class relevance, while noisy examples that are not relevant to the current class are expected to get a class relevance near zero.
The impact of parameter À is validated in Section 0 where we show that the fewer the available clean images are (smaller k) the smaller the importance weight should be. As is standard practice for GCNs in classification (Kipf & Welling 2017). training is performed in batches of size N, that is the entire set of features.
Figure 2 shows examples of clean images, corresponding noisy ones and the predicted relevance. Thanks to the visual similarity to the clean image, we can use relevance to resolve cases of polysemy, e.g. black widow (spider) vs. black widow (superhero), cases like pineapple vs. pineapple juice.
Discussion, Loss function (3) is similar to noise-contrustive estimation (NCE) (Gutmann & Hyväri nen 2010) used by Wu et al. (2018) for instance-level discrimination. whereas discriminate clean from noisy examples. The semi-supervised learning setup of GCNs (Kipf & Welling 2017)) uses loss function that applies only the labeled examples, and makes discrete predictions unlabeled examples. In our case, all examples contribute to the loss but with different importance, while we infer real-valued class relevance for the noisy examples, to be used for subsequent learning.
Function Fe in (2) reduces to a Multi-Layer Perceptron (MLP) when the affinity matrix is zero, in which case all examples disconnected. Using MIP to perform cleaning would take each
4

---

Under review as a conference paper at ICLR 2020
(1. 1.000
(2. L.00)
65. 1.001
(500,,
black widow: 61.6 85.6
(824, (1.0))
(825, 0.005
(826, D.06))
(827, (1.00)
(828, (.00))
(82S) (.06)
(8,10, 0.00)
8,297
1.00)
1.000
1.00
1.00)
(100 (500,
MAR
pincopple: 27.2 43.2
(8291, 0.001 (8292, D.OK) (6293, D.DI)
(87044 (U00)
(8295, 0.000 (8296, D.00) (8297, 0,00)
Figure 2: Examples of clean images (left) for 1-shot classification, cumulative histogram of the predicted relevance for noisy images (middle), and representative noisy images (right), each having its position in the (descending) ranked list according to relevance and relevance value reported below. Test accuracy without and with additional data using class prototypes shown next to class names.
example into account independently of the others, while the GCN considers the collection of examples whole. MLP training is performed identically to GCN by minimizing (3). We compare the two alternatives in our experiments.
5 LEARNING A CLASSIFIER WITH FEW CLEAN AND MANY NOISY EXAMPLES
Our cleaning process applies when the clean labeled examples are few, but assumes feature extractor go- That is, representation learning, label cleaning and classifier learning are decoupled. We follow few-shot learning in that we learn the representation by supervised classification on set of base classes, obtaining ge. and then solving new classification tasks on a distinct set of novel classes. In these new tasks, we assume few clean and many noisy labels as specified in Section 3 perform GCN-based cleaning as described in Section 4. and learn classifier by weighing examples according to class relevance. Representation and classifier learning are described below.
5.1 COSINE-SIMILARITY BASED CLASSIFIER
We use a simiiae-imilarity based classifier Gidaris & Komodakis 2018 Qi et al. 2018 or cosine classifier for short. Given classes C' with |C| K, each class C is represented by learnable parameter w. E Rdd The prediction of example x E X' is the class e of maximum cosine similarity w go
N0,w (x) arg max w (x)):
where W = [Wi...., wk] E Raxk
5.2 REPRESENTATION LEARNING: BASE CLASSES
XB C X' of examples, each having clean label in of base classes CE with Csl = Kg. These data are used to lear a feature representation, i.e. feature extractor ga- by learning a g-wway base-class classifier for unseen data in X. The parameters f of the feature extractor and WB of the classifier are jointly learned by minimizing the cross entropy loss
We are given
Le(CB, XB; 0,))
Σ cECs
log(o (x))))),
(5)
X
xEXA
where : K + KK the softmax function with (aa) ea. = Exece for E R is a learnable scale parameter and Wg = [w....., WKel E gdx Learning and inference are
We denote the ég-normalized counterpart of vector by Similarly, if f(x), denote by f(x).
5

---

Under review as a conference paper at ICLR 2020
performed on base classes by Lg(CE XB; W&) (5 and Ne, Wa feature extractor parameters 0 are used for base or novel classes, while the classifier parameters WB can be used for base class or all-class classification, discussed below.
respectively. As result, learned
5.3 NEW CLASSIFICATION TASKS: NOVEL OR ALL CLASSES
Each new task is related to set of novel classes Cc, disjoint from Cg. The goal is to learn a K-y novel-class classifier or a K classifier on all classes CA = Cs Cc for unseen data in X, where a KB Kc. Unlike the typical few-shot learning task, each novel class contains few clean and many noise examples.
Prior to learning classifiers for novel classes, training examples E X2 are weighted by their relevance r(z;) to class c. For noisy example 2; X we define r(z;) (A, V), where Fe (A, V) the output vector of the GCN, while for clean example z; E X we fix r(x;) 1. Note that optimizing (3) does not guarantee (e V); = 1 for clean examples æ; X2. We define r(X) Exex r(z) for any set X C.
We first assume that we no longer have access to examples of base classes new classification tasks and consider two different classifiers, class prototypes and Then, this assumption is dropped and the classifier and feature representation are learned jointly by fine-tuning the entire network.
based classifier.
Class prototypes. For each novel class c E Cc. we define prototype We by
Σ r(z)go(z).
We r(XE)
(6)
Prototypes are fixed vectors, not learnable parameters. Collecting them into matrix Wc [Wi...., WKel Rd Kg-way prediction on novel classes is made by classifier ne.W. w. while Ka-way prediction on all (base and novel) classes by Te,Wa. where W = [Ws. W and Wg learned according to (s( XB; 0 Ws) and then kept fixed.
Cosine classifier learning. Similarly to Section 5.2, given clean and noisy novel-class examples Xg. we learn parametric cosine classifier with parameters We [Wi...,, wK2l Ra by minimizing the weighted cross entropy loss T.c Xe; 0, We) over We, where
Le Xe:C,,X;;000 Xe;0, - ΣT Σ
Σ1(9)
r(x) (o
r(xg)
cEC2
ZEX
while the parameters 0 of the feature extractor are fixed. The scale parameter also fixed the value obtained during base class learning. Prediction on novel only or all classes is then made as in the previous case.
Deep network fine-tuning. We now drop the assumption that base class examples are not accessible and, given all examples XAA = Xg Xg, we jointly learn the parameters 9 of the feature extractor and WA= (WB. Wc) of the Кд-waa cosine classifier for all classes by minimizing loss function
WA) = Ls(Cs X 0, WB) + Le(Ce, Xg; 0, Wc).
(8)
Note that, due to overfitting on the few available examples, such learning avoided few-shot learning setup. In few cases, it takes the form of fine-tuning including all base class data (Qi et al. 2018), or only lasts for a few iterations when the base class data is not accessible (Finn et al. 2017).
EXPERIMENTS
6.1 EXPERIMENTAL SETUP
Datasets and task setup. We extend the Low-Shot ImageNet benchmark introduced by Hariharan & Girshick (2017) by assuming many noisy examples for novel classes, in addition to the few clean ones. In this benchmark, the 1000 ImageNet classes (Russakovsky al. 2015) are split into 389

---

Under review as a conference paper at ICLR 2020
base classes and 611 novel classes. The validation set contains 193 base and 300 novel classes, and the test set the remaining 196 base and 311 novel classes. The standard benchmark includes shot classification, i.e. classification noisy examples per class, with & E {1, 2, 5, 10, 20). Similar to Hariharan & Girshick] (2017) we perform 5 tasks, each drawing subset of ls clean examples per class. We report the average top-5 accuracy over the 5 tasks on novel or all classes of the test set.
& clean examples per class, which we extend to k clean and many
We use the YFCCI00M dataset (Thomee al., 2016) as a source for additional data with noisy labels. It contains approximatively 100M images collected from Flickr. Each image comes with a text description obtained from the user title and caption. We use the text description obtain images with noisy labels. as discussed in Section 3 This process results in very different numbers of additional examples per class, with a minimum of zero for classes maillot and missile, and a maximum of 620,142 for class church/church building.
Representation and classifier learning. In most experiments, we use ResNet-10 (He et al. 2016) as feature extractor as in Gidaris Komodakis
Classification for novel classes is performed with class prototypes (6), cosine classifier learning 7 or deep network fine-tuning (8). Hyper-parameter such as batch size and number of epochs, are tuned on the validation set. Possible values are 2048, 4096, and 8192 for batchsize and 10, 30 and 50 for number of epochs. The learning rate starts from 0.1 and is reduced to 0.001 at the end of training with cosine annealing (Loshchilov & Hutter (017))
We handle the imbalance of the noisy set by normalizing by r(X.) in 77. Prototypes 6 used to initialize We of cosine classifier in 7 and the learned We is used to initialize the corresponding part of WA when fine-tuning the network by (8). In the latter case, we train all layers for 10 epochs with learning rate 0.01. We ignore examples z; with relevance r(x,) < 0.1 to reduce the complexity when fine-tuning the network.
We also report results with ResNet-50 as feature extractor, using the model trained on base classes by Harihhaann & Girshick (2017). Following Douze et al. (2018), we apply PCA the features to reduce their dimensionality to 256. Base classes are represented by class prototypes (6) in this case.
GCN training is performed with Adam optimizer and a learning rate of 0.1 for 100 iterations. We use dropout with probability 0.5. The dimensionality of the input descriptors is d = 512 for ResNet-10 and d 256 for ResNet-50 (after PCA). Dimensionality of the internal representation in (I is m 16. The affinity matrix is constructed with reciprocal top-50 nearest neighbors.
Baselines. We implement and evaluate several baseline methods. B-cleaning assigns r(z,) = B to all additional examples. We report results for 3 1.0 (unit relevance score) and ,9,, the optimal / for all & obtained on the validation set. MLP, discussed in Section 4 learns a nonlinear mapping to
assign relevance, but does not propagate over the graph. Label Propagation (L.P) (Zhou et al. 2003a)
propagates information by a linear operation. It solves the linear system (I oD 1eD AD
Dr
ye (Iscen et al. (017)) for each class c, where D is the degree matrix of A, a = 0.9 and y. E R a k-hot binary vector indicating the labeled examples of class e. Relevance r(x:;) then the i-th element (r=); of the solution.
6.2 EXPERIMENTAL RESULTS
The impact of importance weight À is measured on the validation set and the best performing value used on the test set for each value of k. Results are shown in Appendix [A] The larger the value of X, the more the loss encourages noisy examples to be classified as negatives. As a consequence, large (small) results in smaller (larger) relevance, on average, for noisy examples. The optimal per value of k suggests that the fewer the clean examples the larger the need for additional ones.
Comparison with baselines using additional data is presented in Table 1 The use of additional data is mostly harmful for B-weighting except for and 2-shot. MLP offers improvements in most cases, implying that it manages to appropriately downweigh irrelevant examples. The consistent improvement of our method compared to MLP, especially large for small k, suggests that it is beneficial to incorporate relations, with the affinity matrix A modeling the structure of the feature space. LP is a classic approach that also uses A but is a linear operation with no parameters, and is inferior to method. The gain of cleaning (B 1 vs. ours) ranges from 11 % to 20%.
7

---

Under review as a conference paper at ICLR 2020
Method
k=1
2
5
10
20
FRW CLEAN EXAMPLES
proto Gidaris Kommdakis 2018
45.3-005
57.140.37
69.3420.2
74.84:2.0
77.84:024
FEW CLEAN & MANY NOISY EXAMPLES
B-weighting, / =
56,120006
56.420008
57.1:1005
57.7:10.8
58.7:1:0.06
[ Beweighting,
55.6=0024
58.3=0.14
63.4:0255
67.540034
71.01:022
Label Propagation Zhou et al. 2003a)
62,6220335
67.0=0.41
74.6:10.0
76.34003
77.74:018
MLP
63.6::0.41
68.8::0.42
73.9
75.6:1021
77.6:1021
Ours
67.8-0.10
7..9-0.30
73.7+0.17
76.1+0.12
78.2+-0.14
Table I: Comparison with baselines using noisy examples. We report top-5 accuracy on novel classes with classification by class prototypes (
Mirmon
NOVEL Cl CLASSIS
ALL CLASSES
ROSNET-10
FRw CLPAN EXAMPLES
Proto: Nots
Snell al. 2017
39.3
54.4
66.3
712
73.9
49.5
61.0
69.7
72.9
74,6
Logistic reg
40.7
62.0
59.4
67.6
PMN w/ H
Wang & Jim 20153
45.8
57.8
69.0
74.3
77.4
57.6
64.7
71.9
75.2
71.5
Class proto. Oiuars & K Kunnoakks 2018
45.3=au
57.1
69.3
74.
47.8
$7,0=am
64.7
72.5
75.8
77.4-2.9
Class proto.
2018
45.8.a% 57,4
69.6
75.0sa09
78.2.435
58.1aa
65.2g815
72.9
76.6011
78.8:314
RESNET-10
FEW CLEAN & MANY NOISY EXAMPLES
Ouns class proto.
67.8 43.9 73.7
16.1 78.2 a34
a0.3:aa:
12. 74.1
75.6:a1
76.9.
Ours cosine
73.2 ms4 75.3
75.61924
78.5 80.7
71.9:mm:
74.0
76.5
78.3 80.2
Ours fine tunc
74.6-115
76.6-124
78.24423
80.9
76.0=am 77.3-
78.7
80.7
RoSNEr--0 Frw CLEAN EXAMPLES
Prets Sncll al. 2017
19.6
61.0
74.1
78.1
80.0
61.4
71.4
78.0
80.0
81.1
PMN H Wing
77.4
73.5
80.2
RESNET-50 FEW CLEAN & MANY UNLABELED EXAMPLES
Diffusion Douze
2018
63.6uasi 69.5 75.2a040 78.51aH 808.4
Diffusion Tognte Doize
2018
64.0=amo 71.1ataz 79.74m08 83.9 86.3-a17
RESNET-50 Few CLRAN & MANY NoisY EXAMPLRS
Ours class proto. Ours cosine
69.7 73,7=a5s 77.0 19.9 81.9
73.8:a15
76.6-134 78.9
80.5
82.2
78,0=am
80.9 83.7 85.7=a11
77.6:226
79.1-429 79.9 a3
Ours
fine- tune
$0.8 aas 83.0 83.8 86.4 88.5 42) $16: 83.2 84.3 86.2 87.8
Table 2: Comparison to the state of the art on the Low-shot ImageNet benchmark. We report top-5
accuracy on novel and all classes. We use class prototypes (6), cosine classifier learning 7 and deep network fine-tuning for classification with our CCCN-ased data addition method.
Comparison with the state of the art is presented in Table 2 We significantly improve the perfor- mance by using additional data and cleaning compared to a number of different approaches, including the work by Gidaris & Komodakis (2018), which is our starting point. As expected, the gain is more pronounced for small k, reaching more than 20% improvement for 1-shot novel accuracy.
Closest to ours is the work of Douze et al. (2118), who use the same experimental setup and the same additional data, but without filtering by text and using noisy labels. We outperform their approach in all cases, while requiring much less computation: offline, we construct a separate small graph per class rather than single graph over the entire 100M collection; online, we perform inference by cosine similarity to one prototype per class or a learned classifier rather than iterative diffusion on the entire collection. Note that by ignoring examples that are not given any noisy label, are only using a tiny fraction of the 100M collection: in particular, only 3.744.994 images for the 31 1-class test split of the Low-shot ImageNet benchmark. In contrast to Douze et al. (2018), additional data brings improvement even at 20-shot with classifier learning or network fine-tuning. Most importantly, our approach does not require the entire 100M collection at inference.
CONCLUSIONS
In this paper we have introduced new method for assigning class relevance noisy images obtained by textual queries with class names. Our approach leverages one or a few labeled images per class and relies on graph convolutional network (GCN) to propagate visual information from the labeled images to the noisy ones. The GCN is a binary classifier discriminating clean from noisy examples using a weighted binary cross-entropy loss function and inferring "clean" probability as a relevance measure for that class. Experimental results show that using noisy images weighted by this relevance measure significantly improves the classification accuracy.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: going beyond cuclidean data. IEEE Signal Processing Magazine, 34(4):18-4) 2017.
Joan Bruna, Wojeiech Zaremba, Arthur Szlam, and Yann Lecun. Spectral networks and locally connected networks on graphs. In ICLR, 2014.
Wei-Yu Chen, Yea-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. ICLR, 2019.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks graphs with fast localized spectral filtering. In NewrlPS, 2016.
Matthijs Douze. Arthur Szlam, Bharath Hariharan, and Hervé Jégou. Low-shot learning with large-scale diffusion. In CVPR, 2018.
Chelsea Finn, Picter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic modcl-agnostic meta-leamiin. In NeurlPS, 2018.
Victor Garcia and Joan Bruna. Few-shot learning with graph neural networks. In ICLR, 2018.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In CVPR, 2018.
Spyros Gidaris and Nikos Komodakis. Generating classification weights with gnn denoising autoencoders for few-shot learning. In CVPR, 2019.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: estimation principle for unnormal- ized statistical models. In International Conference on Artificial Intelligence and Statistics, 2010.
Bharath Hariharan and Ross Girshick. Low-shot visual recognition by shrinking and hallucinating features. In CVPR, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Mikael Henaff, Joan Bruna, and Yann LeCun. Deep convolutional networks on graph-structured data. arXiv preprint arXiv:1506. 05163, 2015.
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, Teddy Furon, and Ondrej Chum. Efficient diffusion on region manifolds: Recovering small objects with compact representations. In CVPR, 2017.
Armand Joulin, Laurens van der Maaten, Allan Jahri, and Nicolas Vasilache. Learning visual features from large weakly supervised data. ECCV, 2016.
Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
Kuang-Hude Lee, Xiaodong He, Zhang. and Linjun Yang. CleanNet: Transfer learning scalable image classifier training with label noise. CVPR, 2018.
Yuncheng Li. Jianchao Yang. Yale Song, Liangliang Cao, Jiebo Luo, Li-Jia Learning from noisy labels with distillation. In /CCV, 2017.
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Trans. PAMI, 38(3), 2015.
Yanbin Liu, Juho Lec, Minseop Park, Sachoon Kim, Eunho Yang, Sung Ju Hwang, and Yi Yang. Learning to propagate labels: Transductive propagation network for few-shot learning. In ICLR, 2019.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. ICLR, 2017.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines, In ICML, 2010.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari, Learning with noisy labels. In NeurlPS, 2013.

---

Under review as a conference paper at ICLR 2020
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In CVPR, 2017.
Hang Qi, Matthew Brown, and David G. Lowe. Low-shot learning with imprinted weights. In CVPR, June 2018.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In ICLR, 2017.
Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. ICLR, 2015.
Mengye Ren, Sachin Ravi, Eleni Triantafillou, Jake Snell, Kevin Swersky, Josh B. Tenenbaum, Hugo Larochelle, and Richard Zemel. Meta-learning for semi-supervised few-shot classification. ICLR, 2018a.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In ICML, 2018b.
Marcus Rohrbach, Sandra Ebert, and Bernt Schiele. Transfer learning in a transductive setting. In NeurIPS, 2013.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 115(3), 2015.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NeurlPS, 2017.
Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, and Rob Fergus. Training convolutional networks with noisy labels. arXiv preprint 1406:::400.0000 2014.
Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfc=100m: The new data multimedia research. Commun, ACM, 59(2), 2016.
Andreas Veit, Neil Alldrin, Gal Chechik, Ivan Krasin, Abhinav Cupta, and Serge Belongie. Learning from noisy large-scale datasets with minimal supervision. In CVPR, July 2017.
Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-learning. Artificial intelligence review, 18(2)777-95, 2002.
Oriol Vinyals, Charles Blundell, Tim Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In NewrlPS, 2016.
Xiaolong Wang and Abhinay Gupta. Unsupervised learning of visual representations using videos. In CVPR, 2015.
Yisen Wang, Weiyang Liu, Xingjun Ma. James Bailey. Hongyuan Zha, Le Song, and Shu-Tao Xia. Iterative learning with open-set noisy labels. In CVPR, 2018a.
Yu-Xiong Wang, Ross Girshick, Martial Hebert, and Bharath Hariharan. Low-shot learning from imaginary data. In CVPR, 2018b.
Jason Weston, Frédéric Ratle, and Ronan Collobert. Deep learning via supervisd embedding. In ICML, 2008.
Zhirong Wu, Yuanjun Xiong, Stella Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance-level discrimination. CVPR, 2018
Dengyong Zhou, Olivier Bousquet, Thomas Navin Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. In NeurlPS, 2003a.
Dengyong Zhou, Jason Weston, Arthur Gretton, Olivier Bousquet, and Bernhard Schölkopf. Ranking on data manifolds. In NewrlPS, 2003b.
10

---

Under review as a conference paper at ICLR 2020
10c
60
0.3
[x (log scale)
of classes
0.2
40
Ace
10a
10"
-0.1
100
200
300
10
10a
10a
10a
10a
10a
10a
(class id), sorted by XX
|x
|X
(a)
(b)
(c)
Figure (a) Number of additional images per class sampled from YFCC-100M for all novel classes of Low-Shot ImageNet. (b) Number of classes per group, when |X']| is sampled logarithmically into groups. (c) Accuracy improvement Ace (difference of accuracy between our method with noisy examples and the baseline without noisy examples) for prototype classifier, for same groups as in (b).
A APPENDIX
Noisy data statistics. We present statistics about the noisy examples of novel classes and the improvements of our method per class. Figure 3 (a) shows that the noisy examples for novel classes are long tailed (in log scale). There is significant number of classes where we end up with less than 1000 extra examples, but we improve nevertheless; see Figure 3 (c). A small exception is very rare classes out of 311, with around 3 additional images per class (leftmost bin in Figure 3 (b) and (c)). Note that in real world applications, one could use more resources like web queries for additional data.
Impact of importance weight A. We present the impact of À for different values of k in the validation set of the extended Low-shot ImageNet benchmark in Figure
Top-5 accuracy
75
80
20-shot
10-shot
70
5 S-sbot
65
60
I-shat
55
0.001
0.01
5.0
Figure 4: Impact of on the validation set of the extended Low-shot ImageNet benchmark with YFCC-100M 100M for noisy examples using class prototypes 6


Reviews:
[{"review_id": "rkgIzF3Zcr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a classification method when the data consists of few clean labels and many noisy labels. The authors propose to construct a graph structure within each class and use graph convolutional network to determine the clean/noisy labels of samples in each class. The model is based on a binary cross entropy loss function in each class, which learns the probability of labels to be clean. And such \"clean\" probability is used as the measure of relevance score between the sample different classes.\n\nThe idea of this paper is straightforward and the experimental results seem promising. The authors compare with several related methods and show the proposed method has better performance in few shot learning experiments.\n\nFor the motivation of this methods, why would the graph be constructed within each class? If there is correlation between different classes, how could the model use such class-wise correlation to clean the label?\n\nMaybe I missed it, but how is the relevance score / predicted label determined for testing data given the graphs constructed in each class of training data?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "SyxC53fBir", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper makes a significant attempt at solving one of the practical problems in machine learning -- learning from many noisy and limited number of clean labels. This setting is presumably more practical than the setting of few-shot learning. Noisy labels are often abundantly available and investing in methods that can take the noise into account for building a discriminative model is quite timely. \n\nTo be honest, the theoretical contribution of the paper is limited.  The authors make use of the nearest neighbour graph obtained from a reduced-dimensional set of features to compute the weights of the noisy labels that must guide the predictive model. From this perspective, the paper seems like an application of existing tools (such as CNN, graph convolutional network and binary classification). However, that does not undermine the superior results the authors have received in the novel application they have targeted. I appreciate the effort that went validating these ideas with real-world datasets.\n\nIn future, I would like to see a joint approach to such training, where the function g(), the nearest neighbour graph loss and the classification loss are all tied in the same objective function and are optimized jointly. \n\nThe paper has few really minor grammatical errors and typos. Please fix those before uploading the final draft. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SylESLC6Fr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the problem of learning from multiple tasks and additional noisy data. The proposed representation learning method first assigns each noisy data a relevance score using the topological information. Then the authors propose to minimize a combination of the loss of a class-prototype learning loss and a cosine classifier learning loss to learn a good representation generator g_theta. The empirical study validates the effectiveness of the proposed method.\n\nI have the following comments,\n\n1. The studied problem that learning from few-shot data and large-scale noisy data is interesting. According to the experimental results, the proposed method seems to be promising.\n\n2. The learning procedure is confusing. It is highly recommended to provide the pseudocode of the proposed method.\n\n3. Since there are many tasks and each task has a large-scale data, I'm afraid that the running time will explode. How to deal with this issue?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper combines graph convolutional networks with noisy label learning. The reviewers feel that novelty in the work is limited and there is a need for further experiments and  extensions. 
Decision: Reject