Weak-shot Fine-grained Classification via Similarity Transfer
Authors: Junjie Chen
Li Niu
Liu Liu
Liqing Zhang
Venue: NeurIPS 2021 Poster (2021)
Keywords: transfer learning
similarity transfer
weak-shot learning

Abstract: Recognizing fine-grained categories remains a challenging task, due to the subtle distinctions among different subordinate categories, which results in the need of abundant annotated samples. To alleviate the data-hungry problem, we consider the problem of learning novel categories from web data with the support of a clean set of base categories, which is referred to as weak-shot learning. In this setting, we propose a method called SimTrans to transfer pairwise semantic similarity from base categories to novel categories. Specifically, we firstly train a similarity net on clean data, and then leverage the transferred similarity to denoise web training data using two simple yet effective strategies. In addition, we apply adversarial loss on similarity net to enhance the transferability of similarity. Comprehensive experiments demonstrate the effectiveness of our weak-shot setting and our SimTrans method. 

Weak-shot Fine-grained Classification via Similarity Transfer
Junjie Chen, Li Niu, Liu Liu, Liqing Zhang"
MoE Key Lab of Artificial Intelligence, Department of Computer Science and Engineering,
Shanghai Jiao Tong University
(chen.bys, ustcnewly, shirlley)asji edu. cn, chang-lq@css saaeelllossssstt... edu. cn
Abstract
Recognizing fine-grained categories remains a challenging task, due to the sub- tle distinctions among different subordinate categories, which results in the need of abundant annotated samples. To alleviate the data-hungry problem, we con- sider the problem of learning novel categories from web data with the support of clean set of base categories, which is referred to as weak-shot learning. In this setting, we propose method called SimTrans transfer pairwise seman- tic similarity from base categories to novel categories. Specifically, we firstly train a similarity net on clean data, and then leverage the transferred similarity to denoise web training data using two simple yet effective strategies. In addi- tion, we apply adversarial loss on similarity net to enhance the transferability of similarity. Comprehensive experiments demonstrate the effectiveness of our weak-shot setting and our SimTrans method. Datasets and codes are available https://gihubb.om//cm//// l'rans- Seeak-Sho-CClasificall
Introduction
Deep learning methods have made significant advance extensive computer vision tasks. A large part of this advance has come from the available large-scale labeled datasets. For fine-grained classification, it is necessary but more expensive to collect large-scale datasets. On the one hand, the subtle differences among fine-grained categories dramatically boost the demand for abun- dant samples. On the other hand, professional knowledge is usually required to annotate images for enormous subcategories belonging to one category. As consequence, fine-grained classification is critically limited by the scarcity of well-labeled training images.
In practice, we often have a set of base categories with sufficient well-labeled data, and the problem how to learn novel categories with less expense, in which base categories and novel categories have no overlap. Such problem motivates zero-shot learning [19] few-shot learning [6]] as well setting. To bridge the gap between base (resp., seen) categories and novel (resp., unseen) categories, zero-shot learning requires category-level semantic representation (e.g., word vector [27]] human annotated attributes [[II]) for all categories, while few-shot learning requires few clean examples (e.g., 5, 10) for novel categories. Despite the great success of zero-shot learning and few- shot learning, they have the following drawbacks: 1) Annotating attributes
few clean samples require expert knowledge, which is not always available; 2) Word vector is free, but much weaker than human annotated attributes [II]] Fortunately, large-scale images are freely available from public websites by using category names as queries, which is a promising data source to complement the learning of novel fine- -ennnng categories without any manual annotation.
*Corresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Similarity Transfer
Zero-shot learning Few-shot leurning Weak-shot learning
0.2
0.8
Buse:
0.9
0.1
0.4
Novel:
0.9
Learn pairwise similarity on base sel.
(2) Learn main classifier on novel set.
semantic repecseriation
well-labeled sample
O weak labeled sample
Figure I: Comparison among zero-shot learning, few-shot learning, and weak-shot learning. Dif- ferent colors indicate different categories.
We transfer the similarity learnt from base training set to enhance the main classifier learnt on novel training set. The boxes in differ- ent colors denote different categories. The num- bers above the arrows indicate similarities.
Figure 2:
Considering the drawbacks of zero/few-shot learning and the accessibility of free web data, we in- tend to learn novel categories by virtue of web data with the support of a clean set of base categories, which is referred to as weak-shot learning illustrated in Figure M Formally, given set of novel fine-grained categories which are not associated with any clean training images, we collect web im- ages for novel categories weak-labeled images and meanwhile leverage the clean images from base fine-grain fineggrined categories. We refer to the clean (resp.. web) image set from base (resp., novel) categories as base (resp., novel) training set. Weak-shot learning is a useful and important setting. By taking car model classification as an example, we already have the dataset CompCars [52]] with well-labeled images from base fine-grained categories, but we often need recognize other fine- grained categories beyond this scope, because there are a huge number of car models and new car models are also continuously emerging. In this case, we could apply our setting to recognize novel fine-graing tine-graine categories by collecting the web images for these categories. The closest related work to ours [ET]] but they further assumed the reliability of word vectors [27]] and the availability of unlabeled test images in the training stage.
In weak-shot setting, the key issue of novel training set is label noise, which will significantly de- grade the performance of learnt classifier [GT., [4] We explore using base training set to denoise novel training set, although they have disjoint category sets. As illustrated in Figure E our proposed framework employs the pairwise semantic similarity to bridge the gap between base categories and novel categories. The pairwise similarity which denotes whether two images belong to the same cat- egory is category-agnostic, so it is highly transferable across category sets even if they are disjoint. Meanwhile, the pairwise similarity can be easily learnt from limited data, indicating that small set of already annotated images could help learn extensive novel categories (see Section ) Analo- gously, some methods [B] for few-shot learning transferred similarity from base categories to novel categories, which is directly used for classification. In contrast, we transfer the pairwise similarity to alleviate the label noise issue of web data. For learning from web data, some works [5X, [I] also attempted to denoise by similarity. Nevertheless, their similarities are derived from noisy samples, and likely to be corrupted due to noise overfitting [[7]].
Specifically, our framework consists of two training phases. Firstly, we train a similarity net (Sim- Net) [544] on base training set, which takes in two images and outputs the semantic similarity. Sec- ondly, we apply the trained SimNet to obtain the semantic similarities among web images. In this way, the similarity is transferred from base categories to novel categories. Based on the transferred similarities, we design two simple yet effective methods to assist in learning the main classifier on novel training set. 1) Sample weighting (i.e., assign small weights the images dissimilar to others) reduces the impact of outliers (web images with incorrect labels) and thus alleviates the problem of noise overfitting- 2) Graph regularization (i.e. pull close the features of semantically similar samples (60I) prevents the feature space from being disturbed by noisy labels. In addition, we pro- pose to apply adversarial loss [9] on SimNet to make it indistinguishable for base categories and novel categories, so that the transferability of similarity is enhanced. Since the key of our method is similarity transfer, we name our method SimTrans. We conduct extensive experiments on three fine- grained datasets to demonstrate that the pairwise similarity is highly transferable and dramatically

---

benefits learning from web data, even when the category sets are disjoint. Although the focus of this paper fine- aine-grined classification, we also explore the effectiveness of our setting and method on coarse-grained dataset ImageNet [S]. We summarize our contributions as
We propose to use transferred similarity to denoise web training data in weak-shot classifi- cation task, which has never been explored before.
We propose two simple yet effective methods using transferred similarity to tackle label noise: sample weighting and graph regularization.
One minor contribution is applying adversarial loss to similarity net to enhance the trans- ferability of similarity.
Extensive experiments on four datasets demonstrate the practicality of our weak-shot set- ting and the effectiveness of our SimTrans method.
2 Related Work
2.1 Zero-shot and Few-shot Learning
Zero-shot learning employs category-level semantic representation (e.g., word vector annotated at- tributes) to bridge the gap between seen (resp., base) categories and unseen (resp., novel) categories. A large part of works [ X. 32. ST
[0] learn mapping between visual features and category-level
semantic representations. Our learning scenario is closer to few-shot learning.
Few-shot learning depends on a few clean images (e.g., 5-shot or 10-shot) to learn each novel cate- gory, which could be roughly categorized the following three types. Optimization based methods [M, [6] optimize the classifier on variety of learning tasks (e.g.. learn each category by a few im- ages), such that it can solve new learning tasks using only small number images (e.g. learn novel categories with few images). Memory-based methods [5ir, 351, 28] employ memory architectures to store key training images or directly encode fast adaptation algorithms. Metric-based methods [39, 42) 33, B]] learn deep representation with similarity metric in feature space and classify test images nearest neighbors manner.
Concerning the problem setting, both zero-shot learning and few-shot learning ignore the freely available web images, whereas learn novel categories by collecting web images using category names as queries. Concerning the technical solution, metric-based few-shot learning methods mainly learn categry similarities and directly recognize test images according to the similarity. In contrast, we transfer image-image similarities to reveal the semantic relationships among web train- ing images, which are used denoise the web data for better classifier.
2.2 Webly Supervised Learning
Due to the data-hungry property of deep learning, learning from web data has attracted increasing attention. Many methods have been proposed to deal with noisy images by outlier removal [22] [r]
robust loss function [2y, 45, 577] label correction [64]] EK A6], multiple instance learning [61,, 56
and [55, I. 53 16. ROJ. A prevalent research direction closely related to ours is dealing with
label noise using similarities. To name a few, CurriculumNet [[ل]] computed Huclidean distances between image features, and then designed curriculums according to the distances. SelfLearning [[T] employed the cosine similarity between image features to select category prototypes and correct labels. SOMNet [ва] leveraged self-organizing memory module construct similarity between image and category, which could simultaneously tackle label noise and background noise of web images. However, the similarities used in above methods are derived from the noisy training set, which are likely to be corrupted by noise and lead to sub-optimal results [17 51]. To alleviate this problem, recent works [177, 51] introduced additional annotations to correct label noise. For example, CleanNet [17]] directly learned the image-category similarity based on verification labels, which involves human verification of noisy images. Distinctive from the above methods, we do not require any manual annotation on crawled web images.
3

---

2.3 Weak-shot Learning
In broad sense, weak-shot learning means learning novel categories from cheap weak labels with the support of set of base categories already having strong labels. Similar weak-shot setting has been explored in other computer vision tasks, including object detection [T3 25, 58, B. 22].. semantic segmentation [[9]] and instance segmentation [044, [8]. For weak-shot object detection (also named muxed-superrvsed or cross-supervised), base categories have bounding box annotations while novel categories only have image class labels. For weak-shot semantic segmentation, base categories have semantic mask annotations while novel categories only have image class labels. For weak-shot instance segmentation (also named
base categories have instance mask anno- tations while novel categories only have bounding box annotations. To the best of our knowledge, we are the first work on weak-shot classification, which employs transferred similarity to denoise web training data.
3 Problem Definition
In the training stage, we have C2 base categories with No clean images for the c-th base category. Our main goal is learning to classify C, novel categories given No web training images for the c-th novel category. Base categories and novel categories have no overlap. No extra information (e.g., word vector) is required the training stage. In the testing stage, test images come from novel categories. We refer to the above setting as weak-shot learning.
For some weal-world applications, test images may come from both base categories and novel cate- gories, which is referred to as generalized weak-shot learning. In this paper, we focus on weak-shot learning, while leaving generalized weak-shot learning to Appendix.
4 Approach
Our training stage consists of two training phases: learning similarity net (SimNet) base training set and learning the main classifier on novel training set.
4.1 Learning SimNet on Base Training Set
Although there many forms of similarity, we choose an end-to-end deep network SimNet model the similarity function as in [54]]. The ar- chitecture SimNet shown in the top pipeline of Figure HL which takes in mini-batch of images and outputs the pairwise similarit/dissimilrriyy score for each pair of input images (e.g., M pairs for mati-batch size M). The enumeration (Enum) layer simply concatenates each pair of im- age features which are extracted by the backbone. For instance, suppose the image features have size M x D with the feature dimension D, then the output of the enumeration layer will have size M² 2D. Af- ter that, one fully-connected (FC) layer extracts feature for each concatenated feature pair, which is dubbed as relation feature of each pair. Finally, the relation features are supervised by classification loss with binary labels being "similar" (a pair of images from the same category) and "dissimilar" (a pair of images from different categories).
Backbonc
Enom &C
PC
Cls Loss
copy
Backbome
Enum &
D
Ad
Figure 3: Illustration of similarity net with adversarial loss. The top (resp., bottom) pipeline processes base (resp.. novel) training set. r (resp., r") represents base (resp., novel) re- lation features.
Considering the training and testing for SimNet, if we construct the mini-batch randomly, similar and dissimilar pairs will be dramatically imbalanced. Specifically, if there are C' categories in mini- batch, the probability for two images to from the same category &. To reduce the imbalance between similar pairs and dissimilar pairs, when constructing mini-batch, first randomly select

---

Cm (< C) categories and then randomly select M images from each selected category, as in [ C... We use Cm = 10 and M 100 for both training and testing of SimNet.
4.1.1 Adversarial Loss
Ideally, the learnt similarity is category-agnostin but there may exist domain gap between the rela- tion features of different category sets. Therefore, we use novel training set in the training stage to further reduce the domain gap between base categories and novel categories, as shown in the bottom pipeline of Figure B.
Specifically, a discriminator D-)) takes in relation features r in SimNet and recognizes whether they come from base categories (r") or novel categories (r"). The SimNet acts as generator, aiming to produce relation features which could not only confuse the discriminator but also benefit the relation classification. Note that the labels of novel training images are noisy, so we exclude the image pairs from novel categories from the relation classification loss. Analogous to [M], optimize SimNet and the discriminator alternatingly in an adversarial manner. Firstly, we freeze the generator and minimize the adversarial loss of discriminator. Secondly, with frozen discriminator, we minimize the relation classification loss of SimNet and maximize the adversarial loss of discriminator. The classification loss and the adversarial loss are balanced with parameee B, set as 0.1 via cross-validation. The optimizing is trivial and we leave the details to Appendix.
4.2 Learning Classifier Novel Training Set
Because of the label noise of web images, the performance will be significantly degraded when directly training the classifier using web data. To address the label noise issue, we employ two simple yet effective methods based on transferred similarities illustrated in Figure H. Transferred similarities mean the similarities among novel training samples, which are calculated by SimNet trained on base training set. Next, we will introduce these two methods, i.e., sample weighting and graph regularization, separately.
4.2.1 Sample Weighting
For the web images within a novel category, we observe that non- outliers (images with the correct label) are usually dominant, while outliers (images with incorrect labels) are from non-dominant inaccurate categories. When calculating the semantic similarities among web im- ages within novel category, outliers are more likely to dissimilar to most other images. Therefore, could determine whether an image is an outlier according to its similarities to other images.
Simi limilarit Matrix
W Weight
1.0
0.1
C.7
0.0
0.4
1.0
0.0
0.9
0.7
0.2
0,6
We
0.7
077
0.0
1.0
0.0
0.1
0.0
SimNet
0.9
C.0
0,
0.2
Avg
W4
0.6
0.7
6.1
0.8
10
0.0
0.4 W3
0.2
0.0
0.6
W2
Xss
W1
Reg
Cls
Loss
Loss
A
X3
Backbone
FC
Formally, for the e-th novel cate- gory with No web images, we first compute the similarity matrix S. E N XN=, with each entry Sc,.j be- ing pairwise similarity calculated by SimNet pre-trained on base training set. Although the size of S. may be large, there are only No times of backbone inference and N x N times of two FCs inference, which are computationally efficient. Then, we employ the average of similarities be-
Feature
Category
Figure 4: The overview of learning novel categories from web data. The similarity net (SimNet) outputs similarity ma- trix with pairwise similarities and generates sample weights. The sample weights are employed to weigh the main classi- fication loss (Cls Loss) of each image, while the similarity matrix is used in graph regularization loss (Reg Loss) based on image features.
5

---

tween image and all the other images as its weight:
No
1
8,,ij 8a...
We,s
No
Then, the sample weights are normalized to have unit mean, i.e., Be.i
2 /o.s/Ng As analysed
Wic.t
before, samples with lower weights We are more likely to outliers. Finally, we employ weighted classification loss based on sample weights Ue,s. In this way, assign lower weights to the training losses of outliers, which enables non-outliers to contribute more to learning a robust classifier:
M
Lete_
Σ m=1
ün log (Xmm) (Xm)
(2)
M
where M is the mini-batch size, Xm the mn-th image in the mini-batch, um its assigned weight, and f ((m)yy is its classification score corresponding to its category ym-
4.2.2 Graph Regularization
When directly learning novel training set, the feature graph structure, that is, the similarities among image features, are determined by noisy labels. In particular, the classification loss implicitly pulls features close for images with the same labels. However, the feature graph structure may be misled by noisy labels, so we attempt to use transferred similarities to rectify the misled feature graph structure. Specifically, we employ typical graph regularization [60]] based on transferred similarities to regulate features, which enforces the features of semantically similar images to be close.
Formally, for each mini-batch of M images, we first compute the similarity matrix E R MMM using the SimNet pre-trained base training set. Regarding the similarity matrix as adjacency matrix, the graph regularization loss is formulated as:
Leeg Σ Bi.]1/(x))) 1(xx
where 52, is each entry in S, and h(x:) the image feature of x; extracted by the backbone.
According to [АN], web images mainly have two types of noise: outlier noise and label-flip noise. Outlier noise means that an image does not belong to any category within the given category set, and label-flip noise means that an image is crroneously labeled as another category within the given category For the samples with label-flip noise, sample weighting directly discards these samples by assigning lower weights. However, graph regularization can utilize them to maintain reasonable feature graph structure and facilitate feature learning, which could complement sample weighting.
4.3
The Full Objective
On the whole, we train the classifier on novel training set by minimizing the weighted classification loss and graph regularization loss:
L = Lets, + LeLorgs
(4)
where a is Syper-parameter set as 0.1 by gross-validation
5 Experiments
5.1 Datasets and Implementation
We conduct experiments based on three fine-gra fne-grained datasets: CompCars [52]] (Car for short), CUB [48], and FGVC [[6]]. We split all categories into base categories and novel categories. The base training/test set and the novel test set are from the original dataset while the novel training set con- structed using web images. For instance, the novel training set of Car dataset is constructed using the released web images in WebCars [6]]], while the novel training sets of CUB and FGVC are con- structed by collecting the web images by ourselves. The statistics of three datasets are summarized

---

Table 1: The statistics of splits on three datasets. Category shows the number of split categories. Train (resp., Test) indicates the average number of training (resp., test) images for each category.
Table 2:
Module contributions on CUB dataset. Cls indicates the cross-entropy classification loss of main classifier. Ad means training Sim- Net with adversarial loss. Weight means sample
Dataset
Split
Category
Train
Test
weighting and Reg means graph regularization.
Base 323
37
34
Cls
Ad
Weight
Reg
Acc (%)
Car
Novel
108
510
36
85.4
Base
150
30
30
90.3
CUB
Novel
50
1000
30
89.5
Base
75
67
33
91.2
FGVC
Novel
25
1000
33
90.7
90.1
91.7
Table 3:
The first three rows show the performances (%) of SimNet evaluated by 150 base categories, 50 base categories, and 50 novel categories. The last shows the performance (%) of uniform random guess on 50 novel categories. PR, RR, and F1 represent pre-
Table 4: The performances (%) of main classifier supported by various similari- ties learnt from various training sets.
Source
Type
Classifier
Euclidean
86.2
cision rate, recall rate, and F1 Fc-re,,, respectively.
Novel
Cosine
86.4
Similar class
Dissimilar class
SimNet
87.6
PR
RR
F1
PR
RR
F1
Euclidean
87.5
Novel+
B-150
84.4
87.7
86.0
98.6
98.2
98.4
Cosine
87.6
Base
B-50
89.0
88.2
88.6
98.7
99.0
98.8
SimNet
89.5
N-50
88.4
87.6
88.0
98.6
98.7
98.7
Euclidean
88.8
Rand
10.0
50.0
16.7
90.0
50.0
64.3
Base
Cosine
89.1
SimNet
91.7
backbone. There is no overlap between novel categories and ImageNet 1/ categories. Besides, all baselines use the same backbone for fair comparison. More details of datasets and implementation are left Appendix.
in Table MI We employ ResNet50 [12]] pretrained on ImageNet [[]
Note that the focus of this work fine-grate fine-grained classification, because 1) it especially requires expert knowledge annotate labels for fine-grained data (a drawback of few-shot learning); 2) conjec- ture that semantic similarity should be more transferable across fine-grained categories. However, we also explore the effectiveness of our setting and method on a coarse-grained dataset ImageNet [S]] We leave the experimental results on ImageNet to Appendix.
5.2 Ablation Study
We conduct our ablation study on CUB dataset, considering its prevalence in extensive vision tasks. We evaluate the performances different combinations of our modules, and summarize the results in Table 2 Solely using sample weighting leads to a dramatic improvement (90.3% v.s. 85.4%). Solely enabling the graph regularization also results in a considerable advance (89.5% v.s. 85.4%). Jointly applying sample weights and graph regularization further improves the performance (91.2% v.s. 85.4%). The adversarial loss on SimNet boosts the performance of classifier for both sample weighting (90.7% 99..7% v.s. 90.3%) and graph regularization (90.1% 1% v.s. 89.5%). Finally, our full method outperforms the baseline by a large margin (91.7% v.s. 85.4%).
5.3 Investigating Similarity Transfer
We evaluate SimNet in various situations to explore the effectiveness of similarity transfer. Although SimNet employed to denoise web images, we cannot evaluate its performance on noisy images without ground-truth labels, so we perform evaluation on clean test images. The performance mea- surements (e.g., precision rate (PR), recall rate (RR), and FI-score (F1)) are computed at the pair level, instead of image level. The experiments in this subsection conducted on CUB dataset.
7

---

92-
90
90
80
90
90
89
150
125
75
5040
25
25
20
150
125
100
75
50400 25
25
10
(a) Classifier wir Cb-
Classifier wax NV
(c) Simnet wond Ca-
(d) SimNet No.
Figure 5: Performance variations of main classifier (accuracy %) or similarity net (Fl-score %) w.r.t
different numbers of used base categories (C₂) or used clean images per base category (No).
5.3.1 The Transferability of Similarity
The performance gap of SimNet between base categories and novel categories indicates the trans- ferability of learnt similarity. We first train SimNet on base training set with 150 categories, and evaluate its performances on base test set with 150 categories and novel test set with 50 categories. We also evaluate on the base test set formed by 50 random base categories to exclude the influence of different category numbers. The results are summarized in Table B Surprisingly, SimNet trained on base categories achieves a comparable performance for base categories and novel categories (Base- 50 v.s. Novel-50). On the whole, the above results suggest that the pairwise similarities are highly transferable across fine-gres one-grained categories.
5.3.2 The Comparison of Similarity Sources and Types
To demonstrate the superiority of the our transferred similarity, we compare different types of sim- ilarities learnt from various training sets (sources) in Table A For Euclidean and Cosine, pairwise similarity is computed based the features extracted by network pretrained on specific train- ing set (source). Cosine similarity is well-known and we omit the details here. For Euclidean, we adopt the reciprocal of Euclidean distance. For SimNet, (sources). By comparing different similarity sources, we observe that all types of similarities learned from base training set perform better, while the performances decrease when novel training set (con- taining noisy images) is involved. This verifies that the similarity could be severely corrupted by noisy images, indicating the necessity of learning similarity using clean images even if the category sets are disjoint. By comparing different similarity types, SimNet performs optimally, showing the superiority of learning similarity in an end-to-end fashion.
train it using different training sets
5.3.3 The Impact of the Scale of Base Training Set
We naturally wonder what is the minimum requirement of the scale of base training set for our similarity net and main classifier to perform well. We explore the scale of base training set from two aspects: category number and image number in each category. We mainly report the Fl-score of similar category evaluated on novel test set as the performance of SimNet, and the results are illustrated in Figure E. The subfigure (a) and (c) shows the robustness when using more than 50 categories, while the subfigure (b) and (d) shows robustness when using more than 5 images per category. Furthermore, we explore the joint impact of two aspects in Table E Surprisingly, supported by only 50 base categories with images in each category, the main classifier could achieve a satisfactory performance, i.e., 89.3% against 85.4%, where 85.4% is the result without the support of base training set. This study indicates the potential of our setting and method, that is, small-scale off-the-shelf base training set could facilitate learning extensive novel fine-grained categories.
5.4 Comparison with Prior Works
We compare with two types of methods: webly supervised learning and transfer learning.
For webly supervised learning, we compare with recent representative methods: SelfLearning [[]]] MetaCleaner [56]]. CurriculumNet [[8]] SOMNet [Аæ]], DivideMix [21]] LearnToLearn [20], NLNL [155], and JoCoR [27]]. Webly supervised learning methods only use novel training set.
For transfer learning across categories, there are mainly two groups of methods: few-shot learning and knowledge transfer. For few-shot learning, we compare with MetaBaseline [3]] MAML [А], and SNAIL [ MetaBaseline is a metric-based method, and we use the averaged feature of each novel category as class centroid following [3]. SNAIL memory-based method, and we use 16
prototypes of each novel category as 16 shots due to hardware limitation. For knowledge transfer,
8

---

Table 5: The performances (%) of SimNet and main classifier supported by various scales C' x N of base training set, in which C' denotes the used category number and N denotes the used image number each category.
Table 7: Accuracies (%) of various methods
on three datasets in the weak-shot learning. The best results are highlighted in boldface.
Method
Car
CUB
FGVC
Cls Loss
83.1
85.4
86.6
SelfLearning
85.1
87.3
88.1
C x
50x5
75x5
50x
150x30
MetaCleaner
84.9
87.1
88.3
SimNet
73.2
79.1
80.6
88.0
CurriculumNet
85.2
86.8
87.9
Classifier
89.3
89.9
90.4
91.7
SOMNet
86.0
87.9
88.6
DivideMix
86.2
88.0
89.1
Learn ToLen
85.3
87.6
88.4
Table 6: Accuracies (%) of representative meth- ods using various numbers of web images on CUB dataset.
NLNL
85.4
87.9
88.2
JoCoR
85.9
87.9
88.3
MetaBaseline
85.8
87.1
88.7
MAML
84.6
86.8
87.9
Number
200
400
600
800
1000
SNAIL
84.1
86.3
87.2
Cls Loss
78.5
82.7
84.3
85.2
85.4
Distillation
83.7
85.9
87.3
Fusionl
80.7
84.9
87.1
88.2
88.3
Finetune
84.2
86.5
87.6
Fusion2
81.0
85.1
87.2
88.3
88.5
Fusionl
86.8
88.3
89.7
Ours
84.3
89.5
91.0
91.6
91.7
Fusion2
86.9
88.5
90.0
SimTrans
89.8
91.7
92.8
the Distillation baseline [23] enhances the classifier for novel categories by additionally predicting category distribution over base categories, which is supervised by a classifier pre-trained on base training set. Another natural baseline, named as Finetune, trains the model on the mixture of base training set and novel training set, and then fine-tunes on novel training set.
In addition, as far as we are concerned, there no method which could jointly denoise web train- ing data and transfer across categories without additional information (e.g., word vector). So we combine competitive methods in their own tracks using late fusion (i.e., average prediction scores). The fusion "SOMNet+Metaaasslin videMix++Metaaaselnne is referred to as Fusion2.
is referred
Fusionl, while the fusion "Di-
All the results summarized in Table M We also include the basic baseline Cls Loss, which is trained on novel training set with standard classification loss (row Table 2). Based on Table we have the following observations:
1) Webly supervised learning methods outperform the basic baseline Cls Loss, showing the general improvement brought by denoising web data without any support.
2) Few-shot learning methods and knowledge transfer methods outperform the basic baseline Cls Loss, indicating the advantage of transfer learning across categories. Note that MetaBaseline achieves a commendable performance, probably because averaging features could denoise web im- ages to some extent. Nevertheless, the overall performances of transfer learning methods are limited by severe label noise.
3) The simple combination of webly supervised learning and transfer learning, i.e., Fusion] or Fu- sion2, outperforms the above baselines, which directly demonstrates the effectiveness of our weak- shot setting. Furthermore, our method achieves the optimal performance against all baselines, which indicates the superiority of sample weighting and graph regularization based on similarity transfer.
5.5 Qualitative Analysis for Weight and Graph
We visualize to verify whether the higher weights are assigned to clean images and the transferred similarities could reveal semantic relations. The visualizations and analyses left to Appendix.
5.6
Robustness Analysis of The Method
We analyse the robustness of our method several factors, including the number of web images, the selection of hhper-ppane parameters, the noise ratio in web images, and the impact of different backbones.

---

Table Accuracies (%) of different methods on three datasets in the generalized weak-sho setting. The best results are highlighted in boldface.
Method
Car
CUB
FGVC
Cls Loss
76.3
73.3
76.9
Fusionl
77.5
74.8
78.8
Fusion2
78.1
75.0
79.3
SimTraas
80.4
76.9
81.7
For the impact of the number of web images, in our setting, the training data consists of clean images for base categories and web images for novel categories. The impact of the former is investi- gated in Section 5.3,3, and here we conduct experiments using different numbers of web images for each novel category on CUB dataset. As shown in Table & the general performances are gradually saturated given more training images (e.g., more than 800 web images), and the observations of performance promotions against baselines are consistent across different numbers of web images.
For other factors (i.e., hyper-parameter, noise ratio, and backbones), left the analysis to Appendix considering the space limitation.
5.7 Extension to Generalized Setting
In the generalized weak-shot learning, we additionally include base test set from base categories in the test set, so that test images may come from a mixture of base categories and novel categories, which is more practical in real-world applications. We extend our method and baselines to general- ized weak-shot learning scenario for comparison.
For simplicity, we directly conduct the experiments by treating the base training set as web images with low noise rate. For our method, the first training stage of training SimNet remains the same, while the second training stage additionally includes base training set when training the main clas- sifier. Thus, the sample weights assigned base training images would be near one, and the graph regularization involving base training images will also do no harm to feature learning at least.
We compare with the basic baseline Cly Loss well learning method and transfer learning method across categories. SOMNet and DivideMix are also conducted by treating base training images as web images (the clean probability thresholds are set as 0 for base training images in DivideMix). MetaBaseline could be directly applied in the generalized setting. One practical problem is that the image number of base categories and the image number of novel categories are highly imbalanced (e.g., 30 and 1000 for CUB), so we weigh the classification loss of each category using their image numbers (higher weight for the category with fewer images) for all methods in this setting. The results are reported in Table & from which we can find that the fusion of transfer learning and webly supervised learning outperforms the basic baseline, while our method further improves the results by a large margin.
combinations of webly supervised
Conclusion
In this paper, we consider the problem of learning novel categories from easily available web images with the support of set of base categories with off-the-shelf well-labeled images. Specifically, have proposed to employ the pairwise semantic similarity to bridge the gap between base categories and novel categories. Based on the transferred similarities, two simple yet effective methods have been proposed to deal with label noise when learning the classifier on web data. Adversarial loss is also applied to enhance the transferability of similarity. Extensive experiments on four datasets have demonstrated the potential of our learning paradigm and the effectiveness of our method.
Acknowledgements
The work was supported by the National Key R&D Program of China (2018AAA0100704), National Natural Science Foundation of China (Grant No. 61902247), the Shanghai Municipal Science and Technology Major Project (Grant No. 2021 SHZDZX0102) and the Shanghai Municipal Science and Technology Key Project (Grant No. 2051 15 100300).
10

---

References
Zeynep Akata, Scott Reed, Daniel Walter, Honglak Lee, and Bernt Schiele. Evaluation of output embed- dings line-grained image classification, CVPR, 2015.
Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, and Fei Sha. Synthesized classifiers for zero-shot learning. CVPR, 2016.
Yinbo Chen, Xiaolong Wang. Zhuang Liu, Huijuan Xu, Trevor Darrell. A new meta-baseline for few-shot learning. In ICLR, 2020.
[4] Zitian Chen, Zhiqiang Shen, Jiahui Yu, and Erik Learned-Miller Cross-supervised object detection, arXiv preprint 2006 15056, 2020.
[5] Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchi- cal image databasc. In CVPR, 2009.
[6] Li Fei-Fei, Rob Fergus, and Pietro Perona. One-shot learning of object categories. JEEE transactions on pattern analysis and machine intelligence, 28(4):594-66 611, 2006.
[7] Chelsea Finn, Pieter Abbeel, and Sergey Levine. deep networks. In ICML, 2017.
Model-agnostic meta-lcarning for fast adaptation of
[8] Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean. Marc' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. In NeurlPS, 2013.
[9] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014.
[10] Zhangxuan Gu, Siyuan Zhou, Li Niu, Zihan Zhao, and Liqing Zhang. Context-aware feature generation for zero-shot semantic segmentation. In ACM MM, 2020.
[11] Jiangfan Han, Ping Luo, and Xiaogang Wang. Deep self-learning from noisy labels. In ICCV, 2019.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[13] Judy Hoffman, Sergio Guadarrama, Eric Tzeng, Ronghang Hu, Jeff Donahue, Ross Girshick, Trevor Darrell, and Kate Saenko. Lsda: Large scale detection through adaptation. NIPS, 2014.
[14] Ronghang Hu, Piotr Dollár, Kaiming He, Trevor Darrell, and Ross Girshick. Learning to segment every thing. In CVPR, 2018.
[15] Youngdong Kim, Junho Yim, Juseung Yun, and Junmo Kim. Ninl: Negative learning for noisy labels. In ICCV, 2019.
[16] Jonathan Krause, Benjamin Sapp, Andrew Howard, Howard Zhou, Alexander Toshev, Tom Duerig, James Philbin, and Li Fei-Fei. The unrcasonable effectiveness of noisy data for fine-grained recognition. In ECCV, 2016.
[17] Lei Zhang Kuang-Huce Lee, Xiaodong He and Linjun Yang. image classifier training with label noise. In CVPR, 2018.
Cleannet: Transfer learning for scalable
[18] Weicheng Kuo, Anelia Angelova, Jitendra Malik, and Tsung- Yi Lin. Shapemask: Learning to segment novel objects by refining shape priors. In ICCV, 2019.
[19] Christoph H Lampert. Hannes Nickisch, and Stefan Harmeling. Attribute-base classification for zero- shot visual object categorization. IEEE transactions pattern analysis and machine intelligence, 36(3): 453-465, 2013.
[20] Junnan Li, Yongkang Wong, Qi Zhao, and Mohan Kankanhalli, Learning to learn from noisy labeled data. In CVPR, 2019.
[21] Junnan Li, Richard Socher, and Steven C.H. Hoi. supervised learning. In ICLR, 2020.
Dividemix: Learning with noisy labels as semi-
[22] Yan Li, Junge Zhang, Kaiqi Huang, and Jianguo Zhang. Mixed supervised object detection with robust objectness transfer. IEEE transactions on pattern analysis and machine intelligence, 41(3):639 653, 2018.

---

[23] Qing Liu, Lingxi Xie, Huiyu Wang, and Alan L Yuille. Semantic-aware knowledge preservation for zero-shot sketch-based image retrieval. In ICCV, 2019.
[24] Wei Liu, Gang Hua, and John R Smith. Unsupervised cle-slass learning for automatic outlier removal. In CVPR, 2014.
[25] Yan Liu, Zhijie Zhang, Li Niu, Junjie Chen, and Liqing Zhang. Mixed supervised object detection by transferring mask prior and semantic similarity. In Neur/PS, 2021.
[26] S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013.
[27] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In NeurlPS, 2013.
[28] Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In ICLR, 2018.
[29] Volodymyr Mnih and Geoffrey E. Hinton. Learning to label aerial images from noisy data. In /CMI, 2012.
[30] Li Niu, Qingtao Tang, Ashok Veeraraghavan, and Ashutosh Sabharwal. Learning from noisy web data with category-level supervision. In CVPR, 2018.
[31] Li Niu, Ashok Veeraraghavan, and Ashutosh Sabharwal. Webly supervised learning meets zero-shot learning: A hybrid approach for fine-grained classification. In CVPR, 2018.
[32] Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer. Jonathon Shlens, Andrea Frome. Greg S Corrado, and Jeffrey Dean. Zero-shot learning by convex combination of semantic embeddings. In ICLR, 2014.
[33] Boris N Oreshkin, Pau Rodriguez Lopez, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In NeurlPS, 2018.
[34] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In CVPR, 2017.
[35] Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. In ICMI., 2016.
[36] Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell, Meta-learning with latent embedding optimization. In ICLR, 2019.
[37] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory-augmented neural networks. ICML, 2016.
[38] Haozhi Zhang Chenfan Zhuang Dengke Dong Matthew R. Scott Sheng Guo, Weilin Huang and Dinglong Huang. Curriculumnet: Weakly supervised learning from large-scale web images. In ECCV, 2018
[39] Jake Snell, Kevin Swersky, and Richard S Zemel. Prototypical networks for sew-shot learning. In NeurlPS, 2017.
[40] Sainhayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, and Rob Fergus. Training convo- lutional networks with noisy labels. In ICLR workshop, 2015.
[41] Xiaoxiao Sun, Liyi Chen, and Jufeng Yang. Learning from web data using adversarial discriminative neural networks for fine-grained classification. In AAAI, 2019.
[42] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip H Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In CVPR, 2018.
[43] Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In NewrIPS, 2018.
[44] Yi Tu, Li Niu, Junjie Chen, Dawei Cheng, and Liqing Zhang, Learning from web data with self-organizing memory module. In CVPR, 2020.
[45] Brendan Van Rooyen, Aditya Menon, and Robert C Williamson. Learning with symmetric label noise: The importance of being unhinged. In NewrIPS, 2015.
[46] Andreas Veit, Neil Alldrin, Gal Chechik, Ivan Krasin, Abhinav Gupta, and Serge Belongie. Learning from noisy large-scale datasets with minimal supervision. In CVPR, 2017.
12

---

[47] Hongxin Wei, Lei Feng. Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In CVPR, 2020.
[48] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSSD Birds California Institute of Technology. 2010.
200. Technical Report
[49] Yan Xia, Xudong Cao, Fang Wen, Gang Hua, and Jian Sun. Learning discriminative reconstructions for unsupervised outlier removal. In CVPR, 2015.
[50] Yongqin Xian, Zeynep Akata, Gaurav Sharma, Quynh Nguyen, Matthias Hein, and Bernt Schiele. Latent embeddings for shot classification. In CVPR. 2016.
[51] Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In CVPR, 2015.
[52] Linjie Yang, Ping Luo, Chen Change Loy, and Xiaoou Tang. A large-scale datasct for fine-grained categorization and verification. In CVPR, 2015.
[53] Jiangehao Yao, Hao Wu, Ya Zhang, Ivor W Tsang, and Jun Sun. Safeguarded dynamic label regression for noisy supervision. In AAAI, 2019.
[54] Zhaoyang Lv Yen-Chang Hsu and Zsolt Kira. Learning to cluster in order transfer across domains and tasks. In ICLR, 2018.
[55] Chuanyi Zhang, Yazhou Yao, Huafeng Liu, Guo-Sen Xie, Xiangho Shu, Tianfei Zhou, Zheng Zhang, Fumin Shen, and Zhenmin Tang. Web-supervised network with softly update-drop training for finc- grained visual classification. In AAAI, 2020.
[56] Weihe Zhang, Yali Wang, and Yu Qiao. Metacleaner: Learning to hallucinate clean representations for noisy-laheled visual recognition. In CVPR, 2019.
[57] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. NeurlPS, 2018
[58] Yuanyi Zhong, Jianfeng Wang, Jian Peng, and Lei Zhang. Boosting weakly supervised ohject detection with progressive knowledge transfer. ECCV, 2020.
[59] Siyuan Zhou, Li Niu, Jianlou Si, Chen Qian, and Liqing Zhang. Weak semantic segmentation by transferring semantic affinity and boundary. arXiv preprint 1rX.592110001519, 2021.
[60] Xiaojin Zhu, Zoubin Ghahramani, and John D Lafferty. Semi-supervised lcarning using gaussian fields and harmonic functions. In ICML, 2003.
[61] Bohan Zhuang, Linggiao Liu, Yao Li, Chunhua Shen, Ian Reid. Attend in groups: weakly- supervised deep learning framework leamning from web data, In CVPR, 2017.
13


Reviews:
[{"review_id": "VDoIQcuhBQ9", "reviewer": "Reviewer_K9m3", "summary": "This paper proposes to transfer pairwise semantic similarity from clean image data to web image data for classifying novel fine-grained categories. The proposed method provides a way to alleviate the data-hungry problem in the fine-grained classification task.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes to utilize the easily available web data to learn novel fine-grained categories by transferring the pairwise semantic similarity from clean data to web data. Experiments are conducted for effectiveness validation on three public datasets. My concerns are as follows:\n\n1.\tThis paper holds the premise that most web images are with correct labels and utilize the proposed SimNet trained on clean data to filter out the noisy web data. Dose the proposed method still work when the crawled web data are very dirty? Corresponding experiments or theoretical analyses should be added.\n2.\tWhen the training set is composed of the base training set and the novel training set, the classification accuracy drops a lot in section 5.10. Respective classification results in the base test set and the novel test set should be given. Corresponding experiments and analyses should be added, which is essential for verifying the effectiveness of the proposed method in the real-world applications.\n3.\tThe authors should describe the difference between weak-shot learning and zero/few-shot learning more clearly. Respective training and test stages should be presented for better understanding of readers.\n4.\tMore qualitative results about finding the outliers among the web data should be added to prove the effectiveness of the proposed method visually.\n5.\tThe layout of this paper should be carefully designed. For example, Figure 3, Figure 4 and their surrounding text should be reset for better readability.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "LWYrhlBGzEz", "reviewer": "Reviewer_hoaz", "summary": "This paper addresses a new task called weak-shot learning. It is to learn novel categories by using the knowledge learned in the clean set of base categories. The clean set of base categories refers to the clean, well-labeled image classification dataset. \n\nThe proposed method can be summarized into two components. First, the authors train a SimNet on the clean base dataset. It is to learn to capture the semantic concepts from the images. Secondly, the authors conduct supervised classification training on the novel images using similarity labels, where the similarity labels are obtained from the similarity matrix computed by the pre-trained SimNet. To address the label noise in the unseen dataset, sample weighting and graph regularization are employed to better leverage the similarity matrix in training.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n- The authors propose an interesting and practical task, and the authors present good baseline to address the problem.\n- The idea of similarity transfer is good and reasonable. \n- The use of SimNet, and the construction of the relation features is interesting.\n- Large number of experiments are conducted to verify different factors in the proposed method.\n\nWeakness\n- It is a complicated paper. The paper seems revised many times, but the presentation is not very effective in my point of view. From a high-level view, I think the main goal is to train a novel-category classifier using the regularizations obtained from clean base categories. It might be easier to understand the paper (at least for me, or general audience) by first presenting the overall pipeline and then talk about Sec 4.2. After that, discuss the regularizations terms such as how to obtain the similarity matrix. In addition, in my opinion, the paper is more relevant to web supervision but not zero/few-shot learning. Probably the term \"weak-shot learning\" is somehow confusing. Moreover, I see the proposed idea fundamentally work for general image classification problem, but the authors tend to restrict the presentation to only fine-grained image classification. To make the paper easier to follow, I would suggest to first address a simple, general problem (general image classification), and then extend it to a more complicated task (fine-grained image classification).\n- In current presentation, the authors present a similarity transfer method. One intuitive question is how to apply such method to real-world problems? Is there any restriction? Although the base and novel sets are disjoint in the experiments, one can see the base and novel sets are from the same dataset such as CUB. This implies there are requirements for the base and novel sets. They must belong to the same meta-category, such as bird. If we would like to train a novel bird classifier, but only CompCars dataset is available as the clean base set, I am not sure if the transferability still hold. This is a natural question when we deal with similarity transfer between fine-grained datasets. Instead of dealing with fine-grained dataset, I feel it is more appealing to do general image classification. By doing so, the whole story becomes more elegant, flexible, and easy to sell. \n- The proposed task is too complicated. As the results, the authors have to conduct very large amount of experiments to verify many components, and it is very difficult to present all the experiments in the main paper. Instead of simply leaving all the analyses to Appendix, the authors should also summarize the key findings of the experiments in the main paper. \n\nOverall\n- The paper addresses an interesting problem, and the proposed method should be technically correct. However, the paper is presenting in a way that is not easy to follow. I think it is a borderline paper. Both the proposed task and method are OK, but it is fine to reject it as well.\n\n****post rebuttal****\nThank you for the response. I would like to keep my rating. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "n/a", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "9k8m1E3RAGs", "reviewer": "Reviewer_wyya", "summary": "This paper proposes to use transferred similarity to denoise web training data in weak-shot fine-grained classification task, with two simple yet effective strategies methods, sample weighting and graph regularization.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The novelty and contribution of this work are marginal. Although the employment of transferred similarity to denoise web training data in weak-shot fine-grained classification task is new, as claimed in the paper, the combination of SimNet and DANN (Domain Adversarial Neural Networks) fails to surprise me. It seems that the proposed sample weighting and graph regularization share similar idea and intuition with MADA (Multi-adversarial Domain Adaptation), making the technical novelty not quite significant.\n2. The organization of the manuscript could be improved. I think it is not a reader-friendly choice to leave too many detailed results and analyses to Appendix (Section 5.8, Section 5.9, and Section 5.10), which may make the flow of the paper hard to follow.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please refer to the section \"Main Review\".\n\n", "review_text": "", "extra_scores": {"ethical_concerns": "I do not think there are ethical issues with this paper.\n\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "-EmSOmKBYwW", "reviewer": "Reviewer_bmx4", "summary": "The paper is about transferring the semantic information between datasets to learn from the web data. A combination of known techniques such as SimNet [47], the graph regularization [51] and adversarial loss [8] are applied to obtain results.", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of using web supervision is interesting and there are large-scale web-based datasets available, these are only a few relevant examples: https://paperswithcode.com/dataset/pinterest, https://arxiv.org/pdf/1611.08321.pdf, https://aclanthology.org/2021.naacl-main.473.pdf. The main motivation of the work seems to be able to learn from web data and transfer the leanings to other datasets. I have the following concerns about this work.\n\n- I do not believe the authors have adequately addressed the previous reviews\n- From the architectural standpoint, the novelty is very limited. A major contribution could have been a definition of new transfer task from the web data.\n- However, in my opinion the authors fail to design the experiments to prove the concept of transferrability from the web data. In particular, I do not believe they use a large-scale web-based dataset as a base and then demonstrate successful transfer to another dataset. \n- The paper is not clearly written. The introduction is jumping around zero-/few- shot and web data concepts without ever clearly defining the use case that the authors want to address. In fact, I cannot figure out the use case even after reading the entire article, because the authors claim that they can learn and transfer from the web data, but experiments do not consider a real-life web dataset at all. The Problem Definition section provides a really short and abstract definition of the task. In my opinion a detailed description of the real use case transitioning to its mathematical description would have done a much better job here. The Experiments section has way too many subsections and it is very hard to navigate. There is no clear message in the section and none of the sections really addresses the main question (according to my guess) whether web driven learning is viable in the context of the task that the authors consider. The research questions that the authors want to answer in the section are not defined either, so it is really hard to understand the motivation behind ten different subsections of Section 5. I believe the authors themselves do not see the forest behind the trees and it is even worse for the reader. Please define your research questions and the message you would like to send with your research in a clear and concise way.\n\nDefinitely a reject.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No concerns", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "1 hour"}, "extra_text": {}}]

Author rebuttal:

Meta-review: None of the reviewers recommend accepting this paper.
There was substantive discussion during the rebuttal period, but the reviewers remained of the opinion that the work should not be accepted, with these opinions based on the quality of the presentation, their view of the novelty of the work and the degree to which the experiments demonstrated the utility of the proposed method.
The AC recommends rejection.
Decision: Accept (Poster)