Weak-shot Fine-grained Classification via Similarity Transfer
Authors: Junjie Chen
Li Niu
Liu Liu
Liqing Zhang
Venue: NeurIPS 2021 Poster (2021)
Keywords: transfer learning
similarity transfer
weak-shot learning

Abstract: Recognizing fine-grained categories remains a challenging task, due to the subtle distinctions among different subordinate categories, which results in the need of abundant annotated samples. To alleviate the data-hungry problem, we consider the problem of learning novel categories from web data with the support of a clean set of base categories, which is referred to as weak-shot learning. In this setting, we propose a method called SimTrans to transfer pairwise semantic similarity from base categories to novel categories. Specifically, we firstly train a similarity net on clean data, and then leverage the transferred similarity to denoise web training data using two simple yet effective strategies. In addition, we apply adversarial loss on similarity net to enhance the transferability of similarity. Comprehensive experiments demonstrate the effectiveness of our weak-shot setting and our SimTrans method. 

Weak-shot Fine-grained Classification via Similarity Transfer
Junjie Chen, Li Niu, Liu Liu, Liqing Zhang"
MoE Key Lab of Artificial Intelligence, Department of Computer Science and Engineering,
Shanghai Jiao Tong University
(chen.bys, ustcnewly, shirlley)asji edu. cn, chang-lq@css saaeelllossssstt... edu. cn
Abstract
Recognizing fine-grained categories remains a challenging task, due to the sub- tle distinctions among different subordinate categories, which results in the need of abundant annotated samples. To alleviate the data-hungry problem, we con- sider the problem of learning novel categories from web data with the support of clean set of base categories, which is referred to as weak-shot learning. In this setting, we propose method called SimTrans transfer pairwise seman- tic similarity from base categories to novel categories. Specifically, we firstly train a similarity net on clean data, and then leverage the transferred similarity to denoise web training data using two simple yet effective strategies. In addi- tion, we apply adversarial loss on similarity net to enhance the transferability of similarity. Comprehensive experiments demonstrate the effectiveness of our weak-shot setting and our SimTrans method. Datasets and codes are available https://gihubb.om//cm//// l'rans- Seeak-Sho-CClasificall
Introduction
Deep learning methods have made significant advance extensive computer vision tasks. A large part of this advance has come from the available large-scale labeled datasets. For fine-grained classification, it is necessary but more expensive to collect large-scale datasets. On the one hand, the subtle differences among fine-grained categories dramatically boost the demand for abun- dant samples. On the other hand, professional knowledge is usually required to annotate images for enormous subcategories belonging to one category. As consequence, fine-grained classification is critically limited by the scarcity of well-labeled training images.
In practice, we often have a set of base categories with sufficient well-labeled data, and the problem how to learn novel categories with less expense, in which base categories and novel categories have no overlap. Such problem motivates zero-shot learning [19] few-shot learning [6]] as well setting. To bridge the gap between base (resp., seen) categories and novel (resp., unseen) categories, zero-shot learning requires category-level semantic representation (e.g., word vector [27]] human annotated attributes [[II]) for all categories, while few-shot learning requires few clean examples (e.g., 5, 10) for novel categories. Despite the great success of zero-shot learning and few- shot learning, they have the following drawbacks: 1) Annotating attributes
few clean samples require expert knowledge, which is not always available; 2) Word vector is free, but much weaker than human annotated attributes [II]] Fortunately, large-scale images are freely available from public websites by using category names as queries, which is a promising data source to complement the learning of novel fine- -ennnng categories without any manual annotation.
*Corresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Similarity Transfer
Zero-shot learning Few-shot leurning Weak-shot learning
0.2
0.8
Buse:
0.9
0.1
0.4
Novel:
0.9
Learn pairwise similarity on base sel.
(2) Learn main classifier on novel set.
semantic repecseriation
well-labeled sample
O weak labeled sample
Figure I: Comparison among zero-shot learning, few-shot learning, and weak-shot learning. Dif- ferent colors indicate different categories.
We transfer the similarity learnt from base training set to enhance the main classifier learnt on novel training set. The boxes in differ- ent colors denote different categories. The num- bers above the arrows indicate similarities.
Figure 2:
Considering the drawbacks of zero/few-shot learning and the accessibility of free web data, we in- tend to learn novel categories by virtue of web data with the support of a clean set of base categories, which is referred to as weak-shot learning illustrated in Figure M Formally, given set of novel fine-grained categories which are not associated with any clean training images, we collect web im- ages for novel categories weak-labeled images and meanwhile leverage the clean images from base fine-grain fineggrined categories. We refer to the clean (resp.. web) image set from base (resp., novel) categories as base (resp., novel) training set. Weak-shot learning is a useful and important setting. By taking car model classification as an example, we already have the dataset CompCars [52]] with well-labeled images from base fine-grained categories, but we often need recognize other fine- grained categories beyond this scope, because there are a huge number of car models and new car models are also continuously emerging. In this case, we could apply our setting to recognize novel fine-graing fine-grained categories by collecting the web images for these categories. The closest related work to ours [ET]] but they further assumed the reliability of word vectors [27]] and the availability of unlabeled test images in the training stage.
In weak-shot setting, the key issue of novel training set is label noise, which will significantly de- grade the performance of learnt classifier [GT., [4] We explore using base training set to denoise novel training set, although they have disjoint category sets. As illustrated in Figure E our proposed framework employs the pairwise semantic similarity to bridge the gap between base categories and novel categories. The pairwise similarity which denotes whether two images belong to the same cat- egory is category-agnostic, so it is highly transferable across category sets even if they are disjoint. Meanwhile, the pairwise similarity can be easily learnt from limited data, indicating that small set of already annotated images could help learn extensive novel categories (see Section ) Analo- gously, some methods [B]] for few-shot learning transferred similarity from base categories to novel categories, which is directly used for classification. In contrast, we transfer the pairwise similarity to alleviate the label noise issue of web data. For learning from web data, some works [5X, [I] also attempted to denoise by similarity. Nevertheless, their similarities are derived from noisy samples, and likely to be corrupted due to noise overfitting [[7]].
Specifically, our framework consists of two training phases. Firstly, we train a similarity net (Sim- Net) [544] on base training set, which takes in two images and outputs the semantic similarity. Sec- ondly, we apply the trained SimNet to obtain the semantic similarities among web images. In this way, the similarity is transferred from base categories to novel categories. Based on the transferred similarities, we design two simple yet effective methods to assist in learning the main classifier on novel training set. 1) Sample weighting (i.e., assign small weights the images dissimilar to others) reduces the impact of outliers (web images with incorrect labels) and thus alleviates the problem of noise overfitting- 2) Graph regularization (i.e., pull close the features of semantically similar samples (60I) prevents the feature space from being disturbed by noisy labels. In addition, we pro- pose to apply adversarial loss [9] on SimNet to make it indistinguishable for base categories and novel categories, so that the transferability of similarity is enhanced. Since the key of our method is similarity transfer, we name our method SimTrans. We conduct extensive experiments on three fine- grained datasets to demonstrate that the pairwise similarity is highly transferable and dramatically

---

benefits learning from web data, even when the category sets are disjoint. Although the focus of this paper fine- aine-grined classification, we also explore the effectiveness of our setting and method on coarse-grained dataset ImageNet [S]. We summarize our contributions as
We propose to use transferred similarity to denoise web training data in weak-shot classifi- cation task, which has never been explored before.
We propose two simple yet effective methods using transferred similarity to tackle label noise: sample weighting and graph regularization.
One minor contribution is applying adversarial loss to similarity net to enhance the trans- ferability of similarity.
Extensive experiments on four datasets demonstrate the practicality of our weak-shot set- ting and the effectiveness of our SimTrans method.
2 Related Work
2.1 Zero-shot and Few-shot Learning
Zero-shot learning employs category-level semantic representation (e.g., word vector annotated at- tributes) to bridge the gap between seen (resp., base) categories and unseen (resp., novel) categories. A large part of works [ X. 32. ST
[0] learn mapping between visual features and category-level
semantic representations. Our learning scenario is closer to few-shot learning.
Few-shot learning depends on a few clean images (e.g., 5-shot or 10-shot) to learn each novel cate- gory, which could be roughly categorized the following three types. Optimization based methods [M, [6] optimize the classifier on variety of learning tasks (e.g.. learn each category by a few im- ages), such that it can solve new learning tasks using only small number images (e.g. learn novel categories with few images). Memory-based methods [5ir, 351, 28] employ memory architectures to store key training images or directly encode fast adaptation algorithms. Metric-based methods [39, 42) 33, B]] learn deep representation with similarity metric in feature space and classify test images nearest neighbors manner.
Concerning the problem setting, both zero-shot learning and few-shot learning ignore the freely available web images, whereas learn novel categories by collecting web images using category names as queries. Concerning the technical solution, metric-based few-shot learning methods mainly learn categry similarities and directly recognize test images according to the similarity. In contrast, we transfer image-image similarities to reveal the semantic relationships among web train- ing images, which are used denoise the web data for better classifier.
2.2 Webly Supervised Learning
Due to the data-hungry property of deep learning, learning from web data has attracted increasing attention. Many methods have been proposed to deal with noisy images by outlier removal [22] [r]
robust loss function [2y, 45, 577] label correction [64]] EK A6], multiple instance learning [61,, 56
and [55, I. 53 16. ROJ. A prevalent research direction closely related to ours is dealing with
label noise using similarities. To name a few, CurriculumNet [[ل]] computed Huclidean distances between image features, and then designed curriculums according to the distances. SelfLearning [[T] employed the cosine similarity between image features to select category prototypes and correct labels. SOMNet [ва] leveraged self-organizing memory module construct similarity between image and category, which could simultaneously tackle label noise and background noise of web images. However, the similarities used in above methods are derived from the noisy training set, which are likely to be corrupted by noise and lead to sub-optimal results [17 51]. To alleviate this problem, recent works [177, 51] introduced additional annotations to correct label noise. For example, CleanNet [17] directly learned the image-category similarity based on verification labels, which involves human verification of noisy images. Distinctive from the above methods, we do not require any manual annotation on crawled web images.
3

---

2.3 Weak-shot Learning
In broad sense, weak-shot learning means learning novel categories from cheap weak labels with the support of set of base categories already having strong labels. Similar weak-shot setting has been explored in other computer vision tasks, including object detection [T3 25, 58, B. 22].. semantic segmentation [[9]] and instance segmentation [044, [8]. For weak-shot object detection (also named muxed-superrvsed or cross-supervised), base categories have bounding box annotations while novel categories only have image class labels. For weak-shot semantic segmentation, base categories have semantic mask annotations while novel categories only have image class labels. For weak-shot instance segmentation (also named
base categories have instance mask anno- tations while novel categories only have bounding box annotations. To the best of our knowledge, we are the first work on weak-shot classification, which employs transferred similarity to denoise web training data.
3 Problem Definition
In the training stage, we have C2 base categories with No clean images for the c-th base category. Our main goal is learning to classify C, novel categories given No web training images for the c-th novel category. Base categories and novel categories have no overlap. No extra information (e.g., word vector) is required the training stage. In the testing stage, test images come from novel categories. We refer to the above setting as weak-shot learning.
For some weal-world applications, test images may come from both base categories and novel cate- gories, which is referred to as generalized weak-shot learning. In this paper, we focus on weak-shot learning, while leaving generalized weak-shot learning to Appendix.
4 Approach
Our training stage consists of two training phases: learning similarity net (SimNet) base training set and learning the main classifier on novel training set.
4.1 Learning SimNet on Base Training Set
Although there many forms of similarity, we choose an end-to-end deep network SimNet model the similarity function as in [54]]. The ar- chitecture SimNet shown in the top pipeline of Figure HL which takes in mini-batch of images and outputs the pairwise similarit/dissimilrriyy score for each pair of input images (e.g., M pairs for nati-batch size M). The enumeration (Enum) layer simply concatenates each pair of im- age features which are extracted by the backbone. For instance, suppose the image features have size M x D with the feature dimension D, then the output of the enumeration layer will have size M² 2D. Af- ter that, one fully-connected (FC) layer extracts feature for each concatenated feature pair, which is dubbed as relation feature of each pair. Finally, the relation features are supervised by classification loss with binary labels being "similar" (a pair of images from the same category) and "dissimilar" (a pair of images from different categories).
Backbonc
Enom &C
PC
Cls Loss
copy
Backbome
Enum &
D
Ad
Figure 3: Illustration of similarity net with adversarial loss. The top (resp., bottom) pipeline processes base (resp.. novel) training set. r (resp., r") represents base (resp., novel) re- lation features.
Considering the training and testing for SimNet, if we construct the mini-batch randomly, similar and dissimilar pairs will be dramatically imbalanced. Specifically, if there are C' categories in mini- batch, the probability for two images to from the same category &. To reduce the imbalance between similar pairs and dissimilar pairs, when constructing mini-batch, first randomly select

---

Cm (< C) categories and then randomly select M images from each selected category, as in [ C... We use Cm = 10 and M 100 for both training and testing of SimNet.
4.1.1 Adversarial Loss
Ideally, the learnt similarity is category-agnostin but there may exist domain gap between the rela- tion features of different category sets. Therefore, we use novel training set in the training stage to further reduce the domain gap between base categories and novel categories, as shown in the bottom pipeline of Figure B.
Specifically, a discriminator D-)) takes in relation features r in SimNet and recognizes whether they come from base categories (r") or novel categories (r"). The SimNet acts as generator, aiming to produce relation features which could not only confuse the discriminator but also benefit the relation classification. Note that the labels of novel training images are noisy, so we exclude the image pairs from novel categories from the relation classification loss. Analogous to [M], optimize SimNet and the discriminator alternatingly in an adversarial manner. Firstly, we freeze the generator and minimize the adversarial loss of discriminator. Secondly, with frozen discriminator, we minimize the relation classification loss of SimNet and maximize the adversarial loss of discriminator. The classification loss and the adversarial loss are balanced with parameee B, set as 0.1 via cross-validation. The optimizing is trivial and we leave the details to Appendix.
4.2 Learning Classifier Novel Training Set
Because of the label noise of web images, the performance will be significantly degraded when directly training the classifier using web data. To address the label noise issue, we employ two simple yet effective methods based on transferred similarities illustrated in Figure H. Transferred similarities mean the similarities among novel training samples, which are calculated by SimNet trained on base training set. Next, we will introduce these two methods, i.e., sample weighting and graph regularization, separately.
4.2.1 Sample Weighting
For the web images within a novel category, we observe that non- outliers (images with the correct label) are usually dominant, while outliers (images with incorrect labels) are from non-dominant inaccurate categories. When calculating the semantic similarities among web im- ages within novel category, outliers are more likely to dissimilar to most other images. Therefore, could determine whether an image is an outlier according to its similarities to other images.
Simi limilarit Matrix
W Weight
1.0
0.1
C.7
0.0
0.4
1.0
0.0
0.9
0.7
0.2
0,6
We
0.7
077
0.0
1.0
0.0
0.1
0.0
SimNet
0.9
C.0
0,
0.2
Avg
W4
0.6
0.7
6.1
0.8
10
0.0
0.4 W3
0.2
0.0
0.6
W2
Xss
W1
Reg
Cls
Loss
Loss
A
X3
Backbone
FC
Formally, for the e-th novel cate- gory with No web images, we first compute the similarity matrix S. E N XN=, with each entry Sc,.j be- ing pairwise similarity calculated by SimNet pre-trained on base training set. Although the size of S. may be large, there are only No times of backbone inference and N x N times of two FCs inference, which are computationally efficient. Then, we employ the average of similarities be-
Feature
Category
Figure 4: The overview of learning novel categories from web data. The similarity net (SimNet) outputs similarity ma- trix with pairwise similarities and generates sample weights. The sample weights are employed to weigh the main classi- fication loss (Cls Loss) of each image, while the similarity matrix is used in graph regularization loss (Reg Loss) based on image features.
5

---

tween image and all the other images as its weight:
No
1
8,,ij 8a...
We,s
No
Then, the sample weights are normalized to have unit mean, i.e., Be.i
2 /o.s/Ng As analysed
Wic.t
before, samples with lower weights We are more likely to outliers. Finally, we employ weighted classification loss based on sample weights Ue,s. In this way, assign lower weights to the training losses of outliers, which enables non-outliers to contribute more to learning a robust classifier:
M
Lete_
Σ m=1
ün log (Xmm) (Xm)
(2)
M
where M is the mini-batch size, Xm the mn-th image in the mini-batch, um its assigned weight, and f ((m)yy is its classification score corresponding to its category ym-
4.2.2 Graph Regularization
When directly learning novel training set, the feature graph structure, that is, the similarities among image features, are determined by noisy labels. In particular, the classification loss implicitly pulls features close for images with the same labels. However, the feature graph structure may be misled by noisy labels, so we attempt to use transferred similarities to rectify the misled feature graph structure. Specifically, we employ typical graph regularization [60]] based on transferred similarities to regulate features, which enforces the features of semantically similar images to be close.
Formally, for each mini-batch of M images, we first compute the similarity matrix E R MMM using the SimNet pre-trained base training set. Regarding the similarity matrix as adjacency matrix, the graph regularization loss is formulated as:
Leeg Σ Bi.]1/(x))) 1xxx
where 52, is each entry in S, and h(x:) the image feature of x; extracted by the backbone.
According to [АN], web images mainly have two types of noise: outlier noise and label-flip noise. Outlier noise means that an image does not belong to any category within the given category set, and label-flip noise means that an image is crroneously labeled as another category within the given category For the samples with label-flip noise, sample weighting directly discards these samples by assigning lower weights. However, graph regularization can utilize them to maintain reasonable feature graph structure and facilitate feature learning, which could complement sample weighting.
4.3
The Full Objective
On the whole, we train the classifier on novel training set by minimizing the weighted classification loss and graph regularization loss:
L = Lets, + arLergs
(4)
where a is Syper-parameter set as 0.1 by gross-validation
5 Experiments
5.1 Datasets and Implementation
We conduct experiments based on three fine-gra fne-grained datasets: CompCars [52]] (Car for short), CUB [48], and FGVC [[6]]. We split all categories into base categories and novel categories. The base training/test set and the novel test set are from the original dataset while the novel training set con- structed using web images. For instance, the novel training set of Car dataset is constructed using the released web images in WebCars [6]]], while the novel training sets of CUB and FGVC are con- structed by collecting the web images by ourselves. The statistics of three datasets are summarized

---

Table 1: The statistics of splits on three datasets. Category shows the number of split categories. Train (resp., Test) indicates the average number of training (resp., test) images for each category.
Table 2:
Module contributions on CUB dataset. Cls indicates the cross-entropy classification loss of main classifier. Ad means training Sim- Net with adversarial loss. Weight means sample
Dataset
Split
Category
Train
Test
weighting and Reg means graph regularization.
Base 323
37
34
Cls
Ad
Weight
Reg
Acc (%)
Car
Novel
108
510
36
85.4
Base
150
30
30
90.3
CUB
Novel
50
1000
30
89.5
Base
75
67
33
91.2
FGVC
Novel
25
1000
33
90.7
90.1
91.7
Table 3:
The first three rows show the performances (%) of SimNet evaluated by 150 base categories, 50 base categories, and 50 novel categories. The last shows the performance (%) of uniform random guess on 50 novel categories. PR, RR, and F1 represent pre-
Table 4: The performances (%) of main classifier supported by various similari- ties learnt from various training sets.
Source
Type
Classifier
Euclidean
86.2
cision rate, recall rate, and F1 Fc-re,,, respectively.
Novel
Cosine
86.4
Similar class
Dissimilar class
SimNet
87.6
PR
RR
F1
PR
RR
F1
Euclidean
87.5
Novel+
B-150
84.4
87.7
86.0
98.6
98.2
98.4
Cosine
87.6
Base
B-50
89.0
88.2
88.6
98.7
99.0
98.8
SimNet
89.5
N-50
88.4
87.6
88.0
98.6
98.7
98.7
Euclidean
88.8
Rand
10.0
50.0
16.7
90.0
50.0
64.3
Base
Cosine
89.1
SimNet
91.7
backbone. There is no overlap between novel categories and ImageNet 1/ categories. Besides, all baselines use the same backbone for fair comparison. More details of datasets and implementation are left Appendix.
in Table MI We employ ResNet50 [12]] pretrained on ImageNet [[]
Note that the focus of this work fine-grate fine-grained classification, because 1) it especially requires expert knowledge annotate labels for fine-grained data (a drawback of few-shot learning); 2) conjec- ture that semantic similarity should be more transferable across fine-grained categories. However, we also explore the effectiveness of our setting and method on a coarse-grained dataset ImageNet [S]] We leave the experimental results on ImageNet to Appendix.
5.2 Ablation Study
We conduct our ablation study on CUB dataset, considering its prevalence in extensive vision tasks. We evaluate the performances different combinations of our modules, and summarize the results in Table 2 Solely using sample weighting leads to a dramatic improvement (90.3% v.s. 85.4%). Solely enabling the graph regularization also results in a considerable advance (89.5% v.s. 85.4%). Jointly applying sample weights and graph regularization further improves the performance (91.2% v.s. 85.4%). The adversarial loss on SimNet boosts the performance of classifier for both sample weighting (90.7% 99..7% v.s. 90.3%) and graph regularization (90.1% 1% v.s. 89.5%). Finally, our full method outperforms the baseline by a large margin (91.7% v.s. 85.4%).
5.3 Investigating Similarity Transfer
We evaluate SimNet in various situations to explore the effectiveness of similarity transfer. Although SimNet employed to denoise web images, we cannot evaluate its performance on noisy images without ground-truth labels, so we perform evaluation on clean test images. The performance mea- surements (e.g., precision rate (PR), recall rate (RR), and FI-score (F1)) are computed at the pair level, instead of image level. The experiments in this subsection conducted on CUB dataset.
7

---

92-
90
90
80
90
90
89
150
125
75
5040
25
25
20
150
125
100
75
50400 25
25
10
(a) Classifier wir Cb-
Classifier wax NV
(c) Simnet wond Ca-
(d) SimNet No.
Figure 5: Performance variations of main classifier (accuracy %) or similarity net (Fl-score %) w.r.t
different numbers of used base categories (C₂) or used clean images per base category (No).
5.3.1 The Transferability of Similarity
The performance gap of SimNet between base categories and novel categories indicates the trans- ferability of learnt similarity. We first train SimNet on base training set with 150 categories, and evaluate its performances on base test set with 150 categories and novel test set with 50 categories. We also evaluate on the base test set formed by 50 random base categories to exclude the influence of different category numbers. The results are summarized in Table B Surprisingly, SimNet trained on base categories achieves a comparable performance for base categories and novel categories (Base- 50 v.s. Novel-50). On the whole, the above results suggest that the pairwise similarities are highly transferable across fine-gres one-grained categories.
5.3.2 The Comparison of Similarity Sources and Types
To demonstrate the superiority of the our transferred similarity, we compare different types of sim- ilarities learnt from various training sets (sources) in Table A For Euclidean and Cosine, pairwise similarity is computed based the features extracted by network pretrained on specific train- ing set (source). Cosine similarity is well-known and we omit the details here. For Euclidean, we adopt the reciprocal of Euclidean distance. For SimNet, (sources). By comparing different similarity sources, we observe that all types of similarities learned from base training set perform better, while the performances decrease when novel training set (con- taining noisy images) is involved. This verifies that the similarity could be severely corrupted by noisy images, indicating the necessity of learning similarity using clean images even if the category sets are disjoint. By comparing different similarity types, SimNet performs optimally, showing the superiority of learning similarity in an end-to-end fashion.
train it using different training sets
5.3.3 The Impact of the Scale of Base Training Set
We naturally wonder what is the minimum requirement of the scale of base training set for our similarity net and main classifier to perform well. We explore the scale of base training set from two aspects: category number and image number in each category. We mainly report the Fl-score of similar category evaluated on novel test set as the performance of SimNet, and the results are illustrated in Figure E. The subfigure (a) and (c) shows the robustness when using more than 50 categories, while the subfigure (b) and (d) shows robustness when using more than 5 images per category. Furthermore, we explore the joint impact of two aspects in Table E Surprisingly, supported by only 50 base categories with images in each category, the main classifier could achieve a satisfactory performance, i.e., 89.3% against 85.4%, where 85.4% is the result without the support of base training set. This study indicates the potential of our setting and method, that is, small-scale off-the-shelf base training set could facilitate learning extensive novel fine-grained categories.
5.4 Comparison with Prior Works
We compare with two types of methods: webly supervised learning and transfer learning.
For webly supervised learning, we compare with recent representative methods: SelfLearning [[]]] MetaCleaner [56]]. CurriculumNet [[8]] SOMNet [Аæ]], DivideMix [21]] LearnToLearn [20], NLNL [155], and JoCoR [27]]. Webly supervised learning methods only use novel training set.
For transfer learning across categories, there are mainly two groups of methods: few-shot learning and knowledge transfer. For few-shot learning, we compare with MetaBaseline [3]] MAML [А], and SNAIL [ MetaBaseline is a metric-based method, and we use the averaged feature of each novel category as class centroid following [3]. SNAIL memory-based method, and we use 16
prototypes of each novel category as 16 shots due to hardware limitation. For knowledge transfer,
8

---

Table 5: The performances (%) of SimNet and main classifier supported by various scales C' x N of base training set, in which C' denotes the used category number and N denotes the used image number each category.
Table 7: Accuracies (%) of various methods
on three datasets in the weak-shot learning. The best results are highlighted in boldface.
Method
Car
CUB
FGVC
Cls Loss
83.1
85.4
86.6
SelfLearning
85.1
87.3
88.1
C x
50x5
75x5
50x
150x30
MetaCleaner
84.9
87.1
88.3
SimNet
73.2
79.1
80.6
88.0
CurriculumNet
85.2
86.8
87.9
Classifier
89.3
89.9
90.4
91.7
SOMNet
86.0
87.9
88.6
DivideMix
86.2
88.0
89.1
Learn ToLen
85.3
87.6
88.4
Table 6: Accuracies (%) of representative meth- ods using various numbers of web images on CUB dataset.
NLNL
85.4
87.9
88.2
JoCoR
85.9
87.9
88.3
MetaBaseline
85.8
87.1
88.7
MAML
84.6
86.8
87.9
Number
200
400
600
800
1000
SNAIL
84.1
86.3
87.2
Cls Loss
78.5
82.7
84.3
85.2
85.4
Distillation
83.7
85.9
87.3
Fusionl
80.7
84.9
87.1
88.2
88.3
Finetune
84.2
86.5
87.6
Fusion2
81.0
85.1
87.2
88.3
88.5
Fusionl
86.8
88.3
89.7
Ours
84.3
89.5
91.0
91.6
91.7
Fusion2
86.9
88.5
90.0
SimTrans
89.8
91.7
92.8
the Distillation baseline [23] enhances the classifier for novel categories by additionally predicting category distribution over base categories, which is supervised by a classifier pre-trained on base training set. Another natural baseline, named as Finetune, trains the model on the mixture of base training set and novel training set, and then fine-tunes on novel training set.
In addition, as far as we are concerned, there no method which could jointly denoise web train- ing data and transfer across categories without additional information (e.g., word vector). So we combine competitive methods in their own tracks using late fusion (i.e., average prediction scores). The fusion "SOMNet+Metaaasslin videMix++Metaaaselnne is referred to as Fusion2.
is referred
Fusionl, while the fusion "Di-
All the results summarized in Table M We also include the basic baseline Cls Loss, which is trained on novel training set with standard classification loss (row Table 2). Based on Table we have the following observations:
1) Webly supervised learning methods outperform the basic baseline Cls Loss, showing the general improvement brought by denoising web data without any support.
2) Few-shot learning methods and knowledge transfer methods outperform the basic baseline Cls Loss, indicating the advantage of transfer learning across categories. Note that MetaBaseline achieves a commendable performance, probably because averaging features could denoise web im- ages to some extent. Nevertheless, the overall performances of transfer learning methods are limited by severe label noise.
3) The simple combination of webly supervised learning and transfer learning, i.e., Fusion] or Fu- sion2, outperforms the above baselines, which directly demonstrates the effectiveness of our weak- shot setting. Furthermore, our method achieves the optimal performance against all baselines, which indicates the superiority of sample weighting and graph regularization based on similarity transfer.
5.5 Qualitative Analysis for Weight and Graph
We visualize to verify whether the higher weights are assigned to clean images and the transferred similarities could reveal semantic relations. The visualizations and analyses left to Appendix.
5.6
Robustness Analysis of The Method
We analyse the robustness of our method several factors, including the number of web images, the selection of hhper-ppane parameters, the noise ratio in web images, and the impact of different backbones.

---

Table Accuracies (%) of different methods on three datasets in the generalized weak-sho setting. The best results are highlighted in boldface.
Method
Car
CUB
FGVC
Cls Loss
76.3
73.3
76.9
Fusionl
77.5
74.8
78.8
Fusion2
78.1
75.0
79.3
SimTraas
80.4
76.9
81.7
For the impact of the number of web images, in our setting, the training data consists of clean images for base categories and web images for novel categories. The impact of the former is investi- gated in Section 5.3,3, and here we conduct experiments using different numbers of web images for each novel category on CUB dataset. As shown in Table & the general performances are gradually saturated given more training images (e.g., more than 800 web images), and the observations of performance promotions against baselines are consistent across different numbers of web images.
For other factors (i.e., hyper-parameter, noise ratio, and backbones), left the analysis to Appendix considering the space limitation.
5.7 Extension to Generalized Setting
In the generalized weak-shot learning, we additionally include base test set from base categories in the test set, so that test images may come from a mixture of base categories and novel categories, which is more practical in real-world applications. We extend our method and baselines to general- ized weak-shot learning scenario for comparison.
For simplicity, we directly conduct the experiments by treating the base training set as web images with low noise rate. For our method, the first training stage of training SimNet remains the same, while the second training stage additionally includes base training set when training the main clas- sifier. Thus, the sample weights assigned base training images would be near one, and the graph regularization involving base training images will also do no harm to feature learning at least.
We compare with the basic baseline Cly Loss well learning method and transfer learning method across categories. SOMNet and DivideMix are also conducted by treating base training images as web images (the clean probability thresholds are set as 0 for base training images in DivideMix). MetaBaseline could be directly applied in the generalized setting. One practical problem is that the image number of base categories and the image number of novel categories are highly imbalanced (e.g., 30 and 1000 for CUB), so we weigh the classification loss of each category using their image numbers (higher weight for the category with fewer images) for all methods in this setting. The results are reported in Table & from which we can find that the fusion of transfer learning and webly supervised learning outperforms the basic baseline, while our method further improves the results by a large margin.
combinations of webly supervised
Conclusion
In this paper, we consider the problem of learning novel categories from easily available web images with the support of set of base categories with off-the-shelf well-labeled images. Specifically, have proposed to employ the pairwise semantic similarity to bridge the gap between base categories and novel categories. Based on the transferred similarities, two simple yet effective methods have been proposed to deal with label noise when learning the classifier on web data. Adversarial loss is also applied to enhance the transferability of similarity. Extensive experiments on four datasets have demonstrated the potential of our learning paradigm and the effectiveness of our method.
Acknowledgements
The work was supported by the National Key R&D Program of China (2018AAA0100704), National Natural Science Foundation of China (Grant No. 61902247). the Shanghai Municipal Science and Technology Major Project (Grant No. 2021 SHZDZX0102) and the Shanghai Municipal Science and Technology Key Project (Grant No. 2051 15 100300).
10

---

References
Zeynep Akata, Scott Reed, Daniel Walter, Honglak Lee, and Bernt Schiele. Evaluation of output embed- dings line-grained image classification, CVPR, 2015.
Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, and Fei Sha. Synthesized classifiers for zero-shot learning. CVPR, 2016.
Yinbo Chen, Xiaolong Wang. Zhuang Liu, Huijuan Xu, Trevor Darrell. A new meta-baseline for few-shot learning. In ICLR, 2020.
[4] Zitian Chen, Zhiqiang Shen, Jiahui Yu, and Erik Learned-Miller Cross-supervised object detection, arXiv preprint 2006 15056, 2020.
[5] Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchi- cal image databasc. In CVPR, 2009.
[6] Li Fei-Fei, Rob Fergus, and Pietro Perona. One-shot learning of object categories. JEEE transactions on pattern analysis and machine intelligence, 28(4):594-66 611, 2006.
[7] Chelsea Finn, Pieter Abbeel, and Sergey Levine. deep networks. In ICML, 2017.
Model-agnostic meta-lcarning for fast adaptation of
[8] Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean. Marc' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. In NeurlPS, 2013.
[9] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014.
[10] Zhangxuan Gu, Siyuan Zhou, Li Niu, Zihan Zhao, and Liqing Zhang. Context-aware feature generation for zero-shot semantic segmentation. In ACM MM, 2020.
[11] Jiangfan Han, Ping Luo, and Xiaogang Wang. Deep self-learning from noisy labels. In ICCV, 2019.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[13] Judy Hoffman, Sergio Guadarrama, Eric Tzeng, Ronghang Hu, Jeff Donahue, Ross Girshick, Trevor Darrell, and Kate Saenko. Lsda: Large scale detection through adaptation. NIPS, 2014.
[14] Ronghang Hu, Piotr Dollár, Kaiming He, Trevor Darrell, and Ross Girshick. Learning to segment every thing. In CVPR, 2018.
[15] Youngdong Kim, Junho Yim, Juseung Yun, and Junmo Kim. Ninl: Negative learning for noisy labels. In ICCV, 2019.
[16] Jonathan Krause, Benjamin Sapp, Andrew Howard, Howard Zhou, Alexander Toshev, Tom Duerig, James Philbin, and Li Fei-Fei. The unrcasonable effectiveness of noisy data for fine-grained recognition. In ECCV, 2016.
[17] Lei Zhang Kuang-Huce Lee, Xiaodong He and Linjun Yang. image classifier training with label noise. In CVPR, 2018.
Cleannet: Transfer learning for scalable
[18] Weicheng Kuo, Anelia Angelova, Jitendra Malik, and Tsung- Yi Lin. Shapemask: Learning to segment novel objects by refining shape priors. In ICCV, 2019.
[19] Christoph H Lampert. Hannes Nickisch, and Stefan Harmeling. Attribute-base classification for zero- shot visual object categorization. IEEE transactions pattern analysis and machine intelligence, 36(3): 453-465, 2013.
[20] Junnan Li, Yongkang Wong, Qi Zhao, and Mohan Kankanhalli, Learning to learn from noisy labeled data. In CVPR, 2019.
[21] Junnan Li, Richard Socher, and Steven C.H. Hoi. supervised learning. In ICLR, 2020.
Dividemix: Learning with noisy labels as semi-
[22] Yan Li, Junge Zhang, Kaiqi Huang, and Jianguo Zhang. Mixed supervised object detection with robust objectness transfer. IEEE transactions on pattern analysis and machine intelligence, 41(3):639 653, 2018.

---

[23] Qing Liu, Lingxi Xie, Huiyu Wang, and Alan L Yuille. Semantic-aware knowledge preservation for zero-shot sketch-based image retrieval. In ICCV, 2019.
[24] Wei Liu, Gang Hua, and John R Smith. Unsupervised cle-slass learning for automatic outlier removal. In CVPR, 2014.
[25] Yan Liu, Zhijie Zhang, Li Niu, Junjie Chen, and Liqing Zhang. Mixed supervised object detection by transferring mask prior and semantic similarity. In Neur/PS, 2021.
[26] S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013.
[27] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In NeurlPS, 2013.
[28] Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In ICLR, 2018.
[29] Volodymyr Mnih and Geoffrey E. Hinton. Learning to label aerial images from noisy data. In /CMI, 2012.
[30] Li Niu, Qingtao Tang, Ashok Veeraraghavan, and Ashutosh Sabharwal. Learning from noisy web data with category-level supervision. In CVPR, 2018.
[31] Li Niu, Ashok Veeraraghavan, and Ashutosh Sabharwal. Webly supervised learning meets zero-shot learning: A hybrid approach for fine-grained classification. In CVPR, 2018.
[32] Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer. Jonathon Shlens, Andrea Frome. Greg S Corrado, and Jeffrey Dean. Zero-shot learning by convex combination of semantic embeddings. In ICLR, 2014.
[33] Boris N Oreshkin, Pau Rodriguez Lopez, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In NeurlPS, 2018.
[34] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In CVPR, 2017.
[35] Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. In ICMI., 2016.
[36] Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell, Meta-learning with latent embedding optimization. In ICLR, 2019.
[37] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory-augmented neural networks. ICML, 2016.
[38] Haozhi Zhang Chenfan Zhuang Dengke Dong Matthew R. Scott Sheng Guo, Weilin Huang and Dinglong Huang. Curriculumnet: Weakly supervised learning from large-scale web images. In ECCV, 2018
[39] Jake Snell, Kevin Swersky, and Richard S Zemel. Prototypical networks for sew-shot learning. In NeurlPS, 2017.
[40] Sainhayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, and Rob Fergus. Training convo- lutional networks with noisy labels. In ICLR workshop, 2015.
[41] Xiaoxiao Sun, Liyi Chen, and Jufeng Yang. Learning from web data using adversarial discriminative neural networks for fine-grained classification. In AAAI, 2019.
[42] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip S. Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In CVPR, 2018.
[43] Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In NewrIPS, 2018.
[44] Yi Tu, Li Niu, Junjie Chen, Dawei Cheng, and Liqing Zhang, Learning from web data with self-organizing memory module. In CVPR, 2020.
[45] Brendan Van Rooyen, Aditya Menon, and Robert C Williamson. Learning with symmetric label noise: The importance of being unhinged. In NewrIPS, 2015.
[46] Andreas Veit, Neil Alldrin, Gal Chechik, Ivan Krasin, Abhinav Gupta, and Serge Belongie. Learning from noisy large-scale datasets with minimal supervision. In CVPR, 2017.
12

---

[47] Hongxin Wei, Lei Feng. Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In CVPR, 2020.
[48] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSSD Birds California Institute of Technology. 2010.
200. Technical Report
[49] Yan Xia, Xudong Cao, Fang Wen, Gang Hua, and Jian Sun. Learning discriminative reconstructions for unsupervised outlier removal. In CVPR, 2015.
[50] Yongqin Xian, Zeynep Akata, Gaurav Sharma, Quynh Nguyen, Matthias Hein, and Bernt Schiele. Latent embeddings for shot classification. In CVPR. 2016.
[51] Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In CVPR, 2015.
[52] Linjie Yang, Ping Luo, Chen Change Loy, and Xiaoou Tang. A large-scale datasct for fine-grained categorization and verification. In CVPR, 2015.
[53] Jiangehao Yao, Hao Wu, Ya Zhang, Ivor W Tsang, and Jun Sun. Safeguarded dynamic label regression for noisy supervision. In AAAI, 2019.
[54] Zhaoyang Lv Yen-Chang Hsu and Zsolt Kira. Learning to cluster in order transfer across domains and tasks. In ICLR, 2018.
[55] Chuanyi Zhang, Yazhou Yao, Huafeng Liu, Guo-Sen Xie, Xiangho Shu, Tianfei Zhou, Zheng Zhang, Fumin Shen, and Zhenmin Tang. Web-supervised network with softly update-drop training for finc- grained visual classification. In AAAI, 2020.
[56] Weihe Zhang, Yali Wang, and Yu Qiao. Metacleaner: Learning to hallucinate clean representations for noisy-laheled visual recognition. In CVPR, 2019.
[57] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. NeurlPS, 2018
[58] Yuanyi Zhong, Jianfeng Wang, Jian Peng, and Lei Zhang. Boosting weakly supervised ohject detection with progressive knowledge transfer. ECCV, 2020.
[59] Siyuan Zhou, Li Niu, Jianlou Si, Chen Qian, and Liqing Zhang. Weak semantic segmentation by transferring semantic affinity and boundary. arXiv preprint 1rX1592110001519, 2021.
[60] Xiaojin Zhu, Zoubin Ghahramani, and John D Lafferty. Semi-supervised lcarning using gaussian fields and harmonic functions. In ICML, 2003.
[61] Bohan Zhuang, Linggiao Liu, Yao Li, Chunhua Shen, Ian Reid. Attend in groups: weakly- supervised deep learning framework leamning from web data, In CVPR, 2017.
13


Reviews:
[{"review_id": "UzXte_UQ76", "reviewer": "Reviewer_4wp8", "summary": "This paper proposes a new setting called weak-shot fine-grained classification, which contains base classes with clean labels and novel classes with noisy labels. To boost the performance of the noisy novel classes, the paper adopts sample weighting and similarity regularization for better tackling the noisy issue. The experiments show the superiority of the proposed method.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n* The writing is good, with smooth logic and clear illustration.\n* The experiments are solid and sufficient to support the claim.\n\nMy major concern is also the motivation of the proposed setting. In my view, if we can obtain the weakly supervised data from the web and the label accuracy is not so bad, then it's easy and cheap to clean the web dataset (For example, utilizing crowdsourcing). Furthermore, for the application scenarios that truly need expert knowledge for human annotation such as intelligent medical, it's hard to collect the labeled data from the web.\n\nAnother concern is the ImageNet pre-train. It is not a standard practice in zero-shot and few-shot learning. How about the performance without ImageNet pre-train? Is it fair to compare with the baseline methods?\n\nOverall, the writing quality of this paper is good, but the proposed setting is not so practical and the methodological novelty is limited, so I vote negative to this paper.\n\nMinor suggestion:\n* There are a lot of experiments that are only shown in the appendix, so I suggest you move sections 5.8 - 5.10 to the appendix.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "M9-oGrK051", "reviewer": "Reviewer_QXjD", "summary": "The paper proposes a simple approach using sample weighting and graph regularization for weak-shot fine-grained classification and demonstrates promising results on CUB, CAR and FGVC datasets. In weak-shot classification, training samples for novel classes are harvested from web search. The paper presents includes results from various experiments most of which give intuitive conclusions such as (1) more categories or images per category in the training data is better to get improved performance metric on novel set (2) Similarity model is better when trained on clean data with known base classes and gets worse when we use noisy data with novel categories.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper demonstrates improved quality of performance on 3 datasets for fine-grained classification. The problem set up is weak-shot where the base classes in the training set have clean samples while the novel classes have noisy samples from web search. ResNet50 trained on ImageNet-1k is used as backbone. There is no overlap between novel class categories and ImageNet-1k. The paper uses a sample weighting scheme which gives low weights for potential outliers as measured by similarity between images by SimNet pre-trained on base training set. The classifier is trained on novel training set using a blend of weighted classification loss and graph regularization loss. \n\nThe paper contains extensive experiments while the outcomes are obvious. The paper was written with clarity with clear motivations and outcomes. Novelty is low since the main contributions are sample weighting, graph regularization and use of machine learned similarity to improve precision of labels in noisy data.\n\n## Post rebuttal ...\nThanks to authors for their feedback - specifically response to \"Q2. The analysis does not give insights into performance of popular and rare classes. Web search of rare class will give more noisy results than popular classes.\". I would like to change my final rating from `Marginally below the acceptance threshold` to `Marginally above the acceptance threshold`\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1) It is not clear how the novel classes were chosen in the experiment. \n2) The analysis does not give insights into performance of popular and rare classes. Web search of rare class will give more noisy results than popular classes. Samples for novel classes were obtained from the web. \n3) Quality of search engines have been dramatically improving over the past few years. It is not clear if comparison with other webly supervised approaches was thus done fairly.\n4) The paper does not list negative societal impact. Use of web search to harvest data to train models can have negative consequence for sensitive applications.\n", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "87EbQ2ho6er", "reviewer": "Reviewer_31ZF", "summary": "This paper proposes an interesting setting, dubbed as weak-shot learning, using limited label resources to recognize novel categories (weakly labeled with certain noises) on the web. \nThe key challenge in weak-shot learning is label noise in the novel category set. This paper proposes to transfer pairwise similarities learned from a clean base set to the novel category set.\nSuch similarities are class-agnostic and easy-to-learn, which is believed to be highly transferable between sets.\nTo further augment the proposed method with fewer noise perturbations, two strategies are designed: 1) sample weighting and 2) graph regularization, which are also evaluated with ablation studies and show significant advances.\nThe proposed method is well evaluated with extensive experiments, and the results verify the effectiveness.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n- The investigated scenario has not been considered before, which is also of practical significance.\n- This paper is well-motivated, well-organized, clearly written, and easy to follow.\n- The proposed solution is simple yet powerful, which is well-evaluated with extensive experiments.\n\n## Weaknesses\n- Lack in discussion of pairwise similarity based methods in related work. Are there any methods that also transfer such similarities? What is the difference between them and the proposed one?\n- Considering that the key issue is the label noise of the novel category set, it could be better to model such an issue with mathematical language. While the proposed solution is quite intuitive -- yet with strong performance -- the insights may be limited. That is to say, the readers may obtain limited knowledge that can be \"transferred\" to their own areas of interest. Honestly, the authors should always bear in mind this ultimate standard, but I still think this is a good paper.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations are well discussed in the appendix.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Following the rebuttal and discussion period, all reviewers lean towards acceptance of this work. The reviewers found the problem statement to be of practical importance and the paper well motivated and well written. Questions raised by QXjD and 4wp8 were adequately addressed during the rebuttal/discussion phase related to analysis of performance vs rarity of class and overall motivation for the work. 
Decision: Accept (Poster)