Weak-shot Fine-grained Classification via Similarity Transfer

1 Introduction . Deep learning methods have made a significant advance on extensive computer vision tasks . A large part of this advance has come from the available large-scale labeled datasets . For fine-grained classification , it is more necessary but more expensive to collect large-scale datasets . On the one hand , the subtle differences among fine-grained categories dramatically boost the demand for abundant samples . On the other hand , professional knowledge is usually required to annotate images for enormous subcategories belonging to one category . As a consequence , fine-grained classification is critically limited by the scarcity of well-labeled training images . In practice , we often have a set of base categories with sufficient well-labeled data , and the problem is how to learn novel categories with less expense , in which base categories and novel categories have no overlap . Such problem motivates zero-shot learning [ 19 ] , few-shot learning [ 6 ] , as well as our setting . To bridge the gap between base ( resp. , seen ) categories and novel ( resp. , unseen ) categories , zero-shot learning requires category-level semantic representation ( e.g. , word vector [ 27 ] or human annotated attributes [ 19 ] ) for all categories , while few-shot learning requires a few clean examples ( e.g. , 5 , 10 ) for novel categories . Despite the great success of zero-shot learning and fewshot learning , they have the following drawbacks : 1 ) Annotating attributes or a few clean samples require expert knowledge , which is not always available ; 2 ) Word vector is free , but much weaker than human annotated attributes [ 1 ] . Fortunately , large-scale images are freely available from public websites by using category names as queries , which is a promising data source to complement the learning of novel fine-grained categories without any manual annotation . ∗Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Considering the drawbacks of zero/few-shot learning and the accessibility of free web data , we intend to learn novel categories by virtue of web data with the support of a clean set of base categories , which is referred to as weak-shot learning as illustrated in Figure 1 . Formally , given a set of novel fine-grained categories which are not associated with any clean training images , we collect web images for novel categories as weak-labeled images and meanwhile leverage the clean images from base fine-grained categories . We refer to the clean ( resp. , web ) image set from base ( resp. , novel ) categories as base ( resp. , novel ) training set . Weak-shot learning is a useful and important setting . By taking car model classification as an example , we already have the dataset CompCars [ 52 ] with well-labeled images from base fine-grained categories , but we often need to recognize other finegrained categories beyond this scope , because there are a huge number of car models and new car models are also continuously emerging . In this case , we could apply our setting to recognize novel fine-grained categories by collecting the web images for these categories . The closest related work to ours is [ 31 ] , but they further assumed the reliability of word vectors [ 27 ] and the availability of unlabeled test images in the training stage . In weak-shot setting , the key issue of novel training set is label noise , which will significantly degrade the performance of learnt classifier [ 31 , 34 ] . We explore using base training set to denoise novel training set , although they have disjoint category sets . As illustrated in Figure 2 , our proposed framework employs the pairwise semantic similarity to bridge the gap between base categories and novel categories . The pairwise similarity which denotes whether two images belong to the same category is category-agnostic , so it is highly transferable across category sets even if they are disjoint . Meanwhile , the pairwise similarity can be easily learnt from limited data , indicating that a small set of already annotated images could help learn extensive novel categories ( see Section 5.3 ) . Analogously , some methods [ 3 ] for few-shot learning transferred similarity from base categories to novel categories , which is directly used for classification . In contrast , we transfer the pairwise similarity to alleviate the label noise issue of web data . For learning from web data , some works [ 38 , 11 ] also attempted to denoise by similarity . Nevertheless , their similarities are derived from noisy samples , and likely to be corrupted due to noise overfitting [ 17 ] . Specifically , our framework consists of two training phases . Firstly , we train a similarity net ( SimNet ) [ 54 ] on base training set , which takes in two images and outputs the semantic similarity . Secondly , we apply the trained SimNet to obtain the semantic similarities among web images . In this way , the similarity is transferred from base categories to novel categories . Based on the transferred similarities , we design two simple yet effective methods to assist in learning the main classifier on novel training set . 1 ) Sample weighting ( i.e. , assign small weights to the images dissimilar to others ) reduces the impact of outliers ( web images with incorrect labels ) and thus alleviates the problem of noise overfitting . 2 ) Graph regularization ( i.e. , pull close the features of semantically similar samples [ 60 ] ) prevents the feature space from being disturbed by noisy labels . In addition , we propose to apply adversarial loss [ 9 ] on SimNet to make it indistinguishable for base categories and novel categories , so that the transferability of similarity is enhanced . Since the key of our method is similarity transfer , we name our method SimTrans . We conduct extensive experiments on three finegrained datasets to demonstrate that the pairwise similarity is highly transferable and dramatically benefits learning from web data , even when the category sets are disjoint . Although the focus of this paper is fine-grained classification , we also explore the effectiveness of our setting and method on a coarse-grained dataset ImageNet [ 5 ] . We summarize our contributions as • We propose to use transferred similarity to denoise web training data in weak-shot classification task , which has never been explored before . • We propose two simple yet effective methods using transferred similarity to tackle label noise : sample weighting and graph regularization . • One minor contribution is applying adversarial loss to similarity net to enhance the transferability of similarity . • Extensive experiments on four datasets demonstrate the practicality of our weak-shot setting and the effectiveness of our SimTrans method . 2 Related Work . 2.1 Zero-shot and Few-shot Learning . Zero-shot learning employs category-level semantic representation ( e.g. , word vector or annotated attributes ) to bridge the gap between seen ( resp. , base ) categories and unseen ( resp. , novel ) categories . A large part of works [ 2 , 8 , 32 , 50 , 10 ] learn a mapping between visual features and category-level semantic representations . Our learning scenario is closer to few-shot learning . Few-shot learning depends on a few clean images ( e.g. , 5-shot or 10-shot ) to learn each novel category , which could be roughly categorized as the following three types . Optimization-based methods [ 7 , 36 ] optimize the classifier on a variety of learning tasks ( e.g. , learn each category by a few images ) , such that it can solve new learning tasks using only a small number of images ( e.g. , learn novel categories with a few images ) . Memory-based methods [ 37 , 35 , 28 ] employ memory architectures to store key training images or directly encode fast adaptation algorithms . Metric-based methods [ 39 , 42 , 33 , 3 ] learn a deep representation with a similarity metric in feature space and classify test images in a nearest neighbors manner . Concerning the problem setting , both zero-shot learning and few-shot learning ignore the freely available web images , whereas we learn novel categories by collecting web images using category names as queries . Concerning the technical solution , metric-based few-shot learning methods mainly learn image-category similarities and directly recognize test images according to the similarity . In contrast , we transfer image-image similarities to reveal the semantic relationships among web training images , which are used to denoise the web data for a better classifier . 2.2 Webly Supervised Learning . Due to the data-hungry property of deep learning , learning from web data has attracted increasing attention . Many methods have been proposed to deal with noisy images by outlier removal [ 24 , 49 ] , robust loss function [ 29 , 45 , 57 ] , label correction [ 34 , 43 , 46 ] , multiple instance learning [ 61 , 56 ] , and so on [ 55 , 41 , 53 , 16 , 30 ] . A prevalent research direction closely related to ours is dealing with label noise using similarities . To name a few , CurriculumNet [ 38 ] computed Euclidean distances between image features , and then designed curriculums according to the distances . SelfLearning [ 11 ] employed the cosine similarity between image features to select category prototypes and correct labels . SOMNet [ 44 ] leveraged self-organizing memory module to construct similarity between image and category , which could simultaneously tackle label noise and background noise of web images . However , the similarities used in above methods are derived from the noisy training set , which are likely to be corrupted by noise and lead to sub-optimal results [ 17 , 51 ] . To alleviate this problem , recent works [ 17 , 51 ] introduced additional annotations to correct label noise . For example , CleanNet [ 17 ] directly learned the image-category similarity based on verification labels , which involves human verification of noisy images . Distinctive from the above methods , we do not require any manual annotation on crawled web images . 2.3 Weak-shot Learning . In broad sense , weak-shot learning means learning novel categories from cheap weak labels with the support of a set of base categories already having strong labels . Similar weak-shot setting has been explored in other computer vision tasks , including object detection [ 13 , 25 , 58 , 4 , 22 ] , semantic segmentation [ 59 ] , and instance segmentation [ 14 , 18 ] . For weak-shot object detection ( also named mixed-supervised or cross-supervised ) , base categories have bounding box annotations while novel categories only have image class labels . For weak-shot semantic segmentation , base categories have semantic mask annotations while novel categories only have image class labels . For weak-shot instance segmentation ( also named partially-supervised ) , base categories have instance mask annotations while novel categories only have bounding box annotations . To the best of our knowledge , we are the first work on weak-shot classification , which employs transferred similarity to denoise web training data . 3 Problem Definition . In the training stage , we have Cb base categories with N bc clean images for the c-th base category . Our main goal is learning to classify Cn novel categories given Nnc web training images for the c-th novel category . Base categories and novel categories have no overlap . No extra information ( e.g. , word vector ) is required in the training stage . In the testing stage , test images come from novel categories . We refer to the above setting as weak-shot learning . For some real-world applications , test images may come from both base categories and novel categories , which is referred to as generalized weak-shot learning . In this paper , we focus on weak-shot learning , while leaving generalized weak-shot learning to Appendix .

Summary: The paper proposes a simple approach using sample weighting and graph regularization for weak-shot fine-grained classification and demonstrates promising results on CUB, CAR and FGVC datasets. In weak-shot classification, training samples for novel classes are harvested from web search. The paper presents includes results from various experiments most of which give intuitive conclusions such as (1) more categories or images per category in the training data is better to get improved performance metric on novel set (2) Similarity model is better when trained on clean data with known base classes and gets worse when we use noisy data with novel categories.