Weak-shot Fine-grained Classification via Similarity Transfer

Abstract: Recognizing fine-grained categories remains a challenging task, due to the subtle distinctions among different subordinate categories, which results in the need of abundant annotated samples. To alleviate the data-hungry problem, we consider the problem of learning novel categories from web data with the support of a clean set of base categories, which is referred to as weak-shot learning. In this setting, we propose a method called SimTrans to transfer pairwise semantic similarity from base categories to novel categories. Specifically, we firstly train a similarity net on clean data, and then leverage the transferred similarity to denoise web training data using two simple yet effective strategies. In addition, we apply adversarial loss on similarity net to enhance the transferability of similarity. Comprehensive experiments demonstrate the effectiveness of our weak-shot setting and our SimTrans method. 


Reviews and author responses:

This paper proposes a new setting called weak-shot fine-grained classification, which contains base classes with clean labels and novel classes with noisy labels. To boost the performance of the noisy novel classes, the paper adopts sample weighting and similarity regularization for better tackling the noisy issue. The experiments show the superiority of the proposed method.  Pros:
* The writing is good, with smooth logic and clear illustration.
* The experiments are solid and sufficient to support the claim.

My major concern is also the motivation of the proposed setting. In my view, if we can obtain the weakly supervised data from the web and the label accuracy is not so bad, then it's easy and cheap to clean the web dataset (For example, utilizing crowdsourcing). Furthermore, for the application scenarios that truly need expert knowledge for human annotation such as intelligent medical, it's hard to collect the labeled data from the web.

Another concern is the ImageNet pre-train. It is not a standard practice in zero-shot and few-shot learning. How about the performance without ImageNet pre-train? Is it fair to compare with the baseline methods?

Overall, the writing quality of this paper is good, but the proposed setting is not so practical and the methodological novelty is limited, so I vote negative to this paper.

Minor suggestion:
* There are a lot of experiments that are only shown in the appendix, so I suggest you move sections 5.8 - 5.10 to the appendix. Yes

The paper proposes a simple approach using sample weighting and graph regularization for weak-shot fine-grained classification and demonstrates promising results on CUB, CAR and FGVC datasets. In weak-shot classification, training samples for novel classes are harvested from web search. The paper presents includes results from various experiments most of which give intuitive conclusions such as (1) more categories or images per category in the training data is better to get improved performance metric on novel set (2) Similarity model is better when trained on clean data with known base classes and gets worse when we use noisy data with novel categories.  The paper demonstrates improved quality of performance on 3 datasets for fine-grained classification. The problem set up is weak-shot where the base classes in the training set have clean samples while the novel classes have noisy samples from web search. ResNet50 trained on ImageNet-1k is used as backbone. There is no overlap between novel class categories and ImageNet-1k. The paper uses a sample weighting scheme which gives low weights for potential outliers as measured by similarity between images by SimNet pre-trained on base training set. The classifier is trained on novel training set using a blend of weighted classification loss and graph regularization loss. 

The paper contains extensive experiments while the outcomes are obvious. The paper was written with clarity with clear motivations and outcomes. Novelty is low since the main contributions are sample weighting, graph regularization and use of machine learned similarity to improve precision of labels in noisy data.

## Post rebuttal ...
Thanks to authors for their feedback - specifically response to "Q2. The analysis does not give insights into performance of popular and rare classes. Web search of rare class will give more noisy results than popular classes.". I would like to change my final rating from `Marginally below the acceptance threshold` to `Marginally above the acceptance threshold`
 1) It is not clear how the novel classes were chosen in the experiment. 
2) The analysis does not give insights into performance of popular and rare classes. Web search of rare class will give more noisy results than popular classes. Samples for novel classes were obtained from the web. 
3) Quality of search engines have been dramatically improving over the past few years. It is not clear if comparison with other webly supervised approaches was thus done fairly.
4) The paper does not list negative societal impact. Use of web search to harvest data to train models can have negative consequence for sensitive applications.


 **Q1**. It is not clear how the novel classes were chosen in the experiment.

**A1**. The detailed base/novel splits are described in Line 27-29 of Appendix. Specifically, for CUB, we follow [11, 6], which is commonly used for zero/few-shot learning. For Car and FGVC, we randomly split base and novel categories by 3 : 1, following the split ratio in [11, 6]. There is no overlap between novel categories and ImageNet 1k categories, as claimed in Line 235-236.

---

**Q2**. The analysis does not give insights into performance of popular and rare classes. Web search of rare class will give more noisy results than popular classes.

**A2**. Firstly, it is hard to formally define the popular and rare classes. One criterion may be the noise ratio of each class, but we have no ground-truth labels for novel classes from web search and thus we cannot examine the noisy ratio for each class in the real-world application. Another criterion may be the image number of each class from web search, but there are abundant web images for all novel classes. 

Nevertheless, we try to employ the number of clean images in each novel class from the test set as the criterion (clean images from training set are not available for novel classes in our setting). The image numbers in CUB and FGVC datasets are balanced, so we only report for Car dataset. We group the novel categories as "rare/moderate/popular" three levels according to their image numbers in the test set, and summarize the results in Tab. 1 below. We can find that the overall performance of "rare" group is inferior to "popular" group, probably due to more noise in web training images. The performance gains on "rare" group and "popular" group of our method are comparable. 

In addition, we also explore the impact of the noise ratio using synthetic noisy data to demonstrate the robustness of our setting and method in Section "5.8 The Impact of the Noise Ratio in Web Images".

**Tab. 1**. Accuracies (%) of class groups of different popularity. The number of categories in "rare/moderate/popular" groups are equal, i.e., containing 36 categories. "Fusion1" and "Fusion2" are the short names for "SOMNet+MetaBaseline" and "DivideMix+MetaBaseline", which are the two most competitive baselines as descriped in Line 312-314.

| Method | Rare | Moderate | Popular |
 :-: | :-: | :-:| :-:
| Cls Loss | 79.5 | 80.6     | 85.3    |
| Fusion1  | 82.9 | 83.2     | 89.6    |
| Fusion2  | 83.4 | 84.5     | 89.0    |
| Ours     | 86.0 | 86.8     | 92.3    |

---
**Q3**. Quality of search engines have been dramatically improving over the past few years. It is not clear if comparison with other webly supervised approaches was thus done fairly.

**A3**. In this work, we run all methods using the same web data for fair comparison. The details of used web data can be found in Section "2 Datasets" in the Appendix.

 **Q1**: My major concern is also the motivation of the proposed setting. In my view, if we can obtain the weakly supervised data from the web and the label accuracy is not so bad, then it's easy and cheap to clean the web dataset (For example, utilizing crowdsourcing). Furthermore, for the application scenarios that truly need expert knowledge for human annotation such as intelligent medical, it's hard to collect the labeled data from the web.

**A1**. According to previous literature on webly supervised learning [34, 39, 13, 44, 49], cleaning web data is very expensive , especially for fine-grained categories, because the difference between fine-grained categories is very subtle and annotating fine-grained categories is in high demand of expert knowledge [34, 39, 12, 31]. As for the availability of web data, it is mainly determined by the popularity of the data over web. Fine-grained categories of bird, car, and aircraft require expert knowledge, but there are abundant web images. For the images hard to obtain from web, it is the problem for all webly supervised learning methods [9, 15, 16, 31, 40], instead of the unique problem of our setting. 

Our setting is very close to that in [24], which also collects web images for novel categories as weak-labeled images and meanwhile leverage the clean images from base fine-grained categories. However, [24] further assumed the reliability of word vectors as well as the availability of unlabeled test images in the training stage. Thus, our setting is at least more practical than [24].

Besides, similar weak-shot setting has been explored in object detection and segmentation. For the detection task, the setting named mixed supervised detection [*1, *2, *3] defines base categories as a set of categories having bounding box annotations and novel categories as another set of categories having only class labels. For the segmentation task [*4], the base categories are defined as a set of categories having instance segmentation mask annotations and novel categories are defined as another set of categories having only class labels. Anyway, all these works as well as ours focus on learning novel categories from cheap weak labels with the support of a set of base categories already having strong labels.

[*1] Li, Yan, et al. "Mixed supervised object detection with robust objectness transfer." IEEE transactions on pattern analysis and machine intelligence 41.3 (2018): 639-653.

[*2] Zhong, Yuanyi, et al. "Boosting weakly supervised object detection with progressive knowledge transfer." ECCV. 2020.

[*3] Chen, Zitian, et al. "Cross-Supervised Object Detection." arXiv preprint arXiv:2006.15056 (2020).

[*4] Kuo, Weicheng, et al. "Shapemask: Learning to segment novel objects by refining shape priors." CVPR. 2019.

---
**Q2**: Another concern is the ImageNet pre-train. It is not a standard practice in zero-shot and few-shot learning. How about the performance without ImageNet pre-train? Is it fair to compare with the baseline methods? 

**A2**: Because the baselines of webly supervised learning [16, 9, 37] usually pre-train on ImageNet, we have to pre-train the backbones on ImageNet for all baselines considering the overall fairness. We also have carefully inspected our base/novel category split to ensure that there is no overlap between novel categories and ImageNet 1k categories.

---
**Q3**. The methodological novelty is limited

**A3**. We acknowledge that the sample weighting and graph regularization have been widely used in previous literature but the key problem is how to use them in different tasks. We design our sample weighting and graph regularization based on the proposed cross-category similarity transfer, which is distinctive from previous works and well-tailored for the new weak-shot setting. Furthermore, since this is the first method in this new setting, we believe that it is not a drawback to make the proposed method simple and effective.


 **Q1**: Lack in discussion of pairwise similarity based methods in related work.

**A1**: Actually we have discussed some similarity based methods in related work, e.g., in few-shot learning (in Line 95-98) and webly supervised learning (in Line 109-110). Transferring pairwise similarities has never been explored in the weak-shot learning scenario before.

This paper proposes an interesting setting, dubbed as weak-shot learning, using limited label resources to recognize novel categories (weakly labeled with certain noises) on the web. 
The key challenge in weak-shot learning is label noise in the novel category set. This paper proposes to transfer pairwise similarities learned from a clean base set to the novel category set.
Such similarities are class-agnostic and easy-to-learn, which is believed to be highly transferable between sets.
To further augment the proposed method with fewer noise perturbations, two strategies are designed: 1) sample weighting and 2) graph regularization, which are also evaluated with ablation studies and show significant advances.
The proposed method is well evaluated with extensive experiments, and the results verify the effectiveness.
  ## Strengths
- The investigated scenario has not been considered before, which is also of practical significance.
- This paper is well-motivated, well-organized, clearly written, and easy to follow.
- The proposed solution is simple yet powerful, which is well-evaluated with extensive experiments.

## Weaknesses
- Lack in discussion of pairwise similarity based methods in related work. Are there any methods that also transfer such similarities? What is the difference between them and the proposed one?
- Considering that the key issue is the label noise of the novel category set, it could be better to model such an issue with mathematical language. While the proposed solution is quite intuitive -- yet with strong performance -- the insights may be limited. That is to say, the readers may obtain limited knowledge that can be "transferred" to their own areas of interest. Honestly, the authors should always bear in mind this ultimate standard, but I still think this is a good paper.
 The limitations are well discussed in the appendix.


Meta-review: Following the rebuttal and discussion period, all reviewers lean towards acceptance of this work. The reviewers found the problem statement to be of practical importance and the paper well motivated and well written. Questions raised by QXjD and 4wp8 were adequately addressed during the rebuttal/discussion phase related to analysis of performance vs rarity of class and overall motivation for the work. 

Decision: accept