Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: summary: The paper is about transferring the semantic information between datasets to learn from the web data. A combination of known techniques such as SimNet [47], the graph regularization [51] and adversarial loss [8] are applied to obtain results.
main_review: The idea of using web supervision is interesting and there are large-scale web-based datasets available, these are only a few relevant examples: https://paperswithcode.com/dataset/pinterest, https://arxiv.org/pdf/1611.08321.pdf, https://aclanthology.org/2021.naacl-main.473.pdf. The main motivation of the work seems to be able to learn from web data and transfer the leanings to other datasets. I have the following concerns about this work.

- I do not believe the authors have adequately addressed the previous reviews
- From the architectural standpoint, the novelty is very limited. A major contribution could have been a definition of new transfer task from the web data.
- However, in my opinion the authors fail to design the experiments to prove the concept of transferrability from the web data. In particular, I do not believe they use a large-scale web-based dataset as a base and then demonstrate successful transfer to another dataset. 
- The paper is not clearly written. The introduction is jumping around zero-/few- shot and web data concepts without ever clearly defining the use case that the authors want to address. In fact, I cannot figure out the use case even after reading the entire article, because the authors claim that they can learn and transfer from the web data, but experiments do not consider a real-life web dataset at all. The Problem Definition section provides a really short and abstract definition of the task. In my opinion a detailed description of the real use case transitioning to its mathematical description would have done a much better job here. The Experiments section has way too many subsections and it is very hard to navigate. There is no clear message in the section and none of the sections really addresses the main question (according to my guess) whether web driven learning is viable in the context of the task that the authors consider. The research questions that the authors want to answer in the section are not defined either, so it is really hard to understand the motivation behind ten different subsections of Section 5. I believe the authors themselves do not see the forest behind the trees and it is even worse for the reader. Please define your research questions and the message you would like to send with your research in a clear and concise way.

Definitely a reject.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper addresses a new task called weak-shot learning. It is to learn novel categories by using the knowledge learned in the clean set of base categories. The clean set of base categories refers to the clean, well-labeled image classification dataset. 

The proposed method can be summarized into two components. First, the authors train a SimNet on the clean base dataset. It is to learn to capture the semantic concepts from the images. Secondly, the authors conduct supervised classification training on the novel images using similarity labels, where the similarity labels are obtained from the similarity matrix computed by the pre-trained SimNet. To address the label noise in the unseen dataset, sample weighting and graph regularization are employed to better leverage the similarity matrix in training.
main_review: Strength
- The authors propose an interesting and practical task, and the authors present good baseline to address the problem.
- The idea of similarity transfer is good and reasonable. 
- The use of SimNet, and the construction of the relation features is interesting.
- Large number of experiments are conducted to verify different factors in the proposed method.

Weakness
- It is a complicated paper. The paper seems revised many times, but the presentation is not very effective in my point of view. From a high-level view, I think the main goal is to train a novel-category classifier using the regularizations obtained from clean base categories. It might be easier to understand the paper (at least for me, or general audience) by first presenting the overall pipeline and then talk about Sec 4.2. After that, discuss the regularizations terms such as how to obtain the similarity matrix. In addition, in my opinion, the paper is more relevant to web supervision but not zero/few-shot learning. Probably the term "weak-shot learning" is somehow confusing. Moreover, I see the proposed idea fundamentally work for general image classification problem, but the authors tend to restrict the presentation to only fine-grained image classification. To make the paper easier to follow, I would suggest to first address a simple, general problem (general image classification), and then extend it to a more complicated task (fine-grained image classification).
- In current presentation, the authors present a similarity transfer method. One intuitive question is how to apply such method to real-world problems? Is there any restriction? Although the base and novel sets are disjoint in the experiments, one can see the base and novel sets are from the same dataset such as CUB. This implies there are requirements for the base and novel sets. They must belong to the same meta-category, such as bird. If we would like to train a novel bird classifier, but only CompCars dataset is available as the clean base set, I am not sure if the transferability still hold. This is a natural question when we deal with similarity transfer between fine-grained datasets. Instead of dealing with fine-grained dataset, I feel it is more appealing to do general image classification. By doing so, the whole story becomes more elegant, flexible, and easy to sell. 
- The proposed task is too complicated. As the results, the authors have to conduct very large amount of experiments to verify many components, and it is very difficult to present all the experiments in the main paper. Instead of simply leaving all the analyses to Appendix, the authors should also summarize the key findings of the experiments in the main paper. 

Overall
- The paper addresses an interesting problem, and the proposed method should be technically correct. However, the paper is presenting in a way that is not easy to follow. I think it is a borderline paper. Both the proposed task and method are OK, but it is fine to reject it as well.

****post rebuttal****
Thank you for the response. I would like to keep my rating. 
limitations_and_societal_impact: n/a
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes to transfer pairwise semantic similarity from clean image data to web image data for classifying novel fine-grained categories. The proposed method provides a way to alleviate the data-hungry problem in the fine-grained classification task.
main_review: This paper proposes to utilize the easily available web data to learn novel fine-grained categories by transferring the pairwise semantic similarity from clean data to web data. Experiments are conducted for effectiveness validation on three public datasets. My concerns are as follows:

1.	This paper holds the premise that most web images are with correct labels and utilize the proposed SimNet trained on clean data to filter out the noisy web data. Dose the proposed method still work when the crawled web data are very dirty? Corresponding experiments or theoretical analyses should be added.
2.	When the training set is composed of the base training set and the novel training set, the classification accuracy drops a lot in section 5.10. Respective classification results in the base test set and the novel test set should be given. Corresponding experiments and analyses should be added, which is essential for verifying the effectiveness of the proposed method in the real-world applications.
3.	The authors should describe the difference between weak-shot learning and zero/few-shot learning more clearly. Respective training and test stages should be presented for better understanding of readers.
4.	More qualitative results about finding the outliers among the web data should be added to prove the effectiveness of the proposed method visually.
5.	The layout of this paper should be carefully designed. For example, Figure 3, Figure 4 and their surrounding text should be reset for better readability.

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposes to use transferred similarity to denoise web training data in weak-shot fine-grained classification task, with two simple yet effective strategies methods, sample weighting and graph regularization.

main_review: 1. The novelty and contribution of this work are marginal. Although the employment of transferred similarity to denoise web training data in weak-shot fine-grained classification task is new, as claimed in the paper, the combination of SimNet and DANN (Domain Adversarial Neural Networks) fails to surprise me. It seems that the proposed sample weighting and graph regularization share similar idea and intuition with MADA (Multi-adversarial Domain Adaptation), making the technical novelty not quite significant.
2. The organization of the manuscript could be improved. I think it is not a reader-friendly choice to leave too many detailed results and analyses to Appendix (Section 5.8, Section 5.9, and Section 5.10), which may make the flow of the paper hard to follow.


limitations_and_societal_impact: Please refer to the section "Main Review".
review_title: null

Meta-review: None of the reviewers recommend accepting this paper.
There was substantive discussion during the rebuttal period, but the reviewers remained of the opinion that the work should not be accepted, with these opinions based on the quality of the presentation, their view of the novelty of the work and the degree to which the experiments demonstrated the utility of the proposed method.
The AC recommends rejection.
Decision: Accept (Poster)