Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors propose Relational Proxies, a novel approach that leverages the relational information between the global and local views of an object for encoding its semantic label.  
strengths_and_weaknesses: I think the main novelty comes from the introduced Relational Proxies and the corresponding comprehensive theoretical and experimental analysis.

Weaknesses One area of improvement for the paper at hand would be clarity, especially with respect to the exposition of the proposed architecture. It takes multiple read throughs in order to identify the actual architecture proposed.

Lack of experiments on larger datasets. In a time of ever-growing dataset sizes it would be good to provide and compare results of suchmodels when trained on larger datasets. This is important for judging the impact asimprovements stemming from architecture engineering typically vanish with growing dataset sizes.
questions: Insufficient experiments and limited performance gains.
limitations: N/A.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Summary.

This paper is dedicated to developing algorithms for fine-grained image recognition. They argue it is not enough to distinguish fine-grained categories only based on partial information. Therefore, they propose relational proxies, which leverage the relational information between the global and local views of an object. They also provide theoretical explanations to support the effectiveness of the proposed methods. Experiments on six fine-grained benchmark datasets offer positive results.
strengths_and_weaknesses: Pros.

1. The proposed methods make sense and are well-motivated. Both theoretical and empirical analyses are provided to support the effectiveness.

2. The paper is well written and easy to follow. Figure 1 is informative and illustrative.



Cons.

1. There are missing numbers in Table 1. For a comprehensive comparison, it is necessary to complete it. Minor: the caption of tables should be on the above content.

2. The performance gains are marginal, especially on CUB (0.3%) and NA Birds (0.2%)? Any explanations? It seems the proposed methods are less working for bird images. Meanwhile, the error bar is required for Table1 since the current accuracy margin is too small.

3. More network backbones are needed to support the generalization of proposed methods across architectures.

4. As for the study in Figure 2, the number of local views |L| and the size of the local patch should be correlated. A detailed analysis is needed.
questions: Refer to the weakness section.
limitations: Both limitations and potential negative social impacts are discussed in the submission.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In the fine-grained setting, discriminating between different classes requires learning how different local parts combine to form the object of interest. In this work, the authors introduce a theoretical framework and a novel method that decomposes FGVC tasks into relation-agnostic feature extraction and cross-view relation learning. They show the superiority of such method through a set of experiments.
strengths_and_weaknesses: The problem this work address is relevant. While I am unable to gauge the relevance of the proposed theoretical framework and its broad usefulness to the community, the proposed approach is solid and of interest.

Strengths:
1. **The experimental setup is solid.** The authors test their method on multiple datasets and consistently show competitive results across all of them. These results support the idea of modeling the cross-view relation in their proposed architecture. This is further demonstrated through ablation studies, that highlight the need for a cross-view relational function and provide further insights into the method.
2. The theoretical framework established in this work is clear, and while it builds on a lot of definitions, the proofs are simple and easy to follow. 

Weaknesses:
1. **The permutation invariance property is counter-intuitive** The authors introduce the permutation invariance property as a necessary property for a model to solve FGVC tasks. In section 3.4, they introduce a novel transformer (AST) that does not use positional embeddings and is thus permutation invariant. While the permutation invariance property can provide certain desired properties like potentially better generalization, the authors do not provide theoretical evidence for why it would be necessary, and it is not fully clear how it is motivated either. Certain claims in the introduction would actually suggest otherwise: “differ only in the way the attributes combine to generate the global view of the object” or “[features like] the distance between the head and the body, or the angular orientation of the legs”. This suggest that features like positional encoding would actually be critical. 
2. The term “relation” is not explicitly defined and it is unclear what the authors mean. The "relation-agnostic representation” is established in Definition 4, and while it is clear what it means in mathematical terms, its relation to FGVC problems is not evident. Providing more clarifications would make the text easier to follow.
3. **Decomposing the problem into relation-agnostic encoder and a cross-view relational function.** The authors do not argue for why this decomposition is necessary when solving FGVC problems- at least in the main text. A discussion of this can be found in appendix 6.3, and I would argue for including this in the main text as it better explains the idea of the relational gap and seems to at least provide an initial motivation for this decomposition.

questions: 1. In Definition 1, k-distinguishability is defined with respect to two classes only. What does this imply at the level of the entire dataset. What happens if different pairs of classes have different corresponding k values?
2. In settings where the datasets includes multiple subgroups of classes (cats, dogs, birds) that would be coarsely separable at the group level,  but would require fine-grained visual modeling within each group (in birds: white-faced plover vs. kentish plover). How would this approach change? How is k-distinguishability defined? How is the number of relational proxies c selected?

limitations: The authors discuss one of the main limitation, being the local view generation.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, authors propose to address the fine-grained image classification from a novel perspective, namely for the fine-grained classes, with the visually similar local features, more attention should be made on leveraging the relational information between the global and local views of an object for encoding its semantic label. Relational proxies are designed based on the proposed theory and achieve the superior results on multiple benchmark datasets.
strengths_and_weaknesses: Strengths:
A hypothesis is made for fine-grained visual classification, i.e. when two categories possess same local attributes and differ only in the way the attributes combine to generate the global view of the object, relation-agnostic approaches do not capture the full semantic information in an input image. This hypothesis is then proved by theory and validated in the experimental parts, which is the main theoretical contribution of the paper. I do like this novel perspective.

Weakness:
I do not have major concerns regarding the technical details, however, the visualization analysis of the proposed method is lacking, for example, under what circumstance that the proposed model can significant improves the performances and when does it fail.
questions: As noted in the weakness part, I'd like to see some visualized validations of the hypothesis to verify the effectiveness and limitations of the proposed method.
limitations: As noted by authors, the main limitation is considered as the local views obtained by the proposed method is cropped from the global view, which may not be the best representations of local parts.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a method based on relationships between views of an object to perform fine grained visual categorization. The authors hypothesise that not only representing local parts but relating them is pivotal to achieve good performance. The authors then experiment using their proposed approach on different FGVC datasets, improving on current methods.
strengths_and_weaknesses: Strengths:
1. The point the paper makes, that the relationship between object parts is important for learning the classification label makes intuitive sense and seems to be important in their experiments.

2. The paper is reasonably clear as to the motivations and experiments run by the authors.

Weaknesses
1. The authors, in my opinion, over complicate the mathematical explanation, spending two pages explaining why modelling the relationships between object parts is important. I do not think this adds much to the paper and the space would be better spent giving a high level, clear explanation of the intuition.

2. I am not sure why their AST (how they combine local parts) is so different from an attention layer (e.g. in TransFG). It seems to be performing the same operation, so I do not understand how this is fundamentally different than the TransFG architecture. Is the main difference the global embedding of the object that the local crops can be compared against and using all three embeddings -- the z_g, z_L, r -- when computing distances and the final metric?

3. In general the paper is clear: their main contribution is using the global, local and r information in the metric learning and so the insight is that using all 3 sources of information is the most useful. However, in the experiments there seems to be limited improvement from these properties over the base encoder (Table 2). So I wonder how useful these things really are. Moreover, the improvements are small over standard methods, and there is no standard deviation to explain if these results are significant. I further wonder if the authors carefully made sure that their setup was similar to the underlying setup and that the improvement was actually due to their method or better data augmentation or underlying architectures.
questions: Stated above.
limitations: The authors discuss limitations.
review_title: null

Meta-review: metareview: This paper proposes a novel approach for fine-grained image recognition, which utilizes the relational information between the global and local views of an object.  It is a reasonable and important finding that not only representing local parts but relating them are critical to establishing superior performance.  The authors validate their proposal’s effectiveness with both theoretical explanations and positive empirical results on various benchmarks.  The authors also did a great job in rebuttal. They provide more clarifications, extra experiments on large datasets, and newly included error bars.    Most of the reviewers are satisfied with the rebuttals and discussions,  and all reviewers have a consistent recommendation.  We think this paper can bring new insights to the visual recognition community and help people understand how the key features and their relations work.   Please also include the newly added experiments and clarifications in the new revision. 

confidence: Certain
Decision: Accept