Relational Proxies: Emergent Relationships as Fine-Grained Discriminators
Authors: Abhra Chaudhuri
Massimiliano Mancini
Zeynep Akata
Anjan Dutta
Venue: NeurIPS 2022 Accept (2022)
Keywords: Fine-grained visual categorization
Relational learning
Information theory

Abstract: Fine-grained categories that largely share the same set of parts cannot be discriminated based on part information alone, as they mostly differ in the way the local parts relate to the overall global structure of the object. We propose Relational Proxies, a novel approach that leverages the relational information between the global and local views of an object for encoding its semantic label. Starting with a rigorous formalization of the notion of distinguishability between fine-grained categories, we prove the necessary and sufficient conditions that a model must satisfy in order to learn the underlying decision boundaries in the fine-grained setting. We design Relational Proxies based on our theoretical findings and evaluate it on seven challenging fine-grained benchmark datasets and achieve state-of-the-art results on all of them, surpassing the performance of all existing works with a margin exceeding 4% in some cases. We also experimentally validate our theory on fine-grained distinguishability and obtain consistent results across multiple benchmarks. Implementation is available at https://github.com/abhrac/relational-proxies.


Relational Proxies: Emergent Relationships as Fine-Grained Discriminators
Abhra Chaudhuri'
Massimiliano Mancini²
Zeynep Akata
Anjan Dutta
Uniiversity of Exeter University of Tübingen
MPI for Informatics
4 MPI for Intelligent Systems
$ University of Surrey
Abstract
Fine-grainee categories that largely share the same set of parts cannot be discrimi- nated based on part information alone, as they mostly differ in the way the local parts relate to the overall global structure of the object. We propose Relational Prox- ies, a novel approach that leverages the relational information between the global and local views of an object for encoding its semantic label. Starting with rigorous formalization of the notion of distinguishability between fine-graing men-igri categories, we prove the necessary and sufficient conditions that model must satisfy in order to learn the underlying decision boundaries in the fine-grained setting. We design Relational Proxies based our theoretical findings and evaluate it lenging fine-grained benchmark datasets and achieve state-of-the-ee results on all of them, surpassing the performance of all existing works with margin exceeding 4% in some cases. We also experimentally validate our theory on fine- fine-grained dis- tinguishability and obtain consistent results across multiple benchmarks, Implemen- tation is available at htthhts://// : htt///ggtttbbbtt thub. com
chal-
-ww.aiiiiiaaa-ooiiiisseee proxies
Introduction
Fine-grained visual categorization (FGVC) primarily requires identifying category-speciffe discrimi- native local attributes [49 44 21). However, the relationship of the attributes with the global view of the object is also known to encode semantic information (..5]. Such relationship can be thought of the way in which local attributes combine to form the overall object. When two categories share a large number of local attributes, this cross-view view relational information becomes the only discriminator. To illustrate this in an intuitive example, Figure shows two fine- Ene-grained categories of birds, the White-faced Plover (left and top-right) and the Kentish Plover (bottom-right). Along with color and texture information, the two categories share a large number of local features like beak, head, body, tail and wings. Given such constraints of largely overlapping attribute sets, relational information like the distance between the head and the body, or the angular orientation of the legs with respect to the body remain the only available discriminators. We thus conjecture that the way the global structure (view) of the object arises out of its local parts (views) must be an emergent 30 property of the object which is implicitly encoded the cross- view relationship. However, all existing methods that consider both global and local information, do so in a relation-agnosie manner, i.e, without considering cross-view relationships (we formalize relation- aggnostciinn in Section 33.
hypothesize that when two categories largely share the same set of local attributes and differ only in the way the attributes combine to generate the global view of the object, relation-agnosti approaches do not capture the full semantic information in an input image. To prove our hypothesis, we develop a rigorous formalization of the notion of distinguishability in the fine-grained setting. Via our theoretical framework, we identify the necessary and sufficient conditions that learner must satisfy to completely learn distribution of fine- iineggraine categories. Specifically, we prove
A. Chaudhuri is with the Department of Computer Science at the University of Exeter. M. Mancini and Akata are with the Cluster of Excellence Machine Learning at the University of Tübingen. A. Dutta is with the Institute for People-Centred AI the University of Surrey. 36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

that a learner must harness both view- new-specifiic (relation (relation-agnostic) and cross-view (relation- -aware) information in an imput image. We also prove that is not possible to design a single encoder that can achieve both of these objectives simultaneously. Based on our theoretical findings, we design learner that separately computes metric space embeddings for the -agnostii and relation-awan components in an input image, through class representative vectors that we call Relational Proxies.
To summarize, we: (1) provide a theoretically rigorous formulation of the FGVC task and formally prove the necessary and sufficient conditions a learner must satisfy for FGVC, (2) introduce a plug-and-play extension on top of conventional CNNs that helps leverage relationships between global and local views of an object in the representation space for obtaining a complete encoding of the fine-gre ane-grained semantic information in an input image, (3) achieve state-of-the-ar results on all benchmark FGVC datasets with significant accuracy gains.
2 Related Work
Fine-grained visual categorization Prior works have demonstrated the importance of learning localized image features for FGVC [T 50 23), with extensions exploiting the relationship between multiple images and between network layers 225.. The high intra-class and low inter-class variations in FGVC datasets can be tackled by designing appropriate inductive biases like normalized object poses 4 or via more data-driven methods like deep metric learning 77. Analysing part-specific features along with the global context was demonstrated through part detection based on activation
regions in CÑN feature maps [] 48 or via context-awarr attention pooling 3. CNNs can also be
modified in novel ways for FGVC by incorporating boosting [27]. kernel pooling g or by randomly
masking out group of correlated channels during training 10 Vision Transformers [40], , with
their ability to attend to specific informative image patches, have also shown great promise in FGVC
[3] 24 To the best of our knowledge, we are the first to provide a rigorous theoretical foundation for FGVC and design cross- view relational metric learning formulation based on the same.
Relation modelling in deep learning Modelling relationships between entities has proven to be useful approach in many areas of deep learning including deep reinforcement learning [7], object detection [1S], question answering [35. graph representation learning 22. few-shot learning 37 and knowledge distillation 31. The usefulness of modelling relationships between different views of the same image has been demonstrated in the self-supervised context by 333 All the above works either leverage or aim to learn relationships between entities, the nature of which is assumed to be known apriori. Our work breaks free from such assumptions by modelling cross-view relationships as learnable representations that optimize the end-task of FGVC.
Proxy-based deep metric learning Motivated by the fact that pairwise losses for deep metric learning incur significant computational overhead leading to slow convergence, the idea of using proxies for learning metric spaces was first proposed in (2 and enhanced in 38 Proxies can also be used to emulate properties of pairwise losses by capturing data-to-date relations (instead of just data-to- to--ooxy) leveraging relative hardness of datapoints [18]. by making data representations follow the semantic
hierarchy inherent in real-world classes 45 or by regularizing sample distributions around proxies to follow non-isotropic distribution 34 However, all the above works perform proxy-based
metric learning directly on data representations. In contrast, our approach is designed to learn class proxies that can be used not only to capture isolated, view specific (local/global) information for the underlying class, but also to learn the cross- view relationships such that they form embeddings in metric space.
3 Relational Proxies
Consider image x X with label y Y. Let cx()) and L (11, 12, 1+) (x) be the global and set of local views of an image x respectively, where Cg and et are cropping functions applied to obtain such views. Let be encoder that takes input v E {g) ULL and maps it Rd where d is the representation dimensionality. Specifically, the representations of the global view g and local views L obtained from f are then denoted by
to latent space representation
z, = f(g) and En = [f()) 1 E L}- {zi,, Ziz.... Zia } respectively. Let R : Lg r be random variable that encodes the relationships r between the global (g) and the set of local (L) views.
2

---

AST
2/2
Align
Local Grops
Dicestmintt
rproxy
Global'Coop (q))
Zg
5
Figure 1: We start by encoding the global and local views using relation- encoder f. We then compute the cross-view relational embedding r between the global Zg and the summary of local Zz, representations. The AST, in conjunction with p, form the cross-view relational function Ç. Finally, the learning of our Relational Proxies is conditioned by both view-specific (z) and Zg) and view relational (r) information. Minimizing Liproxy helps to align representations from the same category, while discriminating across different categories in a metric space.
3.1 Problem Definition
We leverage the qualitative consistency in the definition of the fine-grained visual categorization to formalize the same in more quantitative
(FGVC) problem in the relevant literature [25 48 13, 3 terms as follows.
Definition 1 (k-distinguishability), Two categories C1 and C2 said to be k-distinguishable iff' along with the global view, a classifier needs at least local features to tell them apart, ie., the true hypothesis can only distinguish between Ci and C2 if has access the complete set {2g} U Zt, and it fails to distinguish between C₁ and C2. if it only has access to {Zg) UZz Zi, Vzi E 21.
The notion of k-distinguishabbiity formalizes what it means for two categories only be distinguish- able in the fine- lne-grained but not in the coarse-grained setting. Given the concept of k-distinguishabiiity. the definition of FGVC problem directly follows from here:
Definition (Fine-Grained Visual Categorization Problem Prave). A categorization problem is said to belong the Prove family, iff there exists at least one pair of categories C₁ and C2 such that they are k-distinguishuble
Unless otherwise stated, all datapoints (x, y) are considered to be sampled from d-iistinguishable categories of an instance of Prave. In the subsequent sections, we prove that for learner to completely model the class distribution for an instance of PrGvC. it must, alongside the view specific representations Zy and Zr. also learn a function & that models the cross-view relationship between global and the local views. Thus, a function ç, to model R, must satisfy the following properties: (1) View- Unification: Maps the set of all views {g. 1 L} of image to single output (2) Permutation Invariance: Produces the same output irrespective of the order of the local attributes, i.e., E(zy, izt, , ... Zi }) {ZZg (Zly x ... Zo }). for every permutation æ, where 2, and zı, are the representations of the global and the local views respectively, obtained from f. We provide more details on the necessity of these properties in Section the Appendix.
3.2 Relation-Agnostic Representations and Information Gap
In this section, we formally study the nature the representation spaces learned by models that do not consider the cross-view relational information in the context of Prave. We term such representations as being velation-agnostie and prove via Proposition| I that they suffer from an Information Gap, and thus are unable capture the complete label information encoded in input image.
Definition (Relation-Aguostic Representations Information Theoretic). An encoder is said to produce relation- aggiosic representations if it independently encodes the global view g and local views 1 E IL of x without considering their relationship information r.
Lemma Given a relation- aggostic representation z. of x, the conditional mutual information between and y given be reduced to I(x; r/z).
3

---

Proof. Given relation-agge -agnosiic representation z of x. the only uncertainty that remains about the label information y can be quantified as the cross-view relational information r, i.e. (x; y T(xr)). The proof of this statement is given in Identity 1 of the Appendix.
Intuitively, the conditional mutual information between x and y given z, i.e,, T(x; y/2) represents the information for predicting from x that z is unable to capture. Since is relation-agnostic. the after z is the cross-set relationship between the global and the local views, i.e., r. Therefore, we can write y z = rx; Using this equality and further factorizing (x; r) using the chain rule for mutual information, we get:
only uncertainty that remains
yl) = [x: r) = [xx; r|z) + ((r:z) = r()),
the latter equality following from Definition 3 which implies that [(; 0, since does not explicitly model the -ocallto-globll relationships r.
Lemma 2.
The mutual information between and its relation relation-agnostic representation z does not change with the knowledge of r.
Proof. Following the chain rule 12 the mutual information between x and z, i.e. [xx z) can be expressed as [xx; z/r) + T(z; r). However, since z is relation-agnostic (Definition 3). I(z; r) 0. Thus, [(x; z) [(x; z/r) [(z; r) z(x:
Proposition 1. For relation-agnostee representation z of the label information encoded in is strictly upper-bourded by the label information in x, i.e., [xx;)) > T(z:y) by an amount [xx; rlz).
Proof. The mutual information yx) between datapoint x and its ground- label y can be
expressed as (x:y|z) + +[(x;) based on the chain rule. Here [xx: y zz) represents the information for predicting y from that z is unable to capture, while [xx; z) denotes the predictive information that z does capture from x. We can thus rewrite (xx; y) using Lemmal | and Lemma as:
I yx: I ylz) + [(x: z) (x: 1)) + (xx
(1)
Now, using the chain rule of mutual information. [[z; y) yzz (z: However, consequence of the data processing inequality (12). [(z y x) (since cannot encode any more information about y than x). Applying this and Lemma 2 Equation 1
y(x;y) [xx; r|z) + [x; z|r) = I(x;r/z) + [xx; = (x;r + I(z;y)
Therefore, I(x:y) y). by amount [ rl)
Intuition:
By establishing a strict upper-bound, Proposition I shows that pelation-agnossee encoders cannot fully capture the label information in an input image. The quantity they are unable to capture given by [xxr|z)] which we call the Information Gap.
3.3 Sufficient Learner
Proposition states that the information gap exists if the representation space happens to be relation- agnostic. We now explore if there is really the need to learn relation-agnostic representations in the first place. From there, we identify the necessary and sufficient conditions for a complete learning of yx; and derive the requirements for learner to do the same.
Definition 4 (Relation-Agnostic Representations Geometric). Let ne(:) represent the e- neighbourhood around a point in the limit e + 0 A representation space is relation-agnostic if and only if Viz E Z,, ne (Z:) O ne
An intuitive explanation of Definition can be found in Section 3 of the Appendix.
Axiom 1. f learns representations z such that a classifier operating the domain of learns a distribution y, minimizing its cross-entropy with the true distribution E. y; log(ý:), where denotes the i-th class.
Lemma 3. For an instance of PFavv, the representation space learned by I is relation-agnosic i.e., the global view g and the set of local views II are mapped to disjoint locations in the representation space.
"Th choice of determines the degree of relation-agnosticig of the representation space.

---

Proof. From Definition 4 representation space is not relation-agnostic agnostic iif 3zz E ΣL (n(Zii) n
n. nz(zz) # o. Under this condition, the classifier only has the information from {z,} L Zr. \Zi instead
of the required (Z.g) U Z Thus, for instances of Prove, according to Definition [ removing the
relation-agnostic nature from the representation space of f would cause downstream classifier to
produce misclassifications across the instances of k-distinguishable categories, leading to violation of Axiom Hence, f can only learn relation-agnoste representations.
We can thus conclude from Lemma 3 and Proposition I that the necessary and sufficient conditions for a learner to capture the complete label information I (x;y). are to consider both (1) the relation- agnostic information z and (2) the cross-view relational information r.
Proposition 2. An encoder trained to learn ielation-annoster representations of datapoints cannot be used to model the relationship r between the global and local views of x.
Proof. f:x; + Zu is a unary function that takes as input a (global or local) view X. of an image x and produces view-specific (Lemma) 3 representations z, for a downstream function g Z, y.
For f model the cross-view relationships, must output the same vector irrespective of whether X. g or = 1 E L., ie. whether x, is a global or local view of the input image x (view- unification property of (). However, Lemma prevents this from happening by requiring the output space of f to be relation-agnostic Hence, f cannot be used to model
Thus, to bridge the information gap, a learner must have distinct sub-models that individually satisfy the properties of being relation-agnostic and relation-aware. Only such a learner could qualify as being sufficient for an instance of Prove.
Intuition: In this section, have effectively proven that the properties of relation-agnosticii and awareness are dual to each other. We show that while relation- aagiiiiiiinn is not sufficient, it is necessary condition for encoding the complete label information y(x:y). We also show that a disjoint encoder cannot be used to model the two properties alone without violating one of the necessary criteria. The requirement of a separate, relation-aware sub-model follows from here.
3.4 Learning Relation-Agnostic and Relation-Aware Representations
Figure [ depicts the end-to-end design of our framework. Derived from our theoretical findings, it comprises of both the relation-agnostic agnostic encoder f. and the cross-view relational function, E. expressed a composition of the Attribute Summarization Transformer, AST, and network for view-unification, p. Below, we elaborate on each of these components.
Relation-Agnostic Representations: We follow recent literature 43 48 for localizing the object of interest in the input image x and obtaining the global view g by thresholding the final layer activations of CNN encoder f and detecting the largest connected component in the thresholded feature map. We obtain the set of local views 12 11l..,,,,,,, 1 as sub-crops of g (more details in Section Following the primary requirement of Proposition [ we produce relation-agnostic representations by propagating and I through a CNN encoder f that independently encodes the two view families n fg) and zı, flk).
Relational Embeddings: The second requirement, according to Proposition I for completely learning yx) is to minimize 1(x;r/z) i.e., the uncertainty about the relational information encoded in x, given a relation-agnostic representation z. However, according to Proposition 2 cannot perform the same using the relation- -elation-annsttic agnostic encoder f. Contrary to existing relational learning literature [3 33 that assumes the nature of relationships to be known beforchand, we take novel approach that models cross-view view relationships as learnable representations of the input x. We follow the definition of the relationship modelling function { [g. L) + r, that takes as input relation- agnostic representations of the global view zg and the set of local views Z, [21, 2, ... 21, and outputs a relationship vector r, satisfying the View- Unificationn and Permutation Invariance properties.
We satisfy the Permutation Invariance property by aggregating the local representations via novel Attribute Summarization Transformer (AST). We form a matrix whose columns constitute a learnable summary embedding zu followed by the local representations Zi, Z [ZL, 21, Zi, ... Z4, We compute the self-attention output z', for each column Z. in Zj as z'_ = a Z W, where a = (Z. Wg) (Z WT /VD). and D is the embedding dimension. By iteratively performing
5

---

self-attention operations among the columns of z AST aggregates information across all the local attributes into the final learnable output of z Unlike the usual vision transformer 440] we omit the
usage of positional embeddings, as doing so provides better permutation invariance 22
For satisfying the View-Unification property, we introduce a simple feed-forward multilayer percep- tron that learns the mapping p 2i, + r. It takes as input the representation of the global view Zg and the summary of the set of local views zt, and outputs the relationship as learned vector r. Thus, in our construction, the AST along with p. constitute the relation modelling function &.
The representations zy: z and in unison encode the full semantic information y in x (Proposition I To alleviate the low inter-class variance in Prove. metric learning has been shown to be an effective 77 approach. Furthermore, approaches like [28] and [8] for metric learning have shown that substituting pairwise comparisons with assignment to a fixed set of learnable class proxies reduces the training-time complexity from large polynomial like ((n²) or (nn)) to near linear O (c.n). where e is the number of classes in a dataset, n. is the number of train-set datapoints, and < n. For this purpose, we contrast instance representations across classes through class proxy vectors that are informed by both the view specific and relational representations via learning the conditional distribution p(y (2g. ži, r). We term such class proxies, Relational Proxies, as they leverage cross-view relationship information for encoding class semantics.
Learning Relational Proxies:
Consider set of learnable class proxy vectors IP grained classes. Here, we present a novel formulation of the proxy-anchor loss [18] in cross-entropic terms that allows us conform to the requirement of Axiom in the fine-graing setting. Specifically, for each of the representations w E {Zg, 21, r} for all x E X, we minimize the following:
(P1. P2. pe). where e is the number of fine-
14
C Ceproxy
Σ log DEP
WED
16+ ./
1 +
ca(s(wp)+6)
EEN
where Q is the set of representations in a mini-batch for which p is the true class proxy, 5i is one for which p is not the true class proxy, and s(-, computes the cosine distance. ai+ helps align matching (w: p) pairs close together in the representation space (since Leproyy follows a cross-entropic form, does not violate the relation-agnosticity of f. as proven in Lemma 3 with a more detailed note in Section of the appendix), while v/- helps embedding non-matching (w, p) pairs farther apart. The scaling parameter along with the margin parameter 8 control the intensity with which the alignment and discrimination are performed. 1/( & gives a probability indicating how closely the learned representation space reflects the semantic structure in X. Liprexy thus computes the cross-entropy loss between the ground-truth and the predicted class distributions over the set of proxies.
Given an input image x, we compute its global (zy). summary of local (zz), and relational (r) representations using f. AST and p as explained above. We then predict the class probability distribution y of these representations by computing their soft-assignment scores across the relational proxies. The assignment score for each proxy p E
Inference:
P is computed as follows:
(w.p))
Σ
SP
Σ ('wppp
WE (g...
D'EP
The class corresponding to the relational proxy with the highest assigned score is returned as the prediction.
4 Experiments
We now present the implementation details of Relational Proxy, and the results obtained upon evaluating on benchmark FGVC datasets. We also discuss observations from ablation studies that we performed to validate our theoretical foundations, as well as the implementation specific choices that we made, along with qualitative visualizations of the learned cross-view local relationships.
4.1 Experimental Settings and Datasets
Implementation details- We implement our Relational Proxy model using the PyTorch (2) deep learning framework, on an Ubuntu 20.04 workstation with single NVIDIA GeForce RTX 3090 GPU,

---

Benchmark
Culivar
Method
FGVC Alteralt
Stanford Cars
CUB
NA Birds
INaturalist
Cofton
Soy
MaxEar
NeuriPS 18
89,76
93.85
86.34
DBTNet [5] NeurlPS119
91.60
94,50
88.10
StachNorm 19 Ne
81.79
87.57
79.71
74.94
60.75
45.41
38.50
MMAL 48 MMM'21
94,70
95.00
89.60
87.10
69.85
65.00
47.00
FTVT
E2 BMVC
79.80
91.25
91.65
89.42
70.30
57.92
44.17
AAAT21
94.90
95.70
91.00
Trand
[1 AAAT222
80.59
94.80
91.70
90.80
71.70
45.84
38.67
Ouurs (Relational Proxy)
95.25 + 0.02
96.30 0.04
92.00 + 0.01
91.20 = 0.00
72.15 + 0.03
69.81 + 0.04
51.20 + 0.02
Table I: Comparison of classification accuracies obtained by our method (averaged over 5 independent
runs) on standard FGVC datasets with current state-of-the-ar approaches.
an 8-core Intel Xeon processor and 32 GBs of RAM. Since we proposed a proxy-based approach for learning the relational metrie space, we do not have dependency on batch-size for the purpose of negative sampling as part of our metric learning phase, which enables us to train our entire model end-to-end a single GPU. Also by the virtue of using class-proxies, the convergence time is reduced by a significant amount compared to pairwise losses.
Hyperparameter settings For initial training stability, we consider five disjoint locations (four corners and the centre) of g to be the set of local views. As training progresses, we also allow the model to learn from an increased number views obtained via random cropping. In the same way at inference time, the local views constitute combination of the five disjoint crops along with some random crops. We found that the optimal number of local views k to be equal to 7 for FGVC Aircraft, Stanford Cars and both the cultivar datasets. For CUB and NABirds, A = 8 gave the best performance. We use ResNet50 [1] pretrained f. In Sec. 1.3 of the supplementary. we also provide evaluations using VGG-16 (36] to show that the performance gains achieved by our model do not depend the specific backbone. We train our full Relational Proxy model end-to-end for 200 epochs using the stochastic gradient descent optimizer with initial learning rate of 0.001 (decayed by factor of 0.1 every 50 epochs), momentum of 0.9, and a weight decay of 10-1
ImageNet [9] as the backbone of our relation-agnostic encoder
Datasets and Evaluation We evaluate our model on the four most common fine-grained visual categorization benchmarks (number of classes and train/test splits respectively in brackets): FGVC Aircraft (26) (100 6667/3333), Stanford Cars (2) (196 I 8144/8041), CUB [41] (200 I 5994/5794), and NA Birds [39] (555 23,929/24,633). For large scale benchmark evaluation, we choose the iNaturalist 2017 dataset which consists of 13 super-categories that have been split into total of 5089 fine-grained categories with 675,170 training and 182,707 test images. We also perform experiments on two challenging datasets of the cultivar domain that offer very low inter-class variations, namely Cotton Cultivar [46] (801 240/240) and Soy Cultivar [46] (200 600/600). We use classification accuracy as our metric for evaluating the performance of a model.
4.2 Comparison with State of the Art
Benchmark Datasets- In Table T we report the performance of our method on benchmark datasets along with existing SotA approaches. StochNorm [19] presents a novel way to refactor batch nor- malization that helps prevent overfitting for the task of FGVC. The novel training routine proposed in MaxEnt [I]] improves FGVC performance by maximizing the entropy of the output probability distribution of CNN. By designing a computationally inexpensive bilinear feature transformation mechanism for CNNs. DBT [31] achieves competitive performance on benchmark FGVC datasets. MMAL 48 is one of the most competitive models for FGVC Aireraft and Stanford Cars, which extracts the most informative global and local views by analyzing the activation maps of the final layer of a CNN, and embeds them a relation-agnostic representation space. TransFG [3 proposes a vision transformer based technique for extracting informative local patches, achieving SotA per- formance on iNaturalist, and promising results on CUB and NA Birds. By learning a context aware attention pooling mechanism, CAP 3 reports SotA performance than iNaturalist. From Table we see that our method surpasses the SotA on all four benchmarks by significant margins. Specifically, we beat the SotA on Stanford Cars by 0.60%, iNaturalist by 0.45% on FGVC Aircraft by 0.35%, and on both CUB and NA Birds by 0.20%.
all benchmark datasets other
Cars and Aircrafts can largely vary in color, texture and custom, part-specific styles within a category. However, the geometry of the overall object (represented by cross-vier relationships) within a class remains fairly constant. This leaves room for a large amount of relational information to be captured. This also holds true for the iNaturalist dataset, as the local-to-ghe -o--ooolllll emergent relationships can be used to discriminate between both coarse-grained (super) and fine-grained (sub) categories. For the
7

---

Relation-Agnostic
Learnable Relation
Stanford
ID
AST
RelationNet
Proxies
Aircraft
CUB
Encoder
Cars
1.
94.60
91.25
95.21
2.
94,91
91.50
95.62
3.
95.13
91.90
96.15
94,92
91.55
95.70
95.10
91.81
96.05
6.
95.05
91.73
95,93
7.
95.25
92.00
96.30
Table 2: Results of ablating the key components of our Relational Proxy model (sufficient learner). Grouped as better illustrate the effect of learning cross-view relationships. Note that the meaning of non-exxisiie non-existence of a component may vary according to context / other row elements. Refer to the corresponding paragraph Section 4.3 of the main text for further details.
Attribute
Global Representation (2,))
Relational Representation (r)
ID
FGVC Aircraft
CUB
Stanford Cars
Summary (zz.)
94.91
91.50
95.62
2.
94.85
91.58
95.75
3.
94.60
91.47
95.51
4.
95.25
92.00
96.30
Table 3: Results of ablating the Proxy Conditioning Representations. Note that only the output representation vectors were ablated here, and not the entire model component producing them. The latter has been studied independently with findings reported in Table 2
bird datasets (CUB, NABirds), although this relational information is still there, most categories can be told apart by color, texture and local-atrribute specific information, if they are clearly visible. For this reason, the accuracy gains obtained the Cars and Aircraft datasets surpass those obtained for the birds.
Cultivar Datasets For the highly challenging datasets of the cultivar domain, i.e. Cotton and Soy Cultivar, FFVT 42) provides state-of-the results by using a specialized feature fusion technique for vision transformers. As can be seen in Table our model, by leveraging cross-view relational embeddings, manages to provide a performance boost exceeding 4% over the current SOTA on the cultivar datasets. Cultivar datasets have very low inter-class differences. Cross-view relational information like edge curvature, relative angles between leaf sub-parts, width to height ratio, convergence patterns of leaf ends, etc., largely determine the uniqueness of category. For this reason, our method is extremely effective when applied to such domains.
4.3 Ablation Studies
We perform the following three classes of ablation studies:
Key components of the sufficient learner Table 2 shows the results of ablating the key components of our model. The relation agnostic encoder being the most fundamental component, cannot be removed, and therefore appears in all the rows. Row thus represents training a simple classification head on top of the representations obtained from the celation-agnostee encoder. Row 2 denotes the result of aggregating the local views, computing predefined relationship function, specifically the distance between the local and global representations, and minimizing Huber loss between the relational distance value between instances of the same class. Row 3 introduces idea of learnable relational vectors (instead of predefined functions like distances). Since cross-view relationships are unique to a class, we aim to embed the relational vectors in a metric space by minimizing a pairwise contrastive loss across classes. However, as noted in recent metric learning literature (28 18). computing pairwise losses can be expensive and lead to slower convergence. Motivated by this, we introduce the idea of relational proxies in Rows 4 7. Row 4 replaces the AST and RelationNet by simple concatenation of the inputs and propagation through a linear layer. Row and individually show the effects of replacing the AST and RelationNet with linear layers. Finally, Row 7 denotes the performance of our model with all components included.
Rows 2 and 3 demonstrate the importance of modelling cross-view relationships specifically learnable metric space embedding. Rows 5, and show the contribution of our AST in summarizing the local attributes, as well as the fact that the cross-view relationship is non-linear in nature.

---

Method
T-ImageNet
D-ImageNet
30
Zig Z1.
88.75
91.30
7g. Z1,, 88.91
92.75
Classffictioo Accuracy
A
0.16
1.45
Table 4: Comparison of accuracy gains ob-
tained by using the relational information r along with the view-specific representa- tions Zy and z (second row) over using only relation-agnes row) the boarse-grained Tiny (T) and fine- grained Dogs (D) subsets of ImageNet. As be observed, the performance gain A by accounting for the relational information is significantly more in the fine-grained setting than in the coarse-grained one.
30
representations (first
Allond
CUB
Strfied Car
[0]|
Figure 2: Effect of varying the number of local views [[]
Conditioning of the relational proxies - The relational proxies in our model are conditioned by three representations of the input x the summary of the local attributes zz, the representation of the global view zg. and the relational vector r. We study the contribution of each of these representations and summarize our findings in Table These results demonstrate that the information encoded in all three representations are necessary for learning the complete set of class attributes.
Results ImageNet subsets In order to validate whether findings are in fact particularly applicable to the fine-gree grained setting, we perform experiments to compare the performance boost provided by our method over vanilla relation-agnostic encoder, between coarse-grained (Tiny ImageNet (2)) and fine-grained (Dogs ImageNet/ / Stanford Dogs (17)) subsets of ImageNet. Our findings are summarized in Table which shows that our method does in fact provide more significant improvement over a relation-agnostic encoder in the fine-grained setting.
The optimal value of k for the s-distinguishaabliit criterion Figure 2 empirically illustrates the idea of k-distinguishabiiity for a given local-crop size on the FGVC Aircraft, Stanford Cars and CUB datasets. For an instance of 'Prvvc. the performance of a model is strongly dependent on the number of local views it has access to. When the number of local views [L] is less than the minimum required number k the classification performance is poor the model does not have access to the minimum set of required fine-grained information. As [L] approaches k, the performance increases, reaching its maximum at l: 7 for FGVC Aircraft and Stanford Cars, and 8 for CUB). However, if [L] is increased beyond k, there is no further gain in performance, as the extra information is either redundant or semantically irrelevant.
Correlation between LL] and local patch size - To determine the right computational trade-offs for method, we perform a study to identify possible correlations between the number of local views and size of local patches. We trained our model on FGVC Aircraft (26) by varying the number of local views [L_] and the size of each local patch identify their correlations. We present our results in T where rows represent the number of local views [L] and the columns represent the side-length of each local patch. So, if the global view has spatial dimensions N N, each local patch would be of N/L x N/1, where is the scaling factor that is varied across the columns. In summary, the rows represent increasing the number of local views top-down, and the columns represent increasing the patch-size left-to-right. The numbers are expressed as relative deviations from a reference of 95.25% i.c., the setting corresponding our reported accuracy for FGVC Aircraft in Table !
From Table []] we can see that increasing the patch size beyond certain point has detrimental effect as the lo- cal views tend to lose their granularity and degenerate into global views. Increasing the number of crops has stronger improvement effect on performance if the patch size small, thus influencing the value of k accordingly. However, decreasing the patch size at the cost of an in- creased number of local views also has its downsides the number of attention computations in the attribute summa- rization step increases guadratically. Thus [LL| and the local patch size needs to be determined based on application specific accuracy requirements and the available computational resources.
[1]
N/5
N/4
N/3
N/2
7
-0.03
-0.02
0.00
-0.14
12
+0.01
+0.02
0.00
-0.11
15
+0.05
+0.03
+0.01
-0.11
18
+0.05
+0.03
+0.00
-0.10
Table 5: Correlation between number of local views and size of local patches

---

4.4 Visual Representations of Cross-Viee Local Relationships
Our AST-based aggregation scheme allows to visualize the local relationships that lead to the emergence of the global-view. We aim to construct graph of local views for de- picting the cross-view local relationships. The graph represents the manner in which the lo- cal views combine to form the overall object. The nodes of the graph represent the local views. Two nodes are connected via edge if there exists relationship between them. The thickness of the edges in the illustration is proportional to the degree of relatedness. We compute the topology of this graph by analyzing the final layer mu- tual attention values of the Attribute Summarization Transformer (AST). We add edge between two local views if their mutual attention score higher than a threshold (which choose to be the average of all pair- wise attention scores). The weight of the edge is proportional to the mag- nitude of attention. For the purpose of simplicity, we depict fewer local views the visualization, than are ac- tually used for computation. Figure 3 shows example graphs on images from the Stanford Cars dataset. In Appendix Section 5, we provide more such qualitative results and based on these graphs, we provide an analysis of scenarios under which even relational information cannot distinguish between certain fine-graines categories.
Lamborghini Aventador
Lamborghim Reventón
Figure 3: Visualization of the learned relationships across loeal-views, Despite close similarities between the two car categories, it can be seen that model leverages discrimi- native cross-view relationships to tell the instances apart.
Conclusion and Discussion
Starting with the idea of k-distinguishabiliyy. we derived the necessary and sufficient conditions that model must satisfy in order to completely capture the fine gree-grriddd information in an image. We proved that a model needs to simultancously encode both view-specific and cross-view relational properties of an object in order to bridge the information gap that its representations have with the semantic content in the input image. Based on our theoretical findings, we designed Relational Proxies, a method that achieves state-of-the-rr results on benchmark FGVC datasets by learning class representations conditioned with cross-view relationships. By introducing theoretically rigorous framework, we believe that our work opens up new avenues for studying the problem of FGVC in more systematic manner. One immediate potential outcome of our work that we foresee is the development of explainable fine- ine-grained features. Such features can be used for computing a minimal set of fine-gre fine-grained attributes to limit compute time/resourcees, to perform tasks like cross-modal retrieval in domains with large modality gap.
The process of obtaining local views in our method is somewhat of an uninformed, generic cropping methodology on the global view of the object, which may not necessarily always yield the best set of local object parts. More informed ways of detecting novel object parts from which the global view emerges can lead to obtaining at par performance but with fewer local views.
Limitations
Societal Impacts - The rigorous theoretical basis of our work has a positive societal impact, which only makes our methodology transparent and easy analyze, but also provides framework to study the foundations of FGVC in general. So far, we are not aware of any negative societal impact that is specific our methodology. However, as with all data-driven approaches, underlying biases in the datasets on which our model is trained would influence the patterns learned by it.
A Acknowledgements
This work has been partially supported by the ERC 853489-DEXIM, by the DFG-EXC number 2064/1-Project number 390727645, and as part of the Excellence Strategy of the German Federal and State Governments.
10

---

References
[1] Anelia Angelova and Shenghuo Zhu, Efficientt object detection and segmentation for fine-grained recognition. In CVPR, 2013.
[2] Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Zonzzlzz Vinicius Zam- baldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre, Francis Song, Andrew Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matt Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. arXiv, 6 2018.
[3] Ardhendu Behera, Zachary Wharton, and Asish Bera. Context-aware Attentional Pooling (CAP) for Fine-grained Visual Classification. In AAAI, 2021.
[4] Steve Branson, Grant Van Horn, Serge Belongie, and Pietro Perona. Bird species categorization using pose normalized deep convolutional nets. In BMVC, 2014.
[5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self-Supervised Vision Transformers. In NeurIPS, 2021.
[6] Subhabrata Choudhury, Iro Laina, Christian Rupprecht, and Andrea Vedaldi. Unsupervised Part Discovery from Contrastive Reconstruction. In NeurlPS, 2021.
[7] Yin Cui, Feng Zhou, Yuanqing Lin, and Serge Belongie. Fine-grained categorization and dataset bootstrapping using deep metric learning with humans in the loop. In CVPR, 2016.
[8] Yin Cui, Feng Zhou, Jiang Wang, Xiao Liu, Yuanqing Lin, and Serge Belongie. Kerel pooling for convolutional neural networks. In CVPR, 2017.
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, 2009.
[10] Yifeng Ding, Shuwei Dong, Yujun Tong. Zhanyu Ma, Bo Xiao, and Haibin Ling, Channel DropBlock: An Improved Regularization Method for Fine-Grained Visual Classification. In
BMVC, 2021.
[11]
Abhimanyu Dubey, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Maximum-entree fine grained classification. In NeurlPS, 2018.
[12] Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning Robust Representations via Multi-Vies Information Bottleneck. In ICLR, 2020.
[13] Ju He, Jie-Neng Chen, Shuai Liu, Adam Kortylewski, Cheng Yang. Yutong Bai, and Changhu Wang. TransFG: A Transformer Architecture for Fine-grained Recognition. In AAAI, 2022. [14] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
[15] Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei. Relation Networks for Object Detection. In CVPR, 2018.
[16] Shaoli Huang, Zhe Xu, Dacheng Tao, and Ya Zhang. Part-stacked enn for fine-grat fine-grained visual categorization. In CVPR, 2016.
[17] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine-grained image categorization. In CVPRW, 2011.
[18] Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. Proxy anchor loss for deep metric learning. In CVPR, 2020.
[19] Zhi Kou, Kaichao You, Mingsheng Long, and Jianmin Wang. Stochastic normalization. In NeurlPS, 2020.
[20] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D Object Representations for Fine- GrainGr Categorization. In 3DRR, 2013.
[21] Michael Lam, Behrooz Mahasseni, and Sinisa Todorovic. Fine-grained recognition as hsnet search for informative image parts. In CVPR, 2017.
[22] Fei-Fei Li, Andrej Karpathy, and Justin Johnson. Tiny image net. hhttps///wwy https://ww.kaggge
com/c/ http:///tinnimmmmee
2017.
[23] Di Lin, Xiaoyong Shen, Cewu Lu, and Jiaya Jia. Deep lac: Deep localization, alignment and classification for fine-grat -rained recognition. In CVPR, 2015.
[24] Di Lu, Jinpeng Wang, Ziyun Zeng. Bin Chen, Shudeng Wu, and Shu-Tao Xia. SwinFGHash:
Fine-grained Image Retrieval via Transformer-basee Hashing Network. In BMVC, 2021.
[25] Wei Luo, Xitong Yang, Xianjie Mo, Yuheng Lu, Larry Davis, Jun Li, Jian Yang, and Ser Nam Lim. Cross-x x learning for fine-grained visual categorization. In ICCV, 2019.
11

---

[26]
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Fine-
Grained Visual Classification of Aircraft. arXiv, 2013.
[27]
Mohammad Moghimi, Mohammad Saberian, Jian Yang, Li Jia Li, Nuno Vasconcelos, and Serge
Belongie. Boosted convolutional neural networks. In BMVC, 2016.
[28] Yair Movshovitz-Attia Alexander Toshev, Thomas K. Leung, Sergey loffe, and Saurabh Singh.
No Fuss Distance Metric Learning Using Proxies. In ICCV, 2017.
[29] Muzammal Naseer, Kanchana Ranasinghe, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Intriguing Properties of Vision Transformers. In NeurlPS, 2021. [30] Timothy O' Connor. Emergent Properties. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Winter 2021 edition, 2021.
[31] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In
CVPR, 2019.
[32] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Veiioo, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in Py PyTorch. In NIPSW, 2017.
[33] Massimiliano Patacchiola and Amos Storkey, Self-supervised relational reasoning for represen-
tation learning. In NeurlPS, 2020.
[34] Karsten Roth, Oriol Vinyals, and Zeynep Akata. Non-isotropy Regularization for Proxy-based
Deep Metric Learning. In CVPR, 2022.
[35] Adam Santoro, David Raposo, David G.T. Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In NeurlPS, 2017.
[36] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale
image recognition. In ICLR, 2015.
[37] Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang, Philip H.S. Torr, and Timothy M. Hospedales.
Learning to Compare: Relation Network for Few-Shot Learning. In CVPR. 2018.
(38) Eu Wern Teh, Terrance De and Graham W. Taylor. ProxyNCA++: Revisiting and Revitalizing Proxy Neighborhood Component Analysis. In ECCV, 2020.
[39] Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie. Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In CVPR, 2015.
[40] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, tukasz. Kaiser, and Illia Polosukhin. An image is worth 16x16 words: transformers for image recognition at scale. In NeurlPS, 2017.
[41] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge J. Belongie. The caltech-uesd birds-200-2011 dataset. In Califormia Institute of Technology, CNS-TR-2010-001, 2011.
[42] Jun Wang, Xiaohan Yu, and Yongsheng Gao. Feature Fusion Vision Transformer for Fine- Grained Visual Categorization. In BMVC, 2021.
[43] Xiu Shen Wei, Jian Hao Luo, Jianxin Wu, and Zhi Hua Zhou. Selective Convolutional Descriptor
Aggregation for CrainGr Image Retrieval. JEEE TIP, 2017.
[44] Xiu Shen Wei, Chen Wei Xie, Jianxin Wu, and Chunhua Shen. Mask-cnn: Localizing parts and selecting descriptors for fine me-ggriined bird species categorization. PR, 2018.
[45] Zhibo Yang. Muhammet Bastan, Xinliang Zhu, Doug Gray, and Dimitris Samaras. Hierarchical Proxy-based Loss for Deep Metric Learning. In WACV, 2022.
[46] Xiaohan Yu, Yang Zhao, Yongsheng Gao, Shengwu Xiong, and Xiaohui Yuan. Patchy image structure classification using multi-orientation region transform. In AAAI, 2020.
[47] Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap. Edward Lockhart, Murray Shanahan, Victoria Langston, Razvan Pascanu, Matthew Botvinick, Oriol Vinyals, and Peter Battaglia. Deep reinforcement learning with relational inductive biases. In ICIR, 2019.
[48] Fan Zhang, Meng Li, Guisheng Zhai, and Yizhao Liu. Multi-branch and Multi-scale Attention Learning for Sine-Grriined Visual Categorization. In MMM, 2021.
[49] Han Zhang. Tao Xu, Mohamed Elhoseiny, Xiaolei Huang, Shaoting Zhang, Ahmed Elgammal, and Dimitris Metaxas. Spda-cnn: Unifying semantic part detection and abstraction for fine- grained recognition. In CVPR, 2016.
[50] Ning Zhang, Jeff Donahue, Ross Girshick, and Trevor Darrell. Part-based r-enns for fine-grained category detection. In ECCV, 2014.
12

---

[51] Heliang Zheng, Jianlong Fu, Zheng-Jun Zha, and Jiebo Luo. Learning deep bilinear transforma- tion for fine-grt -oee-graned image representation. In NeurIPS, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes]
(b) Did you include complete proof' of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as URL.)? [N/A] Code and pre-trained models will be made public upon paper acceptance. Details of all experimental settings required reproduce our results are provided in Section 4.1
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No] But the numbers we report are the means of runs with different random seeds.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals. if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "zXWKCL1bPNp", "reviewer": "Reviewer_EFjT", "summary": "In the fine-grained setting, discriminating between different classes requires learning how different local parts combine to form the object of interest. In this work, the authors introduce a theoretical framework and a novel method that decomposes FGVC tasks into relation-agnostic feature extraction and cross-view relation learning. They show the superiority of such method through a set of experiments.", "questions": "1. In Definition 1, k-distinguishability is defined with respect to two classes only. What does this imply at the level of the entire dataset. What happens if different pairs of classes have different corresponding k values?\n2. In settings where the datasets includes multiple subgroups of classes (cats, dogs, birds) that would be coarsely separable at the group level,  but would require fine-grained visual modeling within each group (in birds: white-faced plover vs. kentish plover). How would this approach change? How is k-distinguishability defined? How is the number of relational proxies c selected?\n", "limitations": "The authors discuss one of the main limitation, being the local view generation. \n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The problem this work address is relevant. While I am unable to gauge the relevance of the proposed theoretical framework and its broad usefulness to the community, the proposed approach is solid and of interest.\n\nStrengths:\n1. **The experimental setup is solid.** The authors test their method on multiple datasets and consistently show competitive results across all of them. These results support the idea of modeling the cross-view relation in their proposed architecture. This is further demonstrated through ablation studies, that highlight the need for a cross-view relational function and provide further insights into the method.\n2. The theoretical framework established in this work is clear, and while it builds on a lot of definitions, the proofs are simple and easy to follow. \n\nWeaknesses:\n1. **The permutation invariance property is counter-intuitive** The authors introduce the permutation invariance property as a necessary property for a model to solve FGVC tasks. In section 3.4, they introduce a novel transformer (AST) that does not use positional embeddings and is thus permutation invariant. While the permutation invariance property can provide certain desired properties like potentially better generalization, the authors do not provide theoretical evidence for why it would be necessary, and it is not fully clear how it is motivated either. Certain claims in the introduction would actually suggest otherwise: “differ only in the way the attributes combine to generate the global view of the object” or “[features like] the distance between the head and the body, or the angular orientation of the legs”. This suggest that features like positional encoding would actually be critical. \n2. The term “relation” is not explicitly defined and it is unclear what the authors mean. The \"relation-agnostic representation” is established in Definition 4, and while it is clear what it means in mathematical terms, its relation to FGVC problems is not evident. Providing more clarifications would make the text easier to follow.\n3. **Decomposing the problem into relation-agnostic encoder and a cross-view relational function.** The authors do not argue for why this decomposition is necessary when solving FGVC problems- at least in the main text. A discussion of this can be found in appendix 6.3, and I would argue for including this in the main text as it better explains the idea of the relational gap and seems to at least provide an initial motivation for this decomposition.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "l4iGnf1QNRo", "reviewer": "Reviewer_G6W7", "summary": "In this paper, authors propose to address the fine-grained image classification from a novel perspective, namely for the fine-grained classes, with the visually similar local features, more attention should be made on leveraging the relational information between the global and local views of an object for encoding its semantic label. Relational proxies are designed based on the proposed theory and achieve the superior results on multiple benchmark datasets.", "questions": "As noted in the weakness part, I'd like to see some visualized validations of the hypothesis to verify the effectiveness and limitations of the proposed method.", "limitations": "As noted by authors, the main limitation is considered as the local views obtained by the proposed method is cropped from the global view, which may not be the best representations of local parts.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nA hypothesis is made for fine-grained visual classification, i.e. when two categories possess same local attributes and differ only in the way the attributes combine to generate the global view of the object, relation-agnostic approaches do not capture the full semantic information in an input image. This hypothesis is then proved by theory and validated in the experimental parts, which is the main theoretical contribution of the paper. I do like this novel perspective.\n\nWeakness:\nI do not have major concerns regarding the technical details, however, the visualization analysis of the proposed method is lacking, for example, under what circumstance that the proposed model can significant improves the performances and when does it fail.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Rvj9lGjmM88", "reviewer": "Reviewer_erdD", "summary": "This paper proposes a method based on relationships between views of an object to perform fine grained visual categorization. The authors hypothesise that not only representing local parts but relating them is pivotal to achieve good performance. The authors then experiment using their proposed approach on different FGVC datasets, improving on current methods.", "questions": "Stated above.", "limitations": "The authors discuss limitations.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The point the paper makes, that the relationship between object parts is important for learning the classification label makes intuitive sense and seems to be important in their experiments.\n\n2. The paper is reasonably clear as to the motivations and experiments run by the authors.\n\nWeaknesses\n1. The authors, in my opinion, over complicate the mathematical explanation, spending two pages explaining why modelling the relationships between object parts is important. I do not think this adds much to the paper and the space would be better spent giving a high level, clear explanation of the intuition.\n\n2. I am not sure why their AST (how they combine local parts) is so different from an attention layer (e.g. in TransFG). It seems to be performing the same operation, so I do not understand how this is fundamentally different than the TransFG architecture. Is the main difference the global embedding of the object that the local crops can be compared against and using all three embeddings -- the z_g, z_L, r -- when computing distances and the final metric?\n\n3. In general the paper is clear: their main contribution is using the global, local and r information in the metric learning and so the insight is that using all 3 sources of information is the most useful. However, in the experiments there seems to be limited improvement from these properties over the base encoder (Table 2). So I wonder how useful these things really are. Moreover, the improvements are small over standard methods, and there is no standard deviation to explain if these results are significant. I further wonder if the authors carefully made sure that their setup was similar to the underlying setup and that the improvement was actually due to their method or better data augmentation or underlying architectures.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PpSx7RDZdX-", "reviewer": "Reviewer_1ZBu", "summary": "The authors propose Relational Proxies, a novel approach that leverages the relational information between the global and local views of an object for encoding its semantic label.  ", "questions": "Insufficient experiments and limited performance gains.", "limitations": "N/A.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think the main novelty comes from the introduced Relational Proxies and the corresponding comprehensive theoretical and experimental analysis.\n\nWeaknesses One area of improvement for the paper at hand would be clarity, especially with respect to the exposition of the proposed architecture. It takes multiple read throughs in order to identify the actual architecture proposed.\n\nLack of experiments on larger datasets. In a time of ever-growing dataset sizes it would be good to provide and compare results of suchmodels when trained on larger datasets. This is important for judging the impact asimprovements stemming from architecture engineering typically vanish with growing dataset sizes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "0y1yic1l98q", "reviewer": "Reviewer_9xUV", "summary": "Summary.\n\nThis paper is dedicated to developing algorithms for fine-grained image recognition. They argue it is not enough to distinguish fine-grained categories only based on partial information. Therefore, they propose relational proxies, which leverage the relational information between the global and local views of an object. They also provide theoretical explanations to support the effectiveness of the proposed methods. Experiments on six fine-grained benchmark datasets offer positive results.", "questions": "Refer to the weakness section.", "limitations": "Both limitations and potential negative social impacts are discussed in the submission.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros.\n\n1. The proposed methods make sense and are well-motivated. Both theoretical and empirical analyses are provided to support the effectiveness.\n\n2. The paper is well written and easy to follow. Figure 1 is informative and illustrative.\n\n\n\nCons.\n\n1. There are missing numbers in Table 1. For a comprehensive comparison, it is necessary to complete it. Minor: the caption of tables should be on the above content.\n\n2. The performance gains are marginal, especially on CUB (0.3%) and NA Birds (0.2%)? Any explanations? It seems the proposed methods are less working for bird images. Meanwhile, the error bar is required for Table1 since the current accuracy margin is too small.\n\n3. More network backbones are needed to support the generalization of proposed methods across architectures.\n\n4. As for the study in Figure 2, the number of local views |L| and the size of the local patch should be correlated. A detailed analysis is needed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept