SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Graph-based image representation using patches was recently proposed in ViG [27]. Although the initial decomposition of an image into a graph allowed for flexible cross-view relationship learning, the image graph is eventually abstracted into a single representation in  $\mathbb{R}^n$ . This prevents one from identifying the most informative subgraph responsible for prediction, at both the instance level, and at the class-level. Although interpreting ViGs may be possible via existing GNN explainability algorithms [88, 51, 82], they are typically post-hoc in nature, meaning that only the output predictions can be explained and not the entire classification pipeline. Incurred computational and time overheads that come with post-hoc approaches in general, are additional downsides. Our approach preserves the transparency of the entire classification pipeline by maintaining a graph-based representation all the way from the input to the class-proxy, thereby allowing us to explain predictions at both instance and class levels in an ante-hoc manner.
Although graph metric learning has been shown to be effective for learning fine-grained discriminative features from graphs [45], instance-based contrastive learning happens to be (1) computationally expensive and (2) unable to capture the large intra-class variations in FGVC problems [56, 39]. Proxy-based graph metric learning [99] has been shown to address both issues, while learning generalized representations for each class. However, the metric space embeddings learned by the above methods are not fully explainable in terms of identifying the most influential subgraph towards the final prediction, which is an issue we aim to address in our work.
A.11 Role of Transitivity
The idea of transitivity allows us to narrow down sets of views whose relation-agnostic information have been exhaustively encoded in  $\mathbb{Z}$ , and the only available sources of information comes only when all the views are observed collectively as the emergent relationship. On the other hand, dealing with a general local-to-global relationship recovery would additionally include view-pairs, only one of which might be contributing to the relational information ( $I(\mathbf{v}_1\mathbf{v}_2;\mathbf{g})$  and  $I(\mathbf{v}_2\mathbf{v}_3;\mathbf{g})$ ), thus not exhibiting emergence. Using only the former in our proof helps us to identify the set of nodes responsible for bridging the information gap (via emergence) as the ones satisfying transitivity.
A.12 Learning the Graph Edit Costs
The GNN proposed in [64] would directly learn the mapping  $\phi : \mathbb{Z}_{\mathbb{V}} \to \mathbb{R}$  for computing the edit costs. In TRD, we instead split the GNN for edit cost computation into a prefix network  $\varphi : \mathbb{Z}_{\mathbb{V}} \to \mathcal{M}$  and a cost head  $\psi : \mathcal{M} \to \mathbb{R}$ . Note that this factorization is just a different view of  $\phi$  that allows us to draw parallels of our implementation with the theory of TRD.
A.13 Further comparisons with Relational Proxies
Same number of local views: We evaluate both Relational Proxies and TRD with the same number of local views in the normal (no explicit addition of noise) setting on the FGVC Aircraft dataset, and report our findings in Table 4. TRD marginally outperforms Relational Proxies for all values of the number of local views, and exhibits a trend of scaling in accuracy with increasing number of local views. Relational Proxies, on the other hand, does not seem to benefit from increasing the number of local views, possibly due to its lack of robustness to noisy views.
Compute cost: In Table 5, we provide the computational costs of Relational Proxies and TRD in terms of wall clock time evaluated on FGVC Aircraft (same experimental settings including the number of local views). We can see that TRD is significantly more efficient than Relational Proxies in terms of both single sample inference as well as training time until convergence. This is because of the following reasons:
# Local Views	8	16	32	64
Relational Proxies	95.25	95.30	95.29	95.31
TRD (Ours)	95.27	95.45	95.52	95.60
- The Complementarity Graph in TRD is constructed exactly once before training, and the semantic relevance graph, as well as the proxy graph are learned as part of the training process.
- TRD does not involve updating the relation-agnostic encoder, which is a ResNet50, as part of the training process. Relational Proxies requires it to be updated, thereby exhibiting computationally heavier forward (as local view embeddings cannot be pre-computed) and backward passes.
Table 4: Comparison of TRD with Relational Proxies under the same number of local views.
Average Inference Time (ms)	Training Time (hrs)
Relational Proxies	130	22
TRD (Ours)	110	15
A.14 Over-smoothing
To understand whether the process of modelling emergent relationships is vulnerable to the oversmoothing phenomenon in GNNs, we evaluated TRD using GATs of up to 64 layers on FGVC Aircraft, presenting our findings in Table 6. We see that the performance does drop as the number of layers are increased beyond 8. To validate whether this is due to the oversmoothing phenomenon, we measure the degree of distinguishability among the nodes by taking the average of their pairwise  $L^2$  distances. The table shows that the distinguishability also decreases as we increase the number of layers, suggesting that the over-smoothing phenomenon does occur. Under the light of the above experiments, the 8-layer GAT is an optimal choice for our problem.
Table 5: Compute cost of TRD relative to Relational Proxies.
GAT-Depth	4	8	16	32	64
Accuracy	95.05	95.60	95.32	94.78	94.20
Distinguishability	0.87	0.63	0.49	0.21	0.09
Table 6: Effect of over-smoothing in GAT on learning emergent relationships.
A.15 Results on ImageNet subsets
Following existing FGVC literature [10, 21], we evaluate the contribution of our novel Transitivity Recovery objective in the coarse-grained (multiple fine-grained subcategories in a single class) and fine-grained subsets of ImageNet, namely Tiny ImageNet [44] and Dogs ImageNet (Stanford Dogs [38]) respectively, and report our findings in Table 7. Although our method can surpass existing SOTA in both the settings, larger gains  $(\Delta)$  are achieved in the fine-grained setting, suggesting that TRD is particularly well suited for that purpose.
A.16 Dependence on input image size
We follow recent SOTA FGVC approaches that use relation-agnostic encoders to extract global and local views [10, 94, 84]. In particular, our view extraction process is exactly the same as Relational Proxies [10], with the same input image resolution and backbone (relation-agnostic) encoder. The
Tiny ImageNet	Dogs ImageNet
MaxEnt [21]	82.29	75.66
Relational Proxies [10]	88.91	92.75
a: w/o Transitivity Recovery	88.10	91.03
b: with Transitivity Recovery	89.02	93.10
Δ = (b - a)	0.92	2.07
above approaches first extract the global view from the input image, and crop out local views from the global view. Since there are two scales at which the image is cropped, all the crops are resized to  $224 \times 224$ . In practice, this provides a similar resolution to resizing the full input image to a higher resolution at the start. To evaluate this hypothesis, we re-trained and re-evaluated our model with the input images resized to  $448 \times 448$ , keeping the remainder of the process of view extraction the same. We provide our results on multiple datasets in Table 8, which shows that the performance gains at the higher input resolution are minor, thus supporting our hypothesis.
Table 7: Coarse-grained vs fine-grained classification results.
Soy	FGVC Aircraft	Stanford Cars
TRD: 224 × 224	52.15	95.60	96.35
TRD: 448 × 448	52.23	95.62	96.39
Table 8: Dependence of TRD on input image size.
A.17 Results with Causal Interventions
Figure 6: Top-down: Qualitative results on FGVCAircraft, Stanford Cars, and NA Birds with causal intervention. The first two columns from the left are instance graphs, and the rightmost column contains the proxy graphs. The local views of the correct class are bordered in green, while the ones from a different class are bordered in red. A subset of the local views for each instance was corrupted with the introduction of local views from other classes, both during training and inference. These local views consequently also feature in the proxy graphs. The negative local views from different classes (red) can be seen to have a significantly weaker connection to the global view (bordered in blue), and the correct set of local views (green).