<figure><img src="image_1.png" /></figure>

HAL
open science

# BR-NPA: a non-parametric high-resolution attention model to improve the interpretability of attention

Tristan Gomez, Suiyi Ling, Thomas Fréour, Harold Mouchère

► To cite this version:

Tristan Gomez, Suiyi Ling, Thomas Fréour, Harold Mouchère. BR-NPA: a non-parametric high-resolution attention model to improve the interpretability of attention. Pattern Recognition, 2022, 132, pp.108927. 10.1016/j.patcog.2022.108927. hal-03773655

HAL Id: hal-03773655

https://hal.science/hal-03773655v1

Submitted on 28 May 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# BR-NPA:
A Non-Parametric High-Resolution Attention Model to improve the
Interpretability of Attention

Tristan Gomez<fnref n="1" />

tristan.gomez@etu.univ-nantes.fr

Christian Pauc Street, 44300 Nantes, France

Suiyi Ling<fnref n="1" />

suiyi.ling@univ-nantes.fr

Christian Pauc Street, 44300 Nantes, France

Thomas Fréour<fnref n="2" />,<fnref n="3" />,<fnref n="4" />

thomas.freour@chu-nantes.fr

63 Magellan Quai, 44000 Nantes, France

Harold Mouchère<fnref n="1" />,<fnref n="5" />

harold.mouchere@univ-nantes.fr

Christian Pauc Street, 44300 Nantes, France

## Abstract

The prevalence of employing attention mechanisms has brought along concerns on the interpretability of attention distributions. Although it provides insights about how a model is operating, utilizing attention as the explanation of model predictions is still highly dubious. The community is still seeking more interpretable strategies for better identifying local active regions that contribute the most to the final decision. To improve the interpretability of existing attention models, we propose a novel Bilinear Representative Non-Parametric Attention (BR-NPA) strategy that captures the task-relevant human-interpretable information. The target model is first distilled to have higher-resolution intermediate feature maps. From which, representative features are then grouped based on local pairwise feature similarity, to produce finer-grained, more precise attention maps highlighting task-relevant parts of the input. The obtained attention maps are ranked according to the activity level of the compound feature, which provides information regarding the important level of the highlighted regions. The proposed model can be easily adapted in a wide variety of modern deep models, where classification

<footnote n="1">¹Nantes University, CNRS, LS2N, CNRS UMR 6004, F-44000 Nantes, France</footnote>

<footnote n="2">²Nantes University Hospital, Department of Reproductive Medicine and Biology, F-44000 Nantes, France</footnote>

<footnote n="3">³Nantes University, Nantes University Hospital, Inserm, CRTI, Inserm UMR 1064, F-44000 Nantes, France</footnote>

<footnote n="4">⁴Nantes University, Nantes University Hospital, Inserm, CNRS, SFR Santé, Inserm UMS 016, CNRS UMS 3556, F-44000 Nantes, France</footnote>

<footnote n="5">⁵Corresponding author</footnote>
is involved. Extensive quantitative and qualitative experiments showcase more comprehensive and accurate
visual explanations compared to state-of-the-art attention models and visualizations methods across multiple
tasks including fine-grained image classification, few-shot classification, and person re-identification, without
compromising the classification accuracy. The proposed visualization model sheds imperative light on how
neural networks ‘pay their attention’ differently in different tasks.

Keywords: deep learning; interpretability; spatial attention; resolution; non-parametric

# 1. Introduction

<figure><img src="image_3.png" /><figcaption>Figure 1: BR-NPA yields precise attention maps focused on small details and highlights a potential hierarchy among the object's parts relevance.</figcaption></figure>

Thanks to the unprecedented booming of deep learning, many applications enjoyed a big leap in performance
in the last decade. Nevertheless, these deep learning-based methods are commonly considered as ‘black box’
approaches, where the internal functioning is unclear [1]. As one of the solutions to remedy this lack of
interpretability, attention mechanisms [2] are frequently employed for explaining the deep model. Yet to which
extend those attention models help in interpreting how the model is operating has recently become a novel
debate [3, 4]. An ideal interpretable attention model should not only be able to quantify the important level of a
certain part of the neural network but also be capable of identifying precise local regions of the input to provide
proper visual guidance regarding which key component is playing a greater role in the decision-making of the
corresponding task [3]. To this end, in this study, we propose a novel framework that generates high-resolution
interpretable attention maps without sacrificing the performance of the target model. More specifically, this
framework improves the interpretability of the usual multi-part attention system designed for deep Convolutional
Neural Network models (CNNs) in two ways: (1) we propose to increase the resolution of the output feature maps of a network and the attention map for finner details ; (2) we propose to replace the common learned attention layer [2] with a non-parametric layer that groups pixels based on their *activity level*, and similarity with others. This layer does not require training, which constitutes a simpler function than the standard attention module made of convolutional layers. It produces comprehensive and accurate attention maps focusing on semantic parts of the input while sacrificing little to no classification accuracy.

Examples of multi-parts attention maps obtained using our method are shown in Fig. 1. Our algorithm is designed such that the importance level (in terms of contributions to the final decision-making of the task) of different salient parts of objects could be also friendly visualized via different colors. The $1_{st}$, $2_{nd}$, and $3_{rd}$ attention maps are respectively represented in red, green, and blue. Similarly, attention maps generated by state-of-the-art (SOTA) methods are also visualized for comparison. If the method/model produces a single map, it is visualized using a heatmap, e.g., the OS [5] method for person re-identification. Figure 1 shows that contrarily to SOTA models, BR-NPA precisely highlights small interpretable details consistently across all the studied tasks.

In the spirit of calls for more interpretable attention models, the focal point of this contribution is to develop a more explainable and interpretable attention model that could be easily plugged into different applications<sup>6</sup>. Although the proposed attention framework can help to boost performance, we do not focus on contributing to improving the SOTA in standard computer vision tasks, which is beyond the scope of this study.

## 2. Related works

*Generic explanation methods:* One of the most straightforward ways to identify which local component of the input contributes the most to the model decision-making is the Activation Map (AM) that simply outputs a heatmap showing the average activation across channels. However, not all the activated areas offer necessarily cues for accomplishing the target task, e.g., classifying a certain category. To overcome this drawback, more interpretable visualization approaches including the Class-Activation Map (CAM) [6], Grad-CAM [1] or Grad-CAM++ [7] were proposed to locate the local task-relevant areas. Other methods like Grad-CAM++ [7] or Score-CAM [8] were proposed to improve the interpretability of Grad-CAM. Another line of work called RISE [9] opted instead for a non-gradient-based approach using random masks to find the areas that contribute the most to the prediction. In [10], Guided-Backpropagation (GB) was proposed to visualize the gradients of the input image pixels relative to the predicted output and generate high-resolution saliency maps that highlight important pixels for the prediction. This was later improved with noise tunnel-based methods like Vargrad [11] or Smoothgrad [12].

<sup>6</sup>Source code is available here https://gitlab.univ-nantes.fr/E144069X/br_npa.
*Attention-based architecture for specific task.* Many existing attention-based models were developed for fine-grained classification: Lin *et al.* proposed a bilinear CNN (B-CNN) that employs two distinct CNNs to obtain the feature maps from the intermediate convolutional layer just before the pooling layer, and further multiply them with outer product [13] to explore the coefficient between network features. Inspired by this work, in [14], it was proposed to generate the attention maps by integrating a convolutional module to the intermediate output feature. Several similar soft attention layer-based models were also developed for the same purpose, including the Prototypical Part Network (ProtoPNet) [15] that matches learned feature vectors against the feature volume in the manner of a convolution. With a similar recipe, another follow-up work, *i.e.*, the ProtoTree [16] uses a tree of prototypes to make a decision. By fully exploiting a simple prior of the occurrence of object parts, Huang *et al.* also showed it was possible to learn to separate an object into parts [17].

Several methods proposed to generate part-level crops using reinforcement learning like Fully Convolutional Attention Networks (FCAN) [18] or weakly-supervised learning like Spatial Transformer Networks (ST-CNN) [19], Multi-Granularity CNN (MG-CNN) [20] or MA-CNN [21]. Various neural attention layers were designed for the task of person re-identification [22, 23, 5, 24, 25]. The Relation-aware Global Attention (RGA) module was proposed in [22] to capture the global structural information. In [23], the authors suggest to jointly learn the soft and hard attentions along with simultaneous optimization of feature representations. Omni-Scale (OS) feature learning [5] was proposed to detect features at multiple scales and dynamically fuse them through a novel aggregation gate. Some works have also been introduced to make full use of the particular modalities of the task of person re-identification by exploiting both spatial and temporal clues [24, 25]. There are also attention mechanisms that are tailored to few-shot/meta-learning algorithms. Several models were developed to better localize relevant object regions and improve sample efficiency [26, 27, 28]. Analogously, Zhu *et al.* propose to learn diverse and informative parts [29]. Recently, a unique module is presented to find relevant frames in a video [30]. Notwithstanding all the appalling attention existing task-specific attention models, one common trait of these works is that they yield attention methods that are tailored for only one typical task.

### 3. The Proposed Model

The proposed attention model is composed of (1) extracting high-resolution feature maps; (2) generating representative feature vectors obtained via grouping similar vectors together, and (3) concatenating the representative feature vectors and forwarding them to the classification layer. In this study, the intermediate output of a convolutional network (layer), *i.e.*, a tensor with a size of $C \times H \times W$, is defined as the *feature volume* or *feature maps*. Once a spatial position $(h, w)$ is chosen, where $h \in [1, H], w \in [1, W]$, a vector of dimension *C* can be extracted, namely, the *feature vector*.
### 3.1. High-Resolution Feature Maps

Given a standard CNN, commonly, the size of the last feature map is significantly smaller than the one of the input. For example, after feeding an image of 448 × 448 to a ResNet-50 backbone [31] network, the size of the output feature map is reduced to 14 × 14. This makes it difficult to interpret how the neural network was activated with respect to the image, as each feature vector covers a large and potentially heterogeneous area of the image, e.g. both the object and the background. When visualizing (a part of) the network with features maps of higher-resolution, i.e., with smaller receptive field, each point within the feature map corresponds to a smaller area of the original that is more likely to be homogeneous. To increase the attention map resolution, bi-linear or bi-cubic interpolations are commonly used for visualization. Nonetheless, these operations usually lead to blurry attention maps with no details [1, 32, 33, 34] Another alternative is to consider larger input images as done in [35]. Nevertheless, this method inevitably increases the computation cost and also introduces relevant distortions into original input images during the up-scaling procedure. Instead, we proposed to re-adapt the architecture of the CNN backbone used to prevent the reduction of the spatial resolution of feature maps. Concretely, this is achieved via reducing the stride of the last downsampling layers in the backbone network. By doing so, the resolution of the feature maps increased from 14 × 14 to 56 × 56 (cf. section 4.1). This limits the computation cost increase as only the feature maps of the last layers become larger.

In all our experiments, we employ pretrained backbones and this modification implies that the kernels of the last layers will be applied at a scale for which they were not pretrained for. As a result, during training, the backbone has to learn features that are adapted to the task but also the new scale, leading to longer training time and longer hyper-parameter optimization, which reduces the efficiency of the transfer learning procedure 7.

To mitigate this issue, the distillation method proposed in [37] is employed. More specifically, a student network with increased resolution is trained to imitate the lower-resolution teacher network. When training the student network, the usual cross-entropy loss was employed jointly with the KL-divergence between the predictions of the student and teacher network to bridge the gap between the two models:

$$L = \frac{1}{N} \sum_{i} \alpha \mathrm{CE}(\tilde{y}_s, y) + (1 - \alpha) \mathrm{KL}(\tilde{y}_t || \tilde{y}_s), \quad (1)$$

where $\tilde{y}_t$, $\tilde{y}_s$ denote the outputs of the teacher and student models, y is the ground truth and α is a parameter that balances the cross-entropy term and the KL divergence term.

### 3.2. Non-Parametric Attention Layer

In most cases, the object of interest is composed of several parts, where each part potentially plays a different role in classification. Modern classification models process input and predict corresponding labels in

<footnote n="7">7 Note that we also experimented with the HR-Net backbone [36] which provides 56 × 56 features maps, but obtained poor performance, probably due to the small amount of feature maps provided by HR-Net (16 to 64 depending on the number of layers, whereas ResNet [31] provides 512 to 2048 feature maps).</footnote>
an end-to-end manner, the extraction of important local parts, the modeling of their pairwise correlations, and their contribution to final classification are learned throughout the network. It is thus intuitively appealing to explore the pairwise correlations between the local object parts intermediate feature maps within the network and visualize the part-feature interactions as ‘attention’. In [13], features corresponding to each object part were aggregated separately to construct a feature matrix, i.e., a list of feature vectors. The separation of local object parts could be achieved via attention layers composed of multiple convolutional blocks [14], which is parametric.

With the burgeoning improvement of neural network architectures, SOTA models achieve impressive performances on multiple large-scale benchmarks. Therefore, the backbone networks utilized in those models should be equipped with the capabilities of separating the local parts. Grounded by well-performing network architecture, instead of incorporating the usual attention neural module, a list of refined feature vectors are formed sequentially by grouping intermediate feature vectors with respect to their activation and their similarity with each other.

In this study, the activation level $a$ of a feature vector $f$ is quantified by its euclidean norm, i.e., $||f||^2$. The feature vector $f$, whose norm is one of the highest in the corresponding feature volume, is considered to be active. Furthermore, a feature vector is considered singular if the other selected features are not similar to it.

<figure><img src="image_4.png" /><figcaption>Figure 2: Illustration of the method used to group features without any dedicated module.</figcaption></figure>

Alg. 1 and Fig. 2 summarize the proposed feature grouping algorithm. In a nutshell, it selects the top $N$ active singular (i.e., not similar to each other) feature vectors from the feature volume and further refines them by aggregating them with similar vectors. This algorithm outputs an $N \times M$ feature matrix, where $M$ is the dimension of the vectors. For simplicity, the feature vectors are indexed with only $i \in [0, H \cdot W - 1]$, where $H \times W$ is the resolution of the feature map. The $i_{th}$ vector is denoted as $f_i$. The selection of $N$ is discussed in the supplemental materials. Experimentally, we observed that using $N > 3$ does not improve accuracy significantly and makes visualization more complex.

During the first interaction, each $f_i$ was initialized with an activation score $a_i$ equals to its euclidean norm
**Algorithm 1** Identification of representative vectors {$\hat{f}_k$}

Input : feature vectors {$f_i$}

$$ (1) \quad a_i \leftarrow \|f_i\|^2 $$

for $k = 1$ to $N$ do

$$ i_{max} \leftarrow \operatorname{argmax}_{i} a_i $$

for all $i$ do

$$ (2) \quad s_i \leftarrow \cos(f_i; f_{i_{max}}) $$

$$ (3) \quad w_i \leftarrow s_i / \sum_{i'} s_{i'} $$

$$ (4) \quad \hat{f}_k \leftarrow \sum_{i} w_i \times f_i $$

$$ (5,6) \quad a_i \leftarrow (1 - w_i) \times a_i $$

end for

end for

$$ \textbf{return} \hat{f}_1, \hat{f}_2, \dots, \hat{f}_N $$

(corresponds to Step 1 in Alg. 1 and Fig. 2). Afterward, the vector with the highest norm is selected (Step 2). The motivation behind this procedure is that among all the feature vectors in the feature volume of a regular CNN, the ones with higher norms tend to have greater impacts on the final feature vector obtained after the pooling layer e.g., the average/max pooling. When employing a well-performing network architecture for a certain task, the high norm features are expected to be more discriminative than the ones with a lower norm. Higher norm features are also supposed to contribute more to the final prediction. Hence, by selecting the high norm vector, we aim at reproducing this phenomenon in our architecture. This intuition is empirically verified in section 4.3, where experimental results show that accuracy decreases with random vectors selection or with only the $2_{nd}$, $3_{rd}$ ranked features.

After obtaining the feature vector with the greatest norm value $f_{i_{max}}$, it is then refined with other features that are similar to it (Step 2 to 4). Concretely, the cosine similarity $s_i$ between each feature $f_i$ and $f_{i_{max}}$ is computed (Step 2). Subsequently, {$s_i$} is normalized to yield weights {$w_i$} such that $\sum_i w_i = 1$ (Step 3). The $k_{th}$ feature vector $\hat{f}_k$ is then obtained by computing the weighted average across all the vectors (Step 4). Afterward, a dissimilarity map $1 - w_i$ is computed by reversing the weights (Step 5) and the activation scores {$a_i$} are updated by multiplying them by $1 - w_i$ for all $i$ (Step 6). By doing so, it could be ensured that the new chosen location $i_{max}$ and its estimation $f_{i_{max}}$ is different from the ones chosen in the previous iterations, while still extracting high norm vectors. The aforementioned procedure (line 3-8) is repeated for the $k + 1_{th}$ feature vectors until top-$N$ feature vectors are computed. Lastly, the obtained $\hat{f}_1, \hat{f}_2, \dots, \hat{f}_N$ are concatenated and feed to the final classification layer. The weights {$w_i$} corresponding to the top-$N$ representative features serve as attention maps for highlighting local parts that contribute the most to the final decision-making of the task, with
the rank of the features indicating the importance level of the respective object parts. It is worth noting that, as
there are no extra attention neural layers integrated, our attention model is non-parametric. This scheme can be
easily plugged into any modern deep neural network framework, where classification is associated.

4. Experiments

The proposed framework was first evaluated on various computer vision tasks on relevant SOTA bench-
marks, including the fine-grained image classification [38, 39, 40], few-shot classification [41] and person
re-identification [42]<fnref n="8" />. Furthermore, extensive ablation studies were conducted to validate the chosen strategies.
Lastly, a quantitative evaluation of the attention maps using dedicated metrics was performed to demonstrate the
performance of the proposed framework.

We compared the proposed BR-NPA attention model with SOTA models from the literature like B-CNN
[43], ProtoPNet [15], ProtoTree [16], Interpretability by parts (IBN) [44], RGA [22], HA-CNN [23], OS
[5] and Cross-Attention [27] and also SOTA post-hoc algorithms dedicated to model visualization including
Grad-CAM [1], Grad-CAM++ [7], Guided backpropagation [10], Score-CAM [8], RISE [9], VarGrad [11],
SmoothGrad [12], and also the baseline activation map (AM) method. The attention models were trained using
the author's code and we used Captum [45] for the generic visualization methods. For fine-grained classification,
we also report for reference the performance of 2-level attn. [46], MG-CNN [20], FCAN [18], ST-CNN [19]
and MA-CNN [21]. These models were not included in the qualitative or quantitative evaluations of the saliency
maps because they are hard attention models that produce bounding boxes, preventing comparison with the soft
attention maps produced by BR-NPA, but they are still attention models which performance is interesting to
compare to.

B-CNN is a simple baseline architecture that makes minimal hypotheses about the task at hand, making it
appropriate to be applied to other tasks than the one it was developed for. Therefore, among the aforementioned
non-post-hoc models, there is only B-CNN was trained on all considered tasks, since the other attention models
were developed/designed for a certain dedicated task. On the other hand, since the post-hoc algorithms are
generic methods, each of them was applied to all of the three tasks. The visualized results (maps) of the post-hoc
algorithms were obtained using a regular CNN without attention trained with the same setting as the attention
models.

4.1. *Implementation Details*

As mentioned in section3.2, we chose *N* = 3 for BR-NPA as it is a good trade-off between accuracy and
interpretability. More detail about the selection of *N* can be found in supplementary materials. As the number of

<footnote n="8">⁸More visualisations are available in supplementary material.</footnote>
attention maps generated for an image varies greatly among other models (e.g., 32 for InterByParts, up to 2000 for ProtoPNet), we selected 3 attention maps among the ones produced by each model. This ensures concise and fair qualitative comparison among models. For visualization, each of the 3 attention maps is mapped to one channel of an RGB image (details are given below). Only the B-CNN model was trained using 3 parts, as using a larger number of object parts tends to reduce its performance<fnref n="9" />. For other models, we kept the same number of part/attention maps as reported in the original papers. To select the attention maps, we adopted different strategies. For the ProtoTree and ProtoPNet, we chose the most active maps, i.e., the maps that have the highest similarity score with the input image, as they are the most relevant regarding the target task. For RGA and HA-CNN we used the attention maps generated by the last 3 layers, as they are the closest to the decision-making layers. Once the 3 maps are selected, they were visualized by showing the $1_{st}$, $2_{nd}$ and $3_{rd}$ maps respectively in red, green, and blue. More precisely, the red, green, and blue channels of each pixel are were to represent its contribution w.r.t. the $1_{st}$, $2_{nd}$ and $3_{rd}$ attention maps. The three channels were also multiplied by the norm of the corresponding feature vector, to make salient pixels brighter. The OS model and the saliency algorithms only generate one attention map, they were visualized using a simple heatmap.

For all experiments and tasks, hyper-parameters were estimated with the Tree-structured Parzen Estimator (TPE) from the Optuna framework [47], except in section 4.3, where we used the default hyper-parameter values detailed in supplementary materials. Details of how the proposed attention was adjusted and employed for different tasks are given below.

**Fine-grained image classification:** For both BR-NPA and the baseline model B-CNN, the ResNet-50 [31] pre-trained on the ImageNet dataset [48] was considered as the backbone network since it is one of the most commonly used network architectures in various domains. To obtain higher-resolution feature maps, the stride of the downsampling blocks in the $3_{rd}$ and $4_{th}$ layer of the ResNet-50 backbone network was set to 1 instead of 2, which increased the feature maps from $14 \times 14$ to $56 \times 56$. The teacher model used for distillation is a BR-NPA with an unmodified backbone, i.e. with a $14 \times 14$ resolution.

**Few-shot classification:** The E3BM model proposed in [49] is one of the SOTA meta/few-shot classification models. It was thus employed as a backbone with the studied attention model, i.e., B-CNN, CrossAtt. and BR-NPA for more comprehensive visualization. To compare BR-NPA with B-CNN and CrossAtt., the average pooling layer in the backbone in E3BM was replaced with the studied attention layers.

For B-CNN and BR-NPA, the original classification layer was also extended to take the concatenation of the 3 representative features as input. No such modification was necessary for CrossAtt. as it does not extend the size of the feature vector produced. The model was then trained as described in [49] on one of the standard few-shot classification benchmark datasets TieredImagenet [41] using pretrained weights provided by authors to reduce training time. In line with E3BM, the ResNet-12 was used. The stride of the downsampling blocks in

<footnote n="9">⁹More details can be found in supplementary materials.</footnote>
$2_{nd}$ and $3_{rd}$ layer of the ResNet-12 were set to 1 instead of 2. As a result, the resolution of the feature maps was enlarged from 5 × 5 to 21 × 21. The teacher model used for distillation is the pretrained E3BM model which weights were provided by authors.

**Person re-identification:** Similar to the setup for the few-shot classification task, one of the best-performing models proposed in [50], i.e., DG-Net, was adapted with the studied methods. For BR-NPA and B-CNN, the average pooling layer in the identification module used by [50] was removed and replaced with the attention layer, and the final dense layer was also expanded to match the new feature vector size. To prevent a size mismatch between the weights for the original dense layer with the feature vector extraction (512 × 2048) and the expanded dense layer (512 × 2048 · *N*), we duplicated the weights *N* times along the feature axis. This procedure was used to train DG-Net in combination with both BR-NPA and B-CNN. For RGA, HA-CNN, and OS, the original backbone was replaced by the CNN proposed by these methods, trained in the same setup as DG-Net. Then, the model was trained as elucidated in [50] on the popular Market-1501 dataset [42]. It has to be emphasized that the DG-Net has a complex architecture that is difficult to train from scratch. Thus, we utilized the pretrained weights provided by the authors and modified the DG-Net architecture to speed up the training procedure.

The ResNet-50 was used as the backbone for the task of person re-identification to be consistent with [50]. The strides of the $2_{nd}$ and $3_{rd}$ layers of the ResNet-50 were set to 1 instead of 2, knowing that the stride of the $4_{th}$ layer was already set to 1 by the original authors. This increased the resolution of the feature maps from 16 × 8 to 64 × 32. All models were trained with the same distillation procedure and the same teacher model as the original DG-Net model.

## 4.2. *Experimental Results*

**Fine-grained image classification:** The performances of the proposed model and SOTA visualization models in terms of accuracy on three fine-grained datasets, including the CUB-200-2011 [38], FGVC-Aircraft [39] and the Standford cars [40] datasets are depicted in Table 1. It is shown that BR-NPA achieves SOTA performances, and outperforms the other interpretability-oriented methods on the three datasets.

Selected visualizations of the attention maps are presented in Fig. 3. Among all the images, the attention maps produced by BR-NPA highlight more precise details of the objects that capture well the inter-class unique characteristic. For example, in rows 4-6, highlighted finer-grained local regions are most of the time the tails or wings of the airplanes, which help to differ different airplane models. Similarly, the other attention models are also able to identify similar local important parts, but they are generally less precise and consistent. Conversely, the highlighted regions from Grad-CAM, Grad-CAM++, AM, RISE, and Score-CAM approaches are significantly coarser, which cover the entire objects, while the ones from Guided-Backpropagation, VarGrad, and SmoothGrad are extremely sparse, making it difficult to identify the important areas and also sometimes spread out randomly all over the images. None of those attention regions provide precise information that helps
<table><caption>Table 1: Performance for task of fine-grained classification.</caption><thead><tr><th>Method</th><th>Attention</th><th>CUB</th><th>FGVC</th><th>Stanford cars</th></tr></thead><tbody><tr><td>2-level attn. [46]</td><td>Hard</td><td>77.9</td><td>-</td><td>-</td></tr><tr><td>MG-CNN [20]</td><td>Hard</td><td>81.7</td><td>-</td><td>-</td></tr><tr><td>FCAN [18]</td><td>Hard</td><td>82.0</td><td>-</td><td>-</td></tr><tr><td>ST-CNN [19]</td><td>Hard</td><td>84.1</td><td>-</td><td>-</td></tr><tr><td>MA-CNN [21]</td><td>Hard</td><td>85.4</td><td>88.4</td><td>91.7</td></tr><tr><td>InterByParts [17]</td><td>Soft</td><td>81.9</td><td>87.9</td><td>89.5</td></tr><tr><td>ProtoTree [16]</td><td>Soft</td><td>82.1</td><td>81.3</td><td>85.7</td></tr><tr><td>B-CNN [43]</td><td>Soft</td><td>84.1</td><td>84.2</td><td>91.3</td></tr><tr><td>B-CNN (our impl.)</td><td>Soft</td><td>82.9</td><td>88.6</td><td>89.9</td></tr><tr><td>ProtoPNet [15]</td><td>Soft</td><td>84.8</td><td>84.0</td><td>85.8</td></tr><tr><td>BR-NPA</td><td>Soft</td><td><b>85.5</b></td><td><b>89.6</b></td><td><b>92.2</b></td></tr></tbody></table>

the decision-making of fine-grained classification.

**Person Id-reidentification:** The accuracy of DG-Net equipped with different attention methods on one of the most common benchmark Market-1501 [42] is illustrated in Table 2. Compared to other attention schemes, the lower-resolution version of BR-NPA achieves the best performances. Its performance is slightly poorer compared to the original DG-Net. Yet, as aforementioned, the complex architecture makes it hard to train from scratch, and we hence employed the pre-trained weights. One of the possible reasons for the slight performance drop could be that the pretrained network was not well adapted to our attention framework, and therefore are more prone to a local optimum.

Comprehensive attention maps are shown in Fig. 4. Similar to fine-grained classification, the attention maps produced by BR-NPA emphasize local important details of the clothes on the person that are momentous for the ID re-identification regarding the significant intra-class variations across different cameras. For example, highlighted regions include shoes (rows 1-4), a logo on the bag (row 4), pockets (row 3), or epaulette (row 1), which help to differentiate different persons. B-CNN and OS are only able to focus on larger parts of the person, e.g., the upper-body (rows 1-3) or the whole handbag (row 3), and the RGA and HA-CNN methods barely focus on the body of the person. Similarly to fine-grained recognition, the highlighted regions indicated by the Grad-CAM, Grad-CAM++, AM, RISE and Score-CAM approaches generally cover the entire objects, while the ones from Guided-Backpropagation, VarGrad, and SmoothGrad are extremely sparse, making it difficult to identify the important areas.

Let alone the negligible performance drops compared to the original DG-Net, it is showcased that our
<figure><img src="image_5.png" /><figcaption>Figure 3: Comparison of different visual explanations with BR-NPA for fine-grained classification on 3 datasets: CUB-200-2011 lines 1-3, FGVC-Aircraft line 4-6, and the Standford cars lines 7-9. BR-NPA yields detailed attention maps focused on semantic parts of the object whereas the other methods either produce blurry maps covering the whole object or extremely sparse maps making it difficult to identify the important areas.</figcaption></figure>

<table><caption>Table 2: Performance for the task of person re-identification.</caption><thead><tr><th>Model</th><th>Reference Accuracy</th><th>Attention</th><th>Resolution</th><th>Accuracy</th></tr></thead><tbody><tr><td rowspan="5">DG-Net</td><td rowspan="5">94.8</td><td>B-CNN</td><td>16 × 8</td><td>80.1</td></tr><tr><td>HA-CNN</td><td>10 × 4</td><td>86.6</td></tr><tr><td>OS</td><td>16 × 8</td><td>91.2</td></tr><tr><td>RGA</td><td>16 × 8</td><td>93.0</td></tr><tr><td>BR-NPA (ours)</td><td>16 × 8<br>64 × 16</td><td><b>93.6</b><br>88.1</td></tr></tbody></table>

BR-NPA could be easily plugged in a complex pre-trained architecture without suffering from a significant loss of accuracy while providing insightful visualization.
<figure><img src="image_6.png" /><figcaption>Figure 4: Comparison of different visual explanations with BR-NPA for the re-identification task on the Market-1501 dataset.</figcaption></figure>

**Few-shot classification:** We tested the combination of the E3BM model with 3 attention models: B-CNN, CrossAtt. and the BR-NPA modules. Each combination was evaluated in a 5-shot 5-way few-shot setting on the CIFAR-FS dataset [51]. Mean accuracies are summarized in Table 3. As observed, by integrating the 5 × 5 resolution version of BR-NPA, the accuracy of the original E3BM model is improved, and both 5 × 5 and 21 × 21 resolution versions of BR-NPA provides performance superior to the version with B-CNN or CrossAtt. This shows that BR-NPA is a versatile module that can be integrated into an existing pretrained architecture and maintain the model performance.

<table><caption>Table 3: Performance for the task of few-shot classification.</caption><thead><tr><th>Model</th><th>Reference Accuracy</th><th>Attention</th><th>Resolution</th><th>Accuracy</th></tr></thead><tbody><tr><td rowspan="3">E3BM</td><td rowspan="3">85.8</td><td>CAM</td><td>5 × 5</td><td>81.9</td></tr><tr><td>B-CNN</td><td>5 × 5</td><td>83.2</td></tr><tr><td>BR-NPA (ours)</td><td>5 × 5<br>21 × 21</td><td><b>85.9</b><br>85.3</td></tr></tbody></table>

Representative visualized maps are depicted in fig. 5. On one hand, as opposed to fine-grained classification
<figure><img src="image_7.png" /><figcaption>Figure 5: Comparison of different visual explanations with BR-NPA for the few-shot classification task on the TieredImagenet dataset.</figcaption></figure>

and person re-identification, in the task of few-shot classification, attention maps generated by BR-NPA highlight slightly larger areas of the object, while most of the local parts are still well determined and are distinguishable from each other. For instance, BR-NPA separates the body from its head (rows 1-3 and 7-9) or the ears from the nose (rows 4-6), which are all key relevant components for recognizing the corresponding categories. These observations demonstrate the feasibility of adapting BR-NPA for few-shot learning classifications. Interestingly, unlike the visualization maps obtained in the task of fine-grained classification, the model tends to focus on the entire objects instead of finer regions in a few-shot regime, which also verifies the interpretability of the proposed models across tasks. On the other hand, B-CNN is not able to differentiate different local parts as it only generates one meaningful attention map (in green), the other one (in red) exhibiting a seemingly degenerate behavior. Similarly, CrossAtt. does not differentiate object parts either and exhibits an even more degenerate behavior. Again, the highlighted regions from Grad-CAM, Grad-CAM++, AM, RISE and Score-
CAM approaches generally cover the entire objects, while the ones from Guided-Backpropagation, VarGrad, and SmoothGrad are extremely sparse, making it difficult to identify the important areas and also sometimes spread out randomly all over the image.

**Comparisons of attention maps obtained from different tasks:** The attention maps obtained from different tasks highlight the target object at different granularity-level. For the task of fine-grained classification and person re-identification, our method yields more precise attention maps that focus on finer object parts. Differently, for few-shot classification, the proposed model tends to focus on the entire object as the objective of few-shot is to distinguish between categories instead of sub-categories with only a few samples per unseen class. It is demonstrated that our model can generate interpretable visualized maps according to the goal of the task, which could provide important insights for the development of different methodologies, i.e., different network architectures that pay more attention to a certain type of local areas, regarding the goal of the target task.

### 4.3. Ablation study

**Impact of active feature selection and refinement strategy:** Three ablative studies were conducted to explore the impacts of the proposed strategies of active feature vectors selection and the feature refinement on the performances, including (1) the ablative model where the feature vectors are uniformly and randomly selected among all available vectors instead of utilizing the active selection strategy; (2) the ablative model without the refinement phase, which simply concatenates all the active vectors and passes them directly to the classification layer; (3) a model with random selection and no refinement phase. During this ablation experiment, the aforementioned 3 ablative models along with the original model were trained with reduced strides to increase resolution from 14 × 14 to 56 × 56 (cf. section 4.1) without distillation or hyper-parameter optimization for the sake of efficiency. Results concluded in Table 4 on the CUB-200-2011 dataset verifies the validity of both the active feature selection and refinement strategies.

<table><caption>Table 4: Results of the ablation study using the CUB-200-2011 dataset.</caption><thead><tr><td>Raw vector selection method</td><td>Refining</td><td>Accuracy</td></tr></thead><tbody><tr><td>Random</td><td>No</td><td>0.5</td></tr><tr><td>Random</td><td>Yes</td><td>0.5</td></tr><tr><td>Active</td><td>No</td><td>71.2</td></tr><tr><td>Active</td><td>Yes</td><td>80.3</td></tr></tbody></table>

**The relevance of the representative vectors extracted via the proposed model:** For this purpose, 5 classification layers were added to the network and were fed with a different set of representative vectors as input. Precisely, instead of receiving the 3 representative vectors, i.e., {$\hat{f}_1, \hat{f}_2, \hat{f}_3$}, the 5 layers only receive the following vector or combination of vectors: {$\hat{f}_1, \hat{f}_2$}, {$\hat{f}_2, \hat{f}_3$}, {$\hat{f}_1$}, {$\hat{f}_2$} and {$\hat{f}_3$}. Each additional layer
was trained by adding a cross-entropy term to the loss function. The sum is then divided by the total number of terms, which is 6. By doing so, the final new loss function is the average of the 6 cross-entropy terms. To prevent the supplementary layers from adjusting the feature vectors for their needs, we prevent their gradients from back-propagating to the features.

Similar to the previous ablation study, the model with additional layers was trained in high-resolution without distillation. The accuracy of each of the supplementary layers (and the main layer) is summarized in Table 5 on the CUB-200-2011 dataset. As shown, when using only one vector, performance decreases along with the ranking of the feature vectors (from 1 to 3). Similarly, when removing one of the vectors from the full set, the accuracy also drops. This observation proves indirectly that the contribution of each attention map to the task is relevant to the ranking of the feature. Note that the performance of the model using the active selection method with refining from the previous ablation study (80.3%) is different from the performance of the model using all three vectors {$\hat{f}_1, \hat{f}_2, \hat{f}_3$}. This is because, in the latter, the loss is an average of 6 terms, which means that the cross-entropy term of the model is multiplied by a 1/6 factor, which is equivalent to using a learning rate smaller by a factor of 1/6 compared to the previous experiment.

<table><caption>Table 5: Impact of number and rank of features on accuracy, using the CUB-200-2011 dataset.</caption><thead><tr><th>Vector(s)</th><th>{f&#x0302;<sub>1</sub>, f&#x0302;<sub>2</sub>, f&#x0302;<sub>3</sub>}</th><th>{f&#x0302;<sub>1</sub>, f&#x0302;<sub>2</sub>}</th><th>{f&#x0302;<sub>2</sub>, f&#x0302;<sub>3</sub>}</th><th>{f&#x0302;<sub>1</sub>}</th><th>{f&#x0302;<sub>2</sub>}</th><th>{f&#x0302;<sub>3</sub>}</th></tr></thead><tbody><tr><td>Acc.</td><td><strong>82.8</strong></td><td>82.7</td><td>81.4</td><td>81.2</td><td>80.0</td><td>66.3</td></tr></tbody></table>

### 4.4. Increasing the resolution of feature maps

The effects of using larger feature map resolution on both the performances of B-CNN [13] and a BR-NPA were explored. They were both trained on the CUB-200-2011 dataset. We compared two resolutions including 14 × 14 and 56 × 56. The first is the default output resolution and the second one was obtained by reducing the strides of layers 3 and 4 to 1. We used hyper-parameter optimisation and to train higher-resolution models, we employed distillation, as detailed in section 3.1.

Ideally, the attention map is expected to be sparse, i.e., focusing on a small local area of the image. To quantify this, we introduce a new metric, namely, sparsity. First, we averaged the multiple attention maps produced by the model. Then, to take the feature vector norm into account, each element of the map was multiplied by the corresponding feature vector euclidean norm. Finally, the sparsity is defined as $s = a_{max}/a_{mean}$, where $a_{max}$ is the maximum of the resulting map and $a_{mean}$ is its mean value. An attention map focusing on a small part of the image is supposed to have a low mean value which ends out with a high sparsity value. Conversely, a map covering all the input uniformly is supposed to have a maximum value that is close to its mean value. Therefore its sparsity is close to the minimum value, i.e., 1. We are therefore looking for models with a high sparsity value. Note that this metric is independent of the used resolution and the number of produced attention maps. Distilling
the higher-resolution model obtained via reducing strides helps to improve the sparsity of the student network with respect to the teacher network (11.2 → 19.6 for B-CNN, and 7.46 → 21.3 for BR-NPA). This shows that distillation combined with a reduction of strides helps to generate more precise, interpretable attentions maps and improve the accuracy of the model. Results depicted in Table 6 show that the higher-resolution models outperform their corresponding teacher network (+1.7 and +0.3 respectively for B-CNN and BR-NPA). For comparison, a CNN with default resolution (14 × 14) obtains an accuracy of 84.2%.

<table><caption>Table 6: Impact of resolution on performance using the CUB-200-2011 dataset</caption><thead><tr><th>Model</th><th>Map size</th><th>Accuracy</th><th>Sparsity</th></tr></thead><tbody><tr><td rowspan="2">B-CNN</td><td>14 × 14</td><td>82.9</td><td>11.2</td></tr><tr><td>56 × 56</td><td>84.6</td><td>19.6</td></tr><tr><td rowspan="2">BR-NPA</td><td>14 × 14</td><td>85.2</td><td>7.46</td></tr><tr><td>56 × 56</td><td><b>85.5</b></td><td><b>21.3</b></td></tr></tbody></table>

Additionally, we also evaluated the impact of increasing the resolution of the feature maps on the interpretability of the attention map qualitatively. We trained a student CNN to imitate a lower-resolution teacher CNN with distillation and visualize the activation maps for both the teacher and student CNN in Fig. 6. It could be observed that the attention maps from the higher-resolution CNN are more interpretable as they are sparser and the activated areas are located on the sub-parts of the object instead of the whole object.

<figure><img src="image_8.png" /><figcaption>Figure 6: From left to right: original image, attention maps of CNN with lower and then higher-resolution feature maps.</figcaption></figure>

## 4.5. Part discriminability hierarchy

Fig 7 shows visually whether there is a hierarchy i.e., ranking of importance level among the parts’ relevance that contributes to the decision-making of the task. For example, on the bird dataset, the first map (in red) focuses more on the head and the second map (green) focuses more on the body, indicating that the head is more discriminative than the body for the identification of the bird species.
Furthermore, the blue map is not visible on the bird dataset because it has focused on vectors with very low norms compared to ones selected by the red or green map. Low euclidean norm feature vectors have an ignorable impact on the final prediction of the task. In other words, these low-norm features are not relevant for the decision. The visualization on the aircraft dataset shows that the three maps are often visible and each can focus on various parts like tails, wheels, engines, etc. This indicates that, according to the model, it is necessary to focus on three parts as they are of equivalent importance level. Similarly, on the cars dataset, the first and second maps focus on the front air inlet and the lights, whereas the blue maps tend to target more other details like the flasher (column 2) or the car body details (column 1). It is shown that the two most important parts are the front air inlet and the lights and that some details like the body details are also but less discriminative. It is showcased that BR-NPA facilitates the discovery of the importance-level hierarchy among the object's parts according to their relevance.

<figure><img src="image_9.png" /><figcaption>Figure 7: Attention maps produced by BR-NPA.</figcaption></figure>

## 4.6. Attention metrics

To evaluate the reliability of the proposed attention mechanism, we employed the Deletion Area Under Curve (DAUC) and Integration Area Under Curve (IAUC) metrics proposed by Petsiuk et al. [9]. These metrics evaluate the reliability of the saliency maps by quantifying the contribution of an area of the image in the predicted class score regarding its importance according to the saliency map. More precisely, the DAUC metric progressively masks the whole input image and lets the model do an inference after each mask application. At each step, the variation of the initially predicted class score is measured. The intuition behind this is that if a saliency map highlights the areas that are actually relevant to the decision-making, masking them will result in decreasing the correct class score. Instead of progressively masking the image, the IAUC metric starts from a blurred image and then progressively unblurs it by starting from the most important areas according to the
saliency map. Similarly, if the area highlighted by the map are relevant for predicting the correct category, the score of the corresponding class (obtained using the unblurred image) is supposed to increase rapidly.

Mean values of DAUC and IAUC over 100 images are shown by table 7. It could be observed that BR-NPA yields better performance than all the other methods, including the SOTA interpretability models (e.g., the InterByParts and ProtoTree) and the SOTA visualization methods (e.g., VarGrad and SmoothGrad).

To further verify whether the performance gains are statistically significant, we computed the Welch's t-test between the performances of each pair of methods, and results are visualized in fig. 8. BR-NPA yields significantly better performance than most of the other methods. The first exception is RISE in terms of DAUC which can be explained by the large standard deviation it exhibits on this metric, as shown in the top left pane of fig. 8, probably preventing statistical significance, even if the score of BR-NPA is almost twice smaller than RISE's score. The second exception is ProtoTree and InterByParts in IAUC which can be explained by the large variance exhibited by all models on this metric and the fact that ProtoTree and InterByParts obtain relatively good performance compared to BR-NPA. However, RISE almost produce the worst performance on the IAUC metric and ProtoTree/InterByParts also perform poorly on the DAUC metrics. In general, all methods obtain different rankings according to a certain metric (DAUC or IAUC), except BR-NPA, which yields the best performance in both cases, highlighting the robustness of the attention maps explainability across several metrics.

<table><caption>Table 7: Evaluation of the attention methods reliability. For DAUC, lower is better; for AUC, higher is better. Each value is the mean over the same 100 randomly selected test images.</caption><thead><tr><th></th><th></th><th>DAUC</th><th>IAUC</th></tr></thead><tbody><tr><td rowspan="7">Post-Hoc Methods</td><td>GuidedBP</td><td>0.1406</td><td>0.20</td></tr><tr><td>VarGrad</td><td>0.0578</td><td>0.28</td></tr><tr><td>SmoothGrad</td><td>0.0394</td><td>0.30</td></tr><tr><td>AM</td><td>0.0362</td><td>0.22</td></tr><tr><td>Grad-CAM</td><td>0.0286</td><td>0.16</td></tr><tr><td>RISE</td><td>0.0279</td><td>0.18</td></tr><tr><td>Score-CAM</td><td>0.0207</td><td>0.27</td></tr><tr><td rowspan="5">Attention Architectures</td><td>Grad-CAM++</td><td>0.0161</td><td>0.21</td></tr><tr><td>ProtoPNet</td><td>0.2964</td><td>0.37</td></tr><tr><td>ProtoTree</td><td>0.2122</td><td>0.43</td></tr><tr><td>B-CNN</td><td>0.0208</td><td>0.30</td></tr><tr><td>InterByParts</td><td>0.0811</td><td>0.48</td></tr><tr><td></td><td>BR-NPA</td><td><strong>0.0155</strong></td><td><strong>0.49</strong></td></tr></tbody></table>
<figure><img src="image_10.png" /><figcaption>Figure 8: Evaluation of the methods' attention reliability. Top row: Mean DAUC and IAUC with standard deviation over 100 random test images. For DAUC, lower is better; for IAUC, higher is better. Bottom row: Mutual comparison of performance gain. Color indicates the difference in performance and size indicates statistical significance (with a *p*<sub>val</sub> < 0.05 criteria), where a large disk indicates statistical significance and a small disk indicates no significance. Models are listed from worst to best from top to bottom. At row *i* column *j* is shown the gain of method *i* relative to *j*.</figcaption></figure>
5. Conclusion

In this work, we proposed a non-parametric attention that can be easily integrated into an existing architecture to significantly improve its interpretability while maintaining its accuracy. Specifically, this method improves the usual neural attention mechanism in two ways. First, the feature maps resolution was increased using distillation to highlight finer details. Second, the neural layer was replaced by a well-designed non-parametric module such that the importance level of the parts composing the object can be visualized using color. Through quantitative and qualitative experiments, it is demonstrated that BR-NPA achieves more comprehensive and accurate visual explanations compared to SOTA attention models and visualizations methods across multiple tasks without compromising the classification accuracy. Furthermore, it is an easy plugin that could be applied to various problems like fine-grained classification, re-identification, and few-shot classification and produces saliency maps matching the task objective. Contrarily to the usual attention, it is easy to integrate it in a complex pretrained architecture without significant accuracy loss. In the future, we will run a subjective study to investigate the use of BR-NPA in understanding computer-assisted decisions.

References

[1] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, D. Batra, Grad-cam: Visual explanations from deep networks via gradient-based localization, in: 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 618–626. 2, 3, 5, 8

[2] D. Bahdanau, K. Cho, Y. Bengio, Neural machine translation by jointly learning to align and translate, arXiv preprint arXiv:1409.0473 (2014). 2, 3

[3] J. Bastings, K. Filippova, The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?, arXiv preprint arXiv:2010.05607 (2020). 2

[4] A. K. Mohankumar, P. Nema, S. Narasimhan, M. M. Khapra, B. V. Srinivasan, B. Ravindran, Towards transparent and explainable attention models, arXiv preprint arXiv:2004.14243 (2020). 2

[5] K. Zhou, Y. Yang, A. Cavallaro, T. Xiang, Omni-scale feature learning for person re-identification (2019). arXiv:1905.00953. 3, 4, 8

[6] B. Zhou, A. Khosla, À. Lapedriza, A. Oliva, A. Torralba, Learning deep features for discriminative localization, CoRR abs/1512.04150 (2015). arXiv:1512.04150. 3

[7] A. Chattopadhay, A. Sarkar, P. Howlader, V. N. Balasubramanian, Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks, in: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), 2018, pp. 839–847. doi:10.1109/WACV.2018.00097. 3, 8
[8] H. Wang, Z. Wang, M. Du, F. Yang, Z. Zhang, S. Ding, P. Mardziel, X. Hu, Score-cam: Score-weighted visual explanations for convolutional neural networks, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, 2020, pp. 24–25. 3, 8

[9] V. Petsiuk, A. Das, K. Saenko, Rise: Randomized input sampling for explanation of black-box models (2018). arXiv:1806.07421. 3, 8, 18

[10] J. T. Springenberg, A. Dosovitskiy, T. Brox, M. A. Riedmiller, Striving for simplicity: The all convolutional net, CoRR abs/1412.6806 (2015). 3, 8

[11] J. Adebayo, J. Gilmer, I. Goodfellow, B. Kim, Local explanation methods for deep neural networks lack sensitivity to parameter values (2018). arXiv:1810.03307. 3, 8

[12] D. Smilkov, N. Thorat, B. Kim, F. Viégas, M. Wattenberg, Smoothgrad: removing noise by adding noise (2017). arXiv:1706.03825. 3, 8

[13] T. Lin, A. RoyChowdhury, S. Maji, Bilinear cnn models for fine-grained visual recognition, in: 2015 IEEE International Conference on Computer Vision (ICCV), 2015, pp. 1449–1457. 4, 6, 16

[14] T. Hu, H. Qi, See better before looking closer: Weakly supervised data augmentation network for fine-grained visual classification, CoRR abs/1901.09891 (2019). arXiv:1901.09891. 4, 6

[15] C. Chen, O. Li, A. Barnett, J. Su, C. Rudin, This looks like that: deep learning for interpretable image recognition, in: NeurIPS, 2019. 4, 8, 11

[16] M. Nauta, R. van Bree, C. Seifert, Neural prototype trees for interpretable fine-grained image recognition (2021). arXiv:2012.02046. 4, 8, 11

[17] Z. Huang, Y. Li, Interpretable and accurate fine-grained recognition via region grouping (2020). arXiv: 2005.10411. 4, 11

[18] X. Liu, T. Xia, J. Wang, Y. Yang, F. Zhou, Y. Lin, Fully convolutional attention networks for fine-grained recognition, arXiv: Computer Vision and Pattern Recognition (2016). 4, 8, 11

[19] M. Jaderberg, K. Simonyan, A. Zisserman, k. kavukcuoglu, Spatial transformer networks, in: C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, R. Garnett (Eds.), Advances in Neural Information Processing Systems 28, Curran Associates, Inc., 2015, pp. 2017–2025. 4, 8, 11

[20] D. Wang, Z. Shen, J. Shao, W. Zhang, X. Xue, Z. Zhang, Multiple granularity descriptors for fine-grained categorization, in: 2015 IEEE International Conference on Computer Vision (ICCV), 2015, pp. 2399–2406. 4, 8, 11
[21] H. Zheng, J. Fu, T. Mei, J. Luo, Learning multi-attention convolutional neural network for fine-grained image recognition, in: 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 5219-5227. 4, 8, 11

[22] Z. Zhang, C. Lan, W. Zeng, X. Jin, Z. Chen, Relation-aware global attention for person re-identification, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 4, 8

[23] W. Li, X. Zhu, S. Gong, Harmonious attention network for person re-identification, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 4, 8

[24] G. Chen, J. Lu, M. Yang, J. Zhou, Learning recurrent 3d attention for video-based person re-identification, IEEE Transactions on Image Processing 29 (2020) 6963-6976. doi:10.1109/TIP.2020.2995272. 4

[25] J. Si, H. Zhang, C.-G. Li, J. Kuen, X. Kong, A. C. Kot, G. Wang, Dual attention matching network for context-aware feature sequence based person re-identification, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 4

[26] S. Yan, S. Zhang, X. He, A dual attention network with semantic embedding for few-shot learning, Proceedings of the AAAI Conference on Artificial Intelligence 33 (01) (2019) 9079-9086. doi:10.1609/aaai.v33i01.33019079. URL https://ojs.aaai.org/index.php/AAAI/article/view/4940 4

[27] R. Hou, H. Chang, B. Ma, S. Shan, X. Chen, Cross attention network for few-shot classification (2019). arXiv:1910.07677. 4, 8

[28] T. Gao, X. Han, Z. Liu, M. Sun, Hybrid attention-based prototypical networks for noisy few-shot relation classification, Proceedings of the AAAI Conference on Artificial Intelligence 33 (01) (2019) 6407-6414. doi:10.1609/aaai.v33i01.33016407. URL https://ojs.aaai.org/index.php/AAAI/article/view/4604 4

[29] Y. Zhu, C. Liu, S. Jiang, Multi-attention meta learning for few-shot fine-grained image recognition, in: C. Bessiere (Ed.), Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, International Joint Conferences on Artificial Intelligence Organization, 2020, pp. 1090-1096, main track. doi:10.24963/ijcai.2020/152. URL https://doi.org/10.24963/ijcai.2020/152 4
[30] H. Zhang, L. Zhang, X. Qi, H. Li, P. H. S. Torr, P. Koniusz, Few-shot action recognition with permutation-invariant attention, in: A. Vedaldi, H. Bischof, T. Brox, J.-M. Frahm (Eds.), Computer Vision – ECCV 2020, Springer International Publishing, Cham, 2020, pp. 525–542. 4

[31] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778. 5, 9

[32] Y. Peng, X. He, J. Zhao, Object-part attention model for fine-grained image classification, IEEE Transactions on Image Processing 27 (3) (2018) 1487–1500. doi:10.1109/tip.2017.2774041. URL http://dx.doi.org/10.1109/TIP.2017.2774041 5

[33] A. Dubey, O. Gupta, P. Guo, R. Raskar, R. Farrell, N. Naik, Pairwise confusion for fine-grained visual classification (2018). arXiv:1705.08016. 5

[34] G. Sun, H. Cholakkal, S. Khan, F. S. Khan, L. Shao, Fine-grained recognition: Accounting for subtle differences between similar classes (2019). arXiv:1912.06842. 5

[35] W. Ge, X. Lin, Y. Yu, Weakly supervised complementary parts models for fine-grained image classification from the bottom up, in: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 3029–3038. 5

[36] K. Sun, B. Xiao, D. Liu, J. Wang, Deep high-resolution representation learning for human pose estimation, CoRR abs/1902.09212 (2019). arXiv:1902.09212. 5

[37] G. Hinton, O. Vinyals, J. Dean, Distilling the knowledge in a neural network (2015). arXiv:1503.02531. 5

[38] C. Wah, S. Branson, P. Welinder, P. Perona, S. Belongie, The Caltech-UCSD Birds-200-2011 Dataset, Tech. Rep. CNS-TR-2011-001, California Institute of Technology (2011). 8, 10

[39] S. Maji, J. Kannala, E. Rahtu, M. Blaschko, A. Vedaldi, Fine-grained visual classification of aircraft, Tech. rep. (2013). arXiv:1306.5151. 8, 10

[40] J. Krause, M. Stark, J. Deng, L. Fei-Fei, 3d object representations for fine-grained categorization, in: 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013. 8, 10

[41] M. Ren, E. Triantafillou, S. Ravi, J. Snell, K. Swersky, J. B. Tenenbaum, H. Larochelle, R. S. Zemel, Meta-learning for semi-supervised few-shot classification (2018). arXiv:1803.00676. 8, 9
[42] L. Zheng, L. Shen, L. Tian, S. Wang, J. Wang, Q. Tian, Scalable person re-identification: A benchmark, in: 2015 IEEE International Conference on Computer Vision (ICCV), 2015, pp. 1116–1124. doi: 10.1109/ICCV.2015.133. 8, 10, 11

[43] T.-Y. Lin, A. Roy Chowdhury, S. Maji, Bilinear cnn models for fine-grained visual recognition, in: Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), ICCV '15, IEEE Computer Society, Washington, DC, USA, 2015, pp. 1449–1457. doi:10.1109/ICCV.2015.170. 8, 11

[44] Z. Huang, Y. Li, Interpretable and accurate fine-grained recognition via region grouping (2020). arXiv: 2005.10411. 8

[45] N. Kokhlikyan, V. Miglani, M. Martin, E. Wang, B. Alsallakh, J. Reynolds, A. Melnikov, N. Kliushkina, C. Araya, S. Yan, O. Reblitz-Richardson, Captum: A unified and generic model interpretability library for pytorch (2020). arXiv:2009.07896. 8

[46] T. Xiao, Y. Xu, K. Yang, J. Zhang, Y. Peng, Z. Zhang, The application of two-level attention models in deep convolutional neural network for fine-grained image classification (2015). 8, 11

[47] T. Akiba, S. Sano, T. Yanase, T. Ohta, M. Koyama, Optuna: A next-generation hyperparameter optimization framework (2019). arXiv:1907.10902. 9

[48] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, L. Fei-Fei, ImageNet Large Scale Visual Recognition Challenge, International Journal of Computer Vision (IJCV) 115 (3) (2015) 211–252. doi:10.1007/s11263-015-0816-y. 9

[49] Y. Liu, B. Schiele, Q. Sun, An ensemble of epoch-wise empirical bayes for few-shot learning, in: A. Vedaldi, H. Bischof, T. Brox, J.-M. Frahm (Eds.), Computer Vision – ECCV 2020, Springer International Publishing, Cham, 2020, pp. 404–421. 9

[50] Z. Zheng, X. Yang, Z. Yu, L. Zheng, Y. Yang, J. Kautz, Joint discriminative and generative learning for person re-identification, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 10

[51] L. Bertinetto, J. F. Henriques, P. Torr, A. Vedaldi, Meta-learning with differentiable closed-form solvers, in: International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=HyxnZhOct7 13
6. Acknowledgments

This study was partly supported by NExT (ANR-16-IDEX-0007). The authors have no conflict of interests to declare.