A$^2$-Net: Learning Attribute-Aware Hash Codes for Large-Scale Fine-Grained Image Retrieval
Authors: Xiu-Shen Wei
Yang Shen
Xuhao Sun
Han-Jia Ye
Jian Yang
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Fine-Grained Image Retrieval
Large-Scale Data
Attribute-Aware
Learning to Hash

Abstract: Our work focuses on tackling large-scale fine-grained image retrieval as ranking the images depicting the concept of interests (i.e., the same sub-category labels) highest based on the fine-grained details in the query. It is desirable to alleviate the challenges of both fine-grained nature of small inter-class variations with large intra-class variations and explosive growth of fine-grained data for such a practical task. In this paper, we propose an Attribute-Aware hashing Network (A$^2$-Net) for generating attribute-aware hash codes to not only make the retrieval process efficient, but also establish explicit correspondences between hash codes and visual attributes. Specifically, based on the captured visual representations by attention, we develop an encoder-decoder structure network of a reconstruction task to unsupervisedly distill high-level attribute-specific vectors from the appearance-specific visual representations without attribute annotations. A$^2$-Net is also equipped with a feature decorrelation constraint upon these attribute vectors to enhance their representation abilities. Finally, the required hash codes are generated by the attribute vectors driven by preserving original similarities. Qualitative experiments on five benchmark fine-grained datasets show our superiority over competing methods. More importantly, quantitative results demonstrate the obtained hash codes can strongly correspond to certain kinds of crucial properties of fine-grained objects.

A A2-NET: Learning Attribute-Aaaee Hash Codes for Large-Scale Fine-Grained Image Retrieval
Xiu-Shen Wei! Yang Shen', Xuhao Sun', Kun', Ye?, Jian Yang' NNajjing University of Science and Technology State Key Lab. for Novel Software Technology, Nanjing University
Abstract
Our work focuses on tackling large-scale fine-grained image retrieval ranking the images depicting the concept of interests (i.e., the same sub-category labels) highest based on the fine- grei-grridd details in the query. the challenges of both fine-grained nature of small inter-class variations with large intra-class variations and explosive growth of ume-grained data for such practical task. In this paper, we propose an Attribute- Aware hashing Network (A² (AT- for generating attribute-aware hash codes not only make the retrieval process efficient, but also establish explicit correspondences between hash codes and visual attributes. Specifically, based on the captured visual representations by attention, we develop encoder-decoder structure network of reconstruction task to unsu- pervisedly distill high-level attribute-specife vectors from the appearance-specific visual representations without attribute annotations. 2 is also equipped with a feature decorrelation constraint upon these attribute vectors to enhance their rep- resentation abilities. Finally, the required hash codes are generated by the attribute vectors driven by preserving original similarities. Qualitative experiments on five benchmark fine-grained datasets show superiority over competing methods. More importantly, quantitative results demonstrate the obtained hash codes can strongly correspond to certain kinds of crucial properties of fine one-grained objects.
desirable alleviate
Introduction
Fine-grained image retrieval in computer vision aims to retrieve images belonging to multiple subordi- nate categories of a super-category (aka meta-category), e.g. different species of animals/plants (36),
different models of cars 200) different kinds of retail products [39] etc. Its key challenge therefore lies with understanding fine-grained visual differences that sufficiently discriminate between objects that are highly similar in overall appearance, but differ in wiff--ggaiine features. Also, fine- fine-grained retrieval still demands ranking all the instances so that images depicting the concept of interest (e.g., the same sub-category label) are ranked highest based on the fine-grat fine-grained details in the query.
In particular, with the explosive growth of fine-grained data in real applications [ 14 26 36 39] fine-grained hashing,
promising solution for dealing with large-scale fine-grained retrieval tasks, has proven to be able to greatly reduce the storage cost and increase the query speed [8] (8) benefiting from the learned compact binary hash code representations. However, although previous works,
"Corresponding author. X-S. Wei, Y. Shen, X. Sun and J. Yang are with PCA Lab, Key Lab of Intelligent Perception and Systems for High- Diinh--iiniiional Information of Ministry of Education, and Jiangsu Key Lab of Image and Video Understanding for Social Security, School Computer Science and Engineering, Nanjing University of Seience and Technology. This work was supported by Natural Science Foundation of Jiangsu Province of China under Grant 1B320200340) the Fundamental Research Funds for the Central Universities (No. 309200411 11 I), and CAAl-Huawei MindSpore Open Fund (CAAIXSILU--220-022A))) The first two authors contributed equally to this work.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Rod,
NO
Bivee tall?
On the tree?
Spotte body?
Attribute-waate bash codes of Ig
Query image Z
A--NET
Top-k retrieved images
Figure 1: Key idea of our AE-,ET, as well as the main process of fine-grained hashing based on our attribute-aware hash codes. In concretely, regarding query image Zg of Red bellied Woodpecker, after returning all the correct results, fine-gra fine-grained image belonging to Red headed Woodpecker closest to the query image terms of Hamming distance also retrieved.
egg., 1 [8]. achieved good retrieval performance, the bits of their hash codes correspond to no semantics, i.., fine-grained attributes. While, such attributes, c.g., head color, tail color, male, female, living habits, are great means of describing fine-grained objects, in a way both humans and computers understand. In this paper, establish explicit correspondence between hash codes and visual attributes for not only further improving large-scale fine-grained retrieval accuracy, but more importantly integrating interpretation into deep learning based hash methods, propose unified Attribute- Aware hashing Network, termed as AE-NET (cf. Figure L. for achieving these goals.
In our A -EET, considering huge labor cost of supervised attribute annotations, we restrict ourselves in an unsupervised setting to automatically capture discriminative visual attributes from still images and then correspond the final learned hash code representations these attributes. Therefore, hash bit of learned hash codes could be both discriminative and intuitive. Additionally, thanks to the unsupervised setting, the attributes derived from A NTN will be not restricted pre-defined attributes like supervised- based attribute learning methods [5 21 455. Moreover, it can distill the most useful properties of fine-graines objects attribute- aware hash codes in such end-to-end trainable manner for accuracy retrieval among multiple similar subordinate categories.
More specifically, as the overall framework shown in Figure 2 our A consists of a fine- grained representation learning module and an attribute-awar hash codes generating module. first leverages attention mechanisms to model fine-grained tailored patterns in terms of both global-level deep features T, and local-level cues T. from input image I. Then, the appearance-speciinn features of these visual patterns T are aggregated and translated into semantic-specific representations ø;, After that, we formulate the aforementioned unsupervised attribute learning as a reconstruction task of projecting Therefore, it can be expected that in the high-level attribute space, U, could correspond to certain kinds of nameable properties of fine-gres fine-grained objects. Moreover, feature decorrelation constraint is further introduced upon v; to both enhance the discriminative ability and remove the redundant correlation among these dimensions of attribute-specific features. Finally, our attribute-aware hash codes u, are generated from v2 by conducting the hash code learning procedure.
attribute vector v; by performing encoder-decoder structure network.
To evaluate model, conduct extensive experiments using five benchmark fine- grained retrieval datasets for both accuracy and interpretability. Quantitative results of retrieval accuracy on these datasets show that the proposed A-- model obviously and consistently outperforms existing state- of-the-art methods. Qualitative visualization of the obtained attribute-aware hash codes demonstrates that these hash bits have strong correspondences visual attributes of fine- emn-nniieed objects (cf. Figure 4 even without employing attribute supervisions or part-level annotations, In addition, the ablation studies of these crucial components in 22 also validate their own effectiveness.
Related Work
2.1 Fine-Grained Image Retrieval
Fine- line-grained image retrieval as an integral part of fine-grained image analysis 41 gained more and more traction in recent years [8]
255 29
40
43
47 48). What makes it challenging is that

---

Feature decorrefation
lass (Section 2.32)
Allentio guidance
Upeepeeet attrbutt guided lown (Section 3.50)
Encoder W
Decoder W
2
I
learning
+(-)
T
22g
loss (Section 2.33)
Appropatio
Attribute-aware hash codes generating
Fine-grained representation learning
Figure 2:
Overall framework of the proposed A--NET model, which consists of crucial modules, ie.. fine-grained representation learning and attribute-aware hash codes generating. The whole network can be end-to-end trainable, and is generally driven by the unsupervised attribute-guided reconstruction loss, the feature decorrelation loss and the hash code learning loss, cf. Section 3.3
objects of fine-grained images have only subtle differences, and often largely vary in pose, scale, and orientation or can exhibit cross-modal differences (e.g., sketch-based retrieval (25)).
Depending the type of query image. the most studied areas of fine-grained image retrieval can be separated into two groups: fine-grained content-based image retrieval (FG-CBIR) and fine- grained sketch-based image retrieval (FG-SBIR). More specifically, in FG-CBIR, unsupervised learning based [40] and supervised learning based methods [44] 47 488 were developed from different perspectives for handling fine- mne-grained retrieval tasks, e.g., localizing fine-grained parts (40), enhancing intra-class separability with inter-class compactness [48), and reducing the confidence of the fine-grained predictions [44), etc. While, FG-SBIR needs to not only capture fine-grained characteristics present the sketches, but also possess the ability to traverse the sketch and image domain gap. In the literature FG-SBIR, the earlier works, e.g., [23 43 46). were mostly based on Siamese-triplet networks [3] to tackle the aforementioned challenges. Recently, some works
tried incorporate the advances recent progress in self-supervised learning 6 and attention
mechanisms 7 for further improving the retrieval accuracy of FG-SBIR, e.g., 22 32)
However, although these fine- enn-grained retrieval methods achieved good results, they still have the limitations in the face of large-scale data, i.e., the searching time for exact nearest neighbor is typically expensive
impossible for the given queries. To alleviate this issue, fine-grained hashing, which aims to generate compact binary codes to represent fine-grained images, as a promising direction has attracted the attention in the fine-grained community very recently [8] [9]. More specifically,
ExchNet [8] was the first to define the fine-gra tine-griined hashing task and develop fine-grained tailored method to firstly locate discriminative object parts and further learn binary hash codes for representing fine-grained images. In the same period, DSaH [19] was proposed to automatically mine salient regions and learn pemanti-- hash codes simultaneously. Unfortunately, the learned hash bits of these methods lack any semantics which are more meaningful to fine fine-grained objects, and thus lack the model interpretability. Compared with them, our proposed A NET can not only outperform the previous fine- grained hashing methods, but more importantly, the learned hash codes of A -EET are uttrbbut-aware i.e., the hash bits of A NET have strong correspondence to semantic visual properties that are useful for fine-grained image retrieval.
2.2 Learning to Hash
Hashing is a widely-studied solution to approximate nearest neighbor search, which transforms the data item to a short code consisting of sequence of bits (i.e., hash codes). The research efforts of hashing can categorized into two groups, including data-independent hashing (aka locality
sensitive hashing 9 27 34) and data-dependent hashing (aka learning to hash 44 12 17 33)).

---

Specifically, locality sensitive hashing methods attempted to adjust hash learning from different perspectives, e.g., the theory or machine learning views, name few: proposing random hash functions satisfying local sensitive property [9) developing better search schemes [27], providing faster computation of hash functions [4], etc. While, compared with locality sensitive hashing methods, since data-dependent hashing methods learn hash functions from specific dataset to achieve similarity preserving, they can generally obtain superior retrieval accuracy. Especially for capitalizing advances in deep learning, many well-performing methods were proposed to integrate feature learning and hash code learning into an end-to-end framework based on deep networks, e.g., [77. In particular, very recently researchers in the vision community have begun to pay attention to the more challenging and practical hashing task, i.e., fine- teme-grained hashing [8 19]. To the best of our knowledge, this is the first work to equip these learned hash codes with strong correspondence to visual attributes for dealing with large-scale fine-graing image retrieval.
2.3 Visual Attributes
Attributes typically mid-level semantic properties of objects [10], such as colors (e.g.. "red", "blue"), texture (e.g. "striped"", "spotted"), or even life habits of animals (e.g. "living on the tree". "living in the water"). Visual attributes have exhibited their impact for strengthening various vision tasks, including facial verification [21], fine-grained categorization 45 zero-shot transfer [42], scene understanding (30), and so on. Most of the previous attribute learning methods are supervised by costly human-generatee
annotations and also are dependent on pre-defined attribute labels, e.g., (15 42 45). In consequence, for large-scale problems, these supervised methods might be not feasible due to the restriction caused by the cumbersomely obtained attribute annotations. Moreover, even for some tasks, their visual attributes are quite hard to define. In this paper, to alleviate the aforementioned issues, we propose an A -EE model to formulate an unsupervised learning structure to project the learned visual features into an attribute space where it finally generates attribute-an aware binary hash codes. Compared with previous attribute learning methods, our A is independent with pre-defined attribute labels, and could automatically learn discriminative attribute-aware hash codes in unified end-to-end trainable fashion. Furthermore, our method can not only correspond hash bits to visual attributes tailored for fine-grain areeggraied objects, which shows significant improvements of retrieval accuracy, but also offer intuitive way of deep hashing interpretation.
3 Methodology
In this section, we introduce the overall framework and notations of the proposed 2 model, as well as elaborating the key modules of 2 and its corresponding optimization algorithm.
3.1 Overall Framework and Notations
As illustrated in Figure 2 our 2-EETT model consists of two crucial modules, i.e,, a fine-grained representation learning module and an attribute-aware hash codes generating module. Given an input image Z, based on its corresponding deep activation tensor T; E RCC extracted by backbone CNN, a set of attention guidance A {A') learned for capturing fine-grained tailored local patterns T from T. To distill semantical cues and further generate the final attribute-wware binary hash codes, we propose to transform these appearance-specifii features towards semantic- specific representations T by performing a transform network o(-). After aggregating T, the obtained attentive local-level features σς are associated with the global-level feature 2. global to form as holistic feature representation E;. In order to generate attribute- aware binary hash codes, we conduct a reconstructing paradigm to project 2; as v; in an attribute space where its data point corresponds to an attribute vector w.r.t. certain kind of nameable properties of fine-grained objects (e.g., "red head" or "spotted body"). Furthermore, with the aid of feature decorrelation, v; is expected to be more discriminative by removing redundant correlation information. Finally, hash code learning is performed upon v; to obtain the final attribute- binary codes u.
3.2 Fine-Grained Representation Learning
Attention plays an important role human perception 7 16
and humans exploit a sequence of partial glimpses and selectively focus on salient parts of an object or a scene in order to better capture

---

visual structure [22] Inspired by this, we incorporate the attention mechanism into representation learning to capture fine- tine-griined local patterns for distinguishing subtle differences between these subordinate categories.
In concretely, extract the deep feature of its input image I via backbone CNN model @CNN by
(-)
T; = PCNN (I:) E RRCHX
(1)
Then, based on T, C attention guidance A " is generated as set of attention maps, i.e. A. The attention guidance A is designed to evaluate which deep descriptors [40] in these H x W cells should be attended
overlooked by conducting
T A
(2)
where @ is the element-wise Hadamard product. To obtain the final attribute- binary codes, it is desirable to transform these appearance-specine (i.e., low-level) features T to semantic-specifii (i.e. mid-level) representations which are closer to the attribute space. Thus, transforming network Φ(-). which is equipped with a stack of convolution layers, is performed on T as follows:
T - (TT; Ohocal)
F. (T;;; Oglobal)
(4)
where presents the parameters of the corresponding transforming networks w.r.t. T. and Tú respec- tively. Then, we aggregate T and T. by conducting global average- eoorage-poooing... and correspondingly obtain the attentive local-level features æi; well as the global-level feature globbal, The holistic feature representation w.r.t. the input image I, is achieved by concatenating both æf and a aoflohal i.e.,
2;
[I; Q) Rd. Note that, hereby abstract the aforementioned fine-grained feature learning process function F(Z;; θ) associated with its parameters O.
f global
3.3 Attribute-Aware Hash Codes Generating
How to generate attribute-aware hash codes is the key of our -ET model. We elaborate it in the following three aspects, i.e., unsupervised attribute-guided learning, attribute-specific feature decorrelation, and hash code learning.
3.3.1 Unsupervised Attribute-Guider Learning
In ceal-applications. especially for the large-scale and fine-graing reee-graiedd tasks, attribute annotations are always infcasible, which limits the learning process to be conducted in unsupervised setting. While, in the literature, the main goal unsupervised learning is to capture regularities in data for the purpose of extracting useful representations for restoring corrupted data (31). Many unsupervised methods explicitly produce internal latent units or codes, from which the data is to be reconstructed.
Inspired by this, we develop unsupervised attribute-guided reconstruction component to project the holistic representation æ; of Z, into a latent space, i.e., the attribute space V. V, its high-level vectors are designed to have certain desirable properties, e.g. corresponding to semantic properties of fine-grained objects (aka "fine-grained attributes").
More specifically, in our A -EET. the unsupervised suribute-guidde learning is realized by a recon- struction paradigm with an encoder-decoder structure, as shown in Figure 2 In coneretely, given a batch of n training data I, their holistic representations X {œ1; 22; : :ı,,) } E Rdxa can be obtained as aforementioned. By formulation, the encoder projects X into the attribute space V with projection matrix W E R to get an internal latent representation V E k w.r.t. X. In particularly, we set that the dimension of latent representation i equals the number of hash bits the final binary hash code Furthermore, each column of V, i.e., E R can derive u, by
U4 sangggttniig (tanh(w:)).
Meanwhile, regarding u,, we also consider to reconstruct its input æ; by decoder as a counterpart of the encoder. Therefore, on one hand, such a reconstruction paradigm can reduce and further distill high-level semantic cues in the attribute space V. While, on the other hand, it can drive the training
5

---

of -ENT by preserving the similarity between queried hash codes and database points in terms of hash code learning (cf. Section 3.3.3)
In concretely, the learning objective of unsupervised attribute-guided reconstruction is written as follows:
min X
W WX s.t. WX V =annh(V),
where the decoder (i.e, a counterpart of the encoder) realized by W simplify the network. However, directly minimizing Eq. (6) with hard constraint is difficult to optimize. Therefore, we relax the constraint into soft constraint, and then the learning objective can be rewritten as
min X - W I + X|WX -VII
3.3.2 Attribute-Specifie Feature Decorrelation
By conducting the aforementioned unsupervised perrbbteeguidee learning, obtain the internal latent vectors V' as the attribute-specific features. In order to both enhance the discriminative ability and remove the redundant correlation among these attribute-specific features, introduce feature decorrelation constraint upon V', which is formulated by
min VV n
where is the identity matrix and n is the batch size. Such a feature decorrelation constraint is preferable construct independent features and reduce redundant information. Therefore, based on both unsupervised attribute-guided reconstruction and attribute-specific feature decorrelation, the final learned hash codes are expected be both attribute-aware and hash-bit independent.
3.3.3 Hash Code Learning
In the following, conduct the hash code learning based on the obtained attribute- features. Assume that we have n query data points which are denoted as {q: ] as well as m database points which are denoted { Note that, both qi and v; belonging to the attribute space V. By following Eq. (5), the corresponding binary hash codes can be obtained via
sgn(tanh(q:)) 2j sgn (tanh(w,)),
(9)
(10)
where u, 2j E (-1,+113* The goal of our hash code learning is to learn binary hash codes for both query points and database points from (=.) {u,))= and the pairwise supervised information, i.e. S E +11++11mxx0 supervised information (aka similarity) and the inner product of databad point binary code pairs. can be formulated as follows:
To preserve the pairwise similarity, we adopt the lz loss between the
miny, 2 Σ EØ (u. 2, - k:S,,))
s.t. U ( Z E {-1, +1]m*
(11)
Overall, we get the final objective of the proposed 2 model by considering Eq. together follows:
7 Eq.
8 and
Eq. 11
min C(T) = X - W + - 1
u 2j
-kSky)
1=1
(12)
where X, a and B are Sypeer-parameter as the trade-off.
In practice, during training, it might be only available set of database points {uy} without query points. Thus, we randomly sample n data points from database to construct query set, and denote the indices of all the database points as I with the indices of the query set as Q. Additionally, because we cannot back pack-propagate the gradient θ due the sgn(-) function, we omit the sgn() (-) function and only apply tanh(-) for relaxation in Eq. (10 of the whole optimization process. Therefore, the

---

optimization formulation of A -EET can be rewritten with only database points {uy] for training
as:
min [(I) = | W v' > W.O
v'Il + V -nI
+ ΣΣ EE JET
Lanh(W O)); 23 -kii)
(13)
For optimization, our A -ET does not require complicated two-stage learning algorithms, e.g., the alternative optimization strategy. In experiments, we employ the proppprppagttion algorithm and follow 17 to train the whole -NE model in a unified end-to-end manner.
3.4 Sut-of-Sample Extension
After training A -ET, the learned model can be applied for generating binary codes for query points including unseen query points in the training phase. Specifically, we can use the following equation to generate the binary code for Iz:
sgn(tanh(W (((
(14)
Experiments
4.1 Datasets
By following ExchNet [8. our experiments are conducted on five fine-grained benchmark datasets, ie.. CUB200-2011 (37]. Aircraft (28). Food101 [2] NABirds (35] and VegFru [14). Specifically, CUB200-2011 is one of the most popular used fine- ane-grained datasets. It contains 11,788 bird images from 200 bird species and is officially split into 5,994 images for training and 5,794 images for test. Aircraft contains 10,000 images spanning 100 aircraft models with 3,334 for training, 3,333 for validation and 3,333 for test. For large-scale datasets, Food/0/ contains 101 kinds of foods with 101,000 images, where for each class, 250 test images are checked manually for correctness while 750 training images still contain certain amount of noises. NABirds is high quality dataset which has 48,562 images of North American birds with 555 sub-categories, where 23,929 for training with 24,633 for test. VegFru is another large-scale fine-grained dataset covering 200 kinds of vegetables and 92 kinds of fruits with 29,200 for training, 14,600 for validation and 11,993 for test.
4.2 Baselines and Implementation Details
Baselines In experiments, we compare our proposed model to the following competitive baselines, ie., ITQ [I], SDH (33), DPSH (24), HashNet (5), and ADSH [17.. Among them, DPSH, HashNet and ADSH are deep learning based methods, while ITQ and SDH are not. Furthermore, we also compare the results of our A 2-NET with state-of-the-arts of fine-grained hashing methods, including ExchNet R Additionally, another fine-graines hashing method, i.e., DSaH (19), also achieved good retrieval accuracy. However, due to its empirical settings quite distant from other existing fine-graing grained hashing methods, for fair comparisons, strictly control empirical settings as the same those of 19 and compare the results of our 2 with its results in the supplementary materials.
Implementation Details For fair comparisons, we follow the efficient training setting in Exch- Net [8]. In concretely, for CUB200-201 Aireraft and Food101, we sample 2,000 images per epoch, while 4,000 samples are randomly selected for NABirds and VegFru, For the training details, regarding the backbone model, we can choose any network structure as the base network for the fine-grained representation learning module. While, by following [) ResNet-50 [K is employed in experiments. The total number of training epochs is 20, and the number of batch size is set as 16. While, different from ExchNet, our model only requires smaller iteration time until convergence. Specifically, for these datasets containing less than 20,000 training images, the itcration time Tmax 60, and the learning rate is divided by 10 at the 50th iteration. For other datasets. Tmax is set as 70, and the learning rate is divided by 10 at the thth iteration. The wypee-parammters i.e., X, a and /3 in Eq. (13) k respectively. The number of attention guidance equals the number of hash bits. The optimizer standard mini-batch stochastic gradient descent with the weight decay as 10-4 All experiments are conducted with GeForce RTX 2080 Ti GPU.
nxc set as 1, 1 and 12
7

---

Table 1: Comparisons of retrieval accuracy (% mAP) on five line-grained benchmark datasets.
Datasets
# bits
ITQ
SDH
DPSH
HashNet
ADSH
ExchNet
Ours
12
6,80
10.52
8.68
12.03
20.03
25.
33.83
24
9,42
16.95
12.51
17.77
50.33
58.98
61.01
CUB200-2011
32
11.19
20.43
12.74
19.93
61.68
67.74
71.61
48
12.45
22.23
15.58
22.13
65.43
71.05
77.33
12
4.38
4.89
8.74
14.91
15.54
33.27
40.00
5.28
6.36
10.87
17.75
23.09
45.83
63.66
Aircraft
32
5.82
6.90
13.54
19.42
30.37
51.83
72.51
48
6,05
7.65
13.94
20.32
50.65
59.05
81.37
12
6,46
10.21
11.82
24.42
35.64
45.63
46.44
24
8.20
11.44
13.05
34.48
40.93
55.48
66.87
Food101
9.70
13.36
16.41
35.90
42.89
56.39
74.27
10.07
15.55
20.06
39.65
48.81
64.19
82.13
2.53
3.10
2.17
2.34
2.53
5.22
8.20
24
4.22
6.72
4.08
3.29
8.23
15.69
19.15
NABirds
32
5.38
8.86
3.61
4.52
14.71
21.94
24.41
48
6.10
10.38
3.20
4.97
25.34
34.81
35.64
12
3,05
5.92
6.33
3.70
8.24
23.55
25.52
5,51
11.55
9.05
6.24
24.90
35.93
44.73
VegFru
32
7.48
14.55
10.28
7.83
36.53
48.27
52.75
48
8.74
16.45
9.11
10.29
55.15
69.30
69.77
Query
image
Top-10 retrieved images
Figure Examples of top-10 retrieved images on CUB200-2011 of 48-bit hash codes by our ---NET.
4.3
Main Results
Table presents the average precision (mAP) results of fine-grained retrieval these five aforementioned fine-grained benchmark datasets. For each dataset, we report the results of four lengths of hash bits, i.e., 12, 24, 32, and 48, for evaluations. As shown in that table, our proposed 2 NET model significantly and consistently outperforms the other baseline methods on these datasets. In partieular, compared with the state-of-the-arr method ExchNet { our 2--NET achieves 17.83% and 17.88% improvements over ExchNet of 24-bit and 32-bit experiments on Aircraft and Food- 101, respectively. Moreover, 2 -ET also obtains superior results with an absolute value of about 80% mAP on CUB200-200 202200-201,, Aircraft and Food10/ with 48 bit hash codes. These observations validate the effectiveness of the proposed A -ET model, as well as promising practicality in applicapitssinns of fine-grained retrieval. Additionally, in Figure 3 we illustrate several retrieval results on CUB200- 2011, which shows that A -EN can retrieve well among multiple subordinate categories when the same species of birds with diverse variations appear in different kinds of background. Also, there also exist several failure cases, where quite tiny differences (e.g., caused by different views) between the query image and the returned images are demanded by carefully observations.
4.4 Ablation Studies
In this section, we demonstrate the effectiveness of these crucial components of the proposed 2 model, i.e., the attention-base based fine-graincl representation learning component 3.2 the unsupervised

---

Table 2: Retrieval accuracy (% mAP) with incremental components of the proposed A2-NET model.
CUB200-201
Food101
Configurations
12 bits
24 bits
32 bits
48 bits
12 bits
24 bits
32 bits
48 bits
Vanilla backbone
20.03
50.33
61.68
65.43
35.64
40.93
42.89
48.81
+ Allention (Sec. 3.2
27.42
58.17
68.24
76.10
41.33
65.07
70.06
78.51
Reconstruction (Sec. 3
33.31
60.65
71.28
77.10
45.02
67.49
73.57
81.63
+ Feature decorrelation (Sec. 3.3.2
33.83
61.01
71.61
77.33
46.44
66.87
74.27
82.13
(a) CUB200-200
(b) Aircraft
(c) Food101
(d) NABirds
(c) VegFru
Figure 4: Quality demonstrations of the learned attribute-aware hash codes by the proposed -2- model. Each column in each sub-figure can strongly correspond to certain kind properties the fine-grane fine-grained objects, e.g., "yellow birds in the forest", "double-wing winged aircrafts", "noodle-like food". "Bromeliaceae fruits", etc. (Best viewed in color and zoomed in.)
attribute-guides reconstruction component (cf. Section 3.3. and the attribute-specifie feature decorrelation component (cf. Section 3.3.2). In ablation studies, we apply these components incrementally on vanilla backbone (i.e., ResNet-50) as the baseline. As evaluated in Table by stacking these two components one by one, the retrieval results are steadily improved, which justifies the effectiieness of our proposed components in -E-N
Qualitative Analyses of Attribute-Aware Hash Codes
We hereby discuss the quality of the learned attribute-aware hash codes u, of 22-NET. After obtaining u, visualize fine- one-grained images retrieved by random single hash bit of u, to demonstrate the strong correspondence between visual attributes and the obtained hash bits. All the five datasets in experiments are used as cxamples to illustrate the quality. As observed in Figure 4 images of each column have some similar fine-grained object properties, iee., visual attributes. Indeed, the learned which could provide explanation of the -
hash codes are apparently attribute-awer success in fine-grate nne-graied retrieval. Meanwhile, it also offers numan-understandaale interpretation for such deep learning based fine-graine fine-grained hashing method.
Conclusion
In this paper, we proposed Attribute-Aware hashing Network, i.e. A for dealing with the large-scale fine-grained image retrieval task. Particularly, 2 was designed as expected be efficient, effective and more importantly interpretable. In concretely, by developing an unsu- pervised attribute-guided reconstruction method based on the obtained appearance-speccffe visual representation with attention, can distill attribute-specific vectors high-level attribute space. After further performing feature decorrelation upon attribute-specifi vectors, their discriminative ability is strengthened for representing fine- fine-graine seeigra object. Then, hash codes can be generated from these attribute-spedt speciffc vectors and thus became attribute-an aware, Both qualitative and quantitative our AE-NET. Additionally, visual attributes have
experiments demonstrate the effectiveness been shown to be useful in describing both known and unknown entities, which motivates us to study identifying unobserved sub-categories, i.e., zero-shot fine-grained recognition, based on our attribute-aware hash codes the future work.

---

References
[1] Thomas Berg, Jiongxin Liu, Seung Woo Lee, Michelle L. Alexander, David W. Jacobs, and Peter N. Belhumeur. Birdsnap: Large-scale fine-grained visual categorization of birds. In Proc.
IEEE Conf. Comp. Vis. Patr. Recogn., pages 2019-2024 2014.
[2] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative components with random forests. In Proc. Eur. Conf. Comp. Vis., pages 446-461, 2014.
[3] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using a "siamese" time delay neural network. In Proc. Advances in Neural Inf. Process. Syst., pages 737-744. 1993.
[4] Fatih Cakir, Kun He, Sarah Adel Bargal, and Stan Sclaroff. Hashing with mutual information.
IEEE Trans. Pattern Anal. Much. Intell., 41(10))2424-2437, 2019.
[5] Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Philip S. Yu. HashNet: Deep learning to hash by continuation. In Proc. IEEE Int. Conf. Comp. Vis., pages 5608-5617, 2017. [6] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proc. Inl. Conf. Mach. Leurn., pages
1597-1607, 2020.
[7] Maurizio Corbetta and Gordon L. Shulman. Control of goal-directed and stimulus-driven attention in the brain. Nature Reviews Neuroscience, 3:201-215, 2002.
[8] Quan Cui, Qing- Yuan Jiang, Xiu-Shen Wei, Wu-Jun Li, and Osamu Yoshie. ExchNet: A unified hashing network for large-scale fine-grained image retrieval. In Proc. Eur. Conf. Comp. Vis., pages 189-205, 2020.
[9] Anirban Dasgupta, Ravi Kumar, and Tamas Sarlos. Fast locality-sensitive hashing. In Proc.
ACM SIGKDD Int. Conf. Knowledge discovery & data mining, pages 1073-1081, 2011. [10] Vittorio Ferrari and Andrew Zisserman. Learning visual attributes. In Proc. Advances in Neural
Inf. Process. Syst., pages 433-440, 2007.
[11] Yunchao Gong. Svetlana Lazebnik, Albert Gordo, and Florent Perronnin. Iterative quantization:
A procrustean approach to learning binary codes for large-scale image retrieval. IEEE Trans. Pattern Anal. Mach. Intell., (5(11):2996-2929 2012.
[12] Kun He, Fatih Cakir, Sarah Adel Bargal, and Stan Sclaroff. Hashing as tie-aware learning to rank. In Proc. IEEE Conf. Comp. Vis. Patr. Recogn., pages
2.18.
[13] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. IEEE Conf. Comp. Vis. Part. Recogn., pages 770-778, 2016.
[14] Saihui Hou, Yushan Feng. and Zilei Wang. VegFru: A domain-specific dataset for fine-grained
visual categorization. In Proc. IEEE Inf. Conf. Comp. Vis., pages 541-549, 2017.
[15] Junshi Huang, Rogerio Feris, Qiang Chen, and Shuicheng Yan. Cross-domain image retrieval with dual attribute- ranking network. In Proc. IEEE Int. Conf. Comp. Vis., pages 1062-1070, 2015.
[16] Laurent Itti, Christof Koch, and Ernst Niebur. A model of saliency- visual attention for rapid scene analysis. JEEE Trans. Pattern Anal. Mach. Intell., 20 (0(11):1254 1259, 1998.
[17] Qing- Yuan Jiang and Wu-Jun Li. Asymmetric deep supervised hashing. In Proc. Conf. AAAI,
pages 3342-3349, 2018.
[18] Sheng Jin, Hongxun Yao, Xiaoshuai Sun, Shangchen Zhou, Lei Zhang, and Xiansheng Hua.
Deep saliency hashing for fine-grained retrieval. IEEE Trans. Image Process., 29:5336-5351, 2020.
[19] Sheng Jin, Hongxun Yao, Xiaoshuai Sun, Shangchen Zhou, Lei Zhang, and Xiansheng Hua. Deep saliency hashing for fine-grat fine-grained retrieval. IEEE Trans. Image Process., 29:5336-5355 2020.
[20] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D object representations for fineggrined categorization. In ICCV Workshop on 3D Representation and Recognition, 2013,
[21] Neeraj Kumar, Alexander Berg, Peter N. Belhumeur, and Shree Nayar. Describable visual attributes for face verification and image search. IEEE Trans. Pattern Anal. Mach. Intell., 33(10):1962-1977) 2011.
[22] Hugo Larochelle and Geoffrey Hinton. Learning to combine foveal glimpses with a third-order boltzmann machine. In Proc. Advances in Neural Inf. Process. Syst., pages 1243-1251, 2010. [23] Ke Li, Kaiyue Pang, Yi-Zhe Song, Timothy M. Hospedales, Tao Xiang, and Honggang Zhang. Synergistic instance-level subspace alignment for grainer ketcch-based image retrieval.
IEEE Trans. Image Process., (1112))5998-522111 2017.
[24)
Wu-Jun Li, Sheng Wang, and Wang-Cheng Kang- Feature learning based deep supervised hashing with pairwise labels. In Proc. Ius. Joint Conf. Artificial Intell., pages 1711-1717, 2015.
[25] Yi Li, Timothy M. Hospedales, Yi-Zhe Song, and Shaogang Gong. Fine-grained sketch-based image retrieval by matching deformable part models. In Proc. British Machine Vis. Conf., pages
10

---

1-12, 2014.
[26] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. DeepFashion: Powering robust clothes recognition and retrieval with rich annotations. In Proc. IEEE Conf. Comp. Vis.
Patt. Recogn., pages 1096-11000 1004, 2016.
[27] Lv, William Josephson, Zhe Wang. Moses Charikar, and Kai Li. Multi-probo LSH: Efficient indexing for high-dimensional similarity search. In Proc. Int. Conf. Very Large Data Bases, pages 950-961, 2007.
[28] Subhransu Maji. Esa Rahtu, Juho Kannala, Mathhew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint arXiv:1306.573: 51511, 2013.
[29] Kaiyue Pang, Yongxin Yang. Timothy M. Hospedales, Tao Xiang. and Yi-Zhe Song. Solving mixed-modal jigsaw puzzle for fineggrined sketch- based image retrieval. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 10347-10355, 2020.
[30] Genevieve Patterson, Chen Xu, Hang Su, and James Hays. The SUN attribute database: Beyond categories for deeper scene understanding. Int. J. Comput. Vision, 108( (08(1-2):59-8) 2014.
[31] Marc' Aurelio Ranzato, Y-Lan Boureau, Sumit Chopra, and Yann LeCun. A unified energy-based framework for unsupervised learning. In ICAIS, pages 371-379, 2007.
[32] Aneeshan Sain, Ayan Kumar Bhunia. Yongxin Yang, Tao Xiang, and Yi-Zhe Song. Cross-modal hicrarchical modelling for fine- aneeggrined sketch based image retrieval. In Proc. British Machine
Vis. Conf.. pages 1-14, 2020.
[33] Fumin Shen, Chunhua Shen, Wei Liu, and Heng Tao Shen. Supervised discrete hashing. In
Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 37-45, 2015.
[34] Anshumali Shrivastava and Ping Li. Densifying one permutation hashing via rotation for fast near neighbor search. In Proc. Int. Conf. Mach. Learn.. pages 557-565, 2014.
[35] Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie. Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 595-604, 2015.
[36] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The iNaturalist species classification and detection dataset. In Proc. JEEE Conf. Comp. Vis. Patt. Recogn. pages 9769-8778, 2017.
[37] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech-UCSD birds-200-201 dataset. Tech. Report CNS-TR-201 100001, 2011.
[38] Jingdong Wang. Heng Tao Shen, and Ting Zhang. A survey on learning to hash. IEEE Truns. Puttern Anul. Mach. Intell., 40(4):769-7)
2018.
[39] Xiu-Shen Wei, Quan Cui, Lei Yang. Peng Wang, and Lingqiao Liu. RPC: A large-scale retail product checkout dataset. arXiv preprin! arXiv:
2019.
[40] Xiu-Shen Wei, Hao Luo. Jianxin Wu, and Zhi-Hua Zhou. Selective convolutional descriptor aggregation for fine-grained image retrieval. IEEE Trans. Image Process. 26(6):2868-286 2017.
[41] Xiu-Shen Wei, Jianxin Wu, and Quan Cui. Deep learning for fine-grained image analysis: A survey. arXiv preprint /907
2019.
[42] Yongqin Xian, Christoph H. Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly. IEEE Trans. Pattern Anal. Mach. Intell., (1(9)22251122 005-2221122665.. 2019.
[43] Qian Yu, Feng Liu, Yi-Zhe Song. Tao Xiang, Timothy M. Hospedales, and Chen-Change Loy. Sketch me that shoe. In Proc. IEEE Conf. Comp. Vis. Pau. Recogn. pages 799-807, 2016. [44] Xianxian Zeng, Yun Zhang, Xiaodong Wang, Kairui Chen, Dong Li, and Weijun Yang. Fine- grained image retrieval via piecewise cross entropy loss. Image and Vision Computing, 93:2868-
2881, 2020.
[45] Hua Zhang, Xiaochun Cao, and Rui Wang. Audio visual attribute discovery for fine-grained object recognition. In Proc. Conf. AAAI, pages 7542-7549, 2018
[46] Jingyi Zhang, Fumin Shen, Li Liu, Fan Zhu, Mengyang Yu, Ling Shao, Heng Tao Shen, and Lue Van Gool. Generative Nomain-migration hashing for sketch-to-image retrieval. In Proc. Eur. Conf. Comp. Vis., pages 297-314, 2018
[47] Xiawu Zheng, Rongrong Ji, Xiaoshuai Sun, Yongjian Wu, Feiyue Huang, and Yanhua Yang. Centralized ranking loss with weakly supervised localization for fine- grained object retrieval.
In Proc. Int. Joint Conf. Artificial Intell., pages 1226-1233 2018
[48] Xiawu Zheng, Rongrong Ji, Xiaoshuai Sun, Baochang Zhang, Yongjian Wu, and Feiyue Huang. Towards optimal fine grained retrieval via decorrelated centralized loss with normalize-scun layer. In Proc. Conf. AAAI, pages 9291 199899998222202 2019.
11


Reviews:
[{"review_id": "yHMdGcdodTd", "reviewer": "Reviewer_yNgD", "summary": "This paper presents an attribute-aware hashing network for generating attribute-aware hash codes. The proposed method includes two keys components: (1) visual feature extraction with attention, (2) learning attribute-specific features with autoencoder. \n\nThe authors conducted experiments on many datasets, and it outperforms recent SOTA approaches.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The proposed framework (feature learning module + binary code module) is technically valid. \n- Compelling results on multiple benchmark. The performance improvements, especially on CUB200, are very strong. \n- Good component analysis to justify the proposed method.\n\nWeakness:\n- Unsupervised Attribute-Guided Learning: I think the proposed method is (almost) the same as learning latent codes in the general unsupervised learning literature. It is unclear to me why it is called \"attribute space\". It looks like it doesn't have any learning objectives that can guarantee the attribute learning. How do you ensure they are attributes? In my opinion, it is just learning latent features, which is commonly studied in the field. There is no novelty here. In other words, the term \"Attribute-Guided Learning\" seems overclaimed in the paper. It would be better to provide more in-depth analysis (including both quantitative and qualitative) to better justify the learned attribute features. \n- Figure 1 is misleading. There is no visual question-answering components in the proposed network to help guide the unsupervised attribute learning. Again, in my opinion, it is overclaimed. The authors only use an autoencoder to obtain latent features and then use the latent features for binary code learning. \n- The hash code learning module require pair-wise supervised training. One possible weakness is that, with the similarity matrix, it may not be scalable to very large dataset due to O(N^2) computational complexity. \n\nOverall:\n- The paper presents very strong results on the benchmark. However, the presentation seems overclaimed. The authors claimed that it is an unsupervised attributed-guided learning approach, but I only found latent feature learning. There is no objectives to guarantee or guide the model to understand the attributes.\n- Novelty is weak. The autoencoder is commonly used to learn latent features. The proposed binary code learning module is also commonly explored in the field. Overall, I couldn't find strong novelty in the paper.\n\nPost rebuttal:\n- After discussions with other reviewers, I am a bit more convinced regarding the novelty and the contribution of the paper. Thus, I am upgrading my rating 4 --> 6.\n- I am OK to accept the paper. But I still have the concern about the attribute learning. It looks like it doesn't have any learning objectives that can guarantee the attribute learning. How can we believe it is learning attribute-aware hash codes? The authors have tried to validate their claims for the attribute-aware hash codes using qualitative visualization. But the qualitative results are usually subjective. I am not sure if it is cherry-picked, but in general it is difficult to evaluate by just looking at the qualitative analysis. In other words, their claims are not thoroughly validated. It remains unclear here.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "n/a", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "kHAymi_Y0Bi", "reviewer": "Reviewer_CEYX", "summary": "(1)\tThis paper aims to address the large-scale fine-grained image retrieval, and it introduces an attention mechanism into a CNN backbone to capture fine-grained local patterns in an image. And it integrates the attentive local-level feature and the extracted global-level feature by the same CNN as the holistic feature of an image,\n(2)\tIt proposes an Attribute-Aware Hash Codes Generating module with an encoder-decoder network by encoding the holistic feature to a latent space, and decoding the latent feature to the holistic feature, and projecting the latent feature to a binary hash code. \n(3)\tIt designs the empirical experiments on 5 fine-grained benchmark datasets to verify the performance improvement of the proposed method compared to existing methods.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes an Attribute-Aware hashing Network, namely A2-NET, to address the large-scale fine-grained image retrieval. The A2-NET integrates an attention mechanism and an encoder-decoder network with a CNN backbone, to learn attribute-aware hash codes, And the experimental results on 5 fine-grained benchmark datasets show improvement of the proposed A2-NET compared to existing methods. However, I still have some concerns as following:\n(1)\tThe novelty of this paper is not so good because both the attention mechanism and the encoder-decoder network have been over-studied in deep learning [8, 19].\n(2)\t The authors do not provide the theoretical proof to prove that the proposed A2-NET   can learn attribute-aware hash codes. And part 4.5 Qualitative Analyses of Attribute-Aware Hash Codes should provide comparisons with other methods and prove that other methods are non-attribute-aware methods.\n(3)\tThe part Attribute-Specific Feature Decorrelation uses an orthogonal regularization, which is common in deep feature learning [a] and deep hashing [b].\n(4)\tThere are some shortcomings about the writing, such as “In particularly” in line 27 and 112, “the explosive growing of fine-grained data” in line 27, “visual representation with attentions” in line 314, “visual attribute have been” in line 318, etc.\n\n[a] “Can we gain more from orthogonality regularizations in training deep networks? \", NeurIPS 2018\n[b] “Deep Multimodal Hashing with Orthogonal Units”. IJCAI, 2015\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No, but the authors should provide the limitations and potential negative societal impact of the work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "f00TAcb3T5l", "reviewer": "Reviewer_gTRX", "summary": "In this paper, authors propose to learn attribute-aware hash codes for fine-grained image retrieval, which establish the explicit correspondences between hash codes and visual attributes. \n\nThe attention mechanism is employed to extract the fine-grained tailored visual patterns from both global-level and local-level, and unsupervised reconstruction task is defined to distill the attribute related vectors through an encoder-decoder structure. Finally the hash codes are generated by the attribute vectors.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The idea of distilling the high-level attribute-specific vectors in an unsupervised learning manner is interesting, and the proposed A2-Net is also equipped with the feature decorrelation constraint to enhance the representation ability of the attribute vectors.\n\n- The experimental results on five fine-grained benchmark datasets validate the proposed method.\n\n- The derivation of hash code generation is compact. And experimental results against state-of-the-art methods demonstrate its superiority. The additional studies are comprehensive, which includes the investigation of length of hash codes and qualitative analysis.\n\nPost rebuttal feedback:\nAfter considering the reviews from other reviewers and the feedback from the authors, I'm willing to raise my score to accept.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No other significant concerns raise from my perspective in this round. The comments from other reviewers are very much appreciated to make further evaluation.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "eG9aRxEjIip", "reviewer": "Reviewer_iHzy", "summary": "This paper proposes a unified framework to generate attribute-aware hash codes for dealing with the large-scale fine-grained image retrieval task. Specifically, it develops an attribute-aware hashing network, termed as A^2-Net, consisting of two main modules including fine-grained representation learning and attribute-aware hash codes generating. In A^2-Net, the proposed reconstruction task realizing by an encoder-decoder structure is proven to be able to unsupervisedly distill high-level attribute-specific vectors from the appearance visual representations without attribute annotations. Thus, the hash codes can be generated by the attribute vectors based on a feature decorrelation constraint and the similarity preserving constraint. Experiments are conducted on several fine-grained benchmarks, which can show outperforming on both qualitative and quantitative aspects.", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ The large-scale fine-grained retrieval problem studied in this paper is a fundamental and practical task that deserves further study.\n+ This paper not only develops an effective method wrt retrieval accuracy, but also brings interesting interpretability. The whole method is well-motivated.\n+ The technique contributions of this paper are novel and reasonable, which is tailored for the challenges of large-scale fine-grained retrieval. Particularly, the proposed encoder-decoder structure for distilling high-level attribute-specific vectors from visual features is interesting. Moreover, the feature decorrelation constraint is introduced for further improving the discriminative ability of these attribute vectors.\n+ Experiments are conducted on five popular fine-grained datasets, which show consistent and significant improvements over baseline methods. Also, the ablation studies are adequate and can validate the effectiveness and robustness of A^2-Net, as well as its main components.\n+ The qualitative analyses of these attribute-aware hash codes in Sec. 4.5 validate the effectiveness of the proposed method and offer good interpretation.\n+ This paper is well written and easy to follow.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The attribute-aware hash code in this paper is interesting. The authors also have shown some visualization results. But, I am still wondering that what is the relation between the attention in the feature learning module and the generated hash codes? Can they correspond one by one? The authors are encouraged to discover that in the future.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thanks for your submission to NeurIPS.

The reviewers were all in agreement that this is a solid paper that deserves to be accepted.  Overall, the reviewers praised the importance of the problem, the proposed solution, and the empirical results.  After the rebuttal and discussion, all advocated for accepting the paper.  Despite being overall very positive about the paper, the reviewers did note a few weaknesses; please keep these in mind when preparing a final version of the manuscript.
Decision: Accept (Spotlight)