A-NET: Learning Attribute-Aware Hash Codes for Large-Scale Fine-Grained Image Retrieval
Year: 2021

Abstract:
Our work focuses on tackling large-scale fine-grained image retrieval as ranking the images depicting the concept of interests (i.e., the same sub-category labels) highest based on the fine-grained details in the query. It is desirable to alleviate the challenges of both fine-grained nature of small inter-class variations with large intra-class variations and explosive growth of fine-grained data for such a practical task. In this paper, we propose an Attribute-Aware hashing Network (A-NET) for generating attribute-aware hash codes to not only make the retrieval process efficient, but also establish explicit correspondences between hash codes and visual attributes. Specifically, based on the captured visual representations by attention, we develop an encoder-decoder structure network of a reconstruction task to unsupervisedly distill high-level attribute-specific vectors from the appearance-specific visual representations without attribute annotations. A-NET is also equipped with a feature decorrelation constraint upon these attribute vectors to enhance their representation abilities. Finally, the required hash codes are generated by the attribute vectors driven by preserving original similarities. Qualitative experiments on five benchmark fine-grained datasets show our superiority over competing methods. More importantly, quantitative results demonstrate the obtained hash codes can strongly correspond to certain kinds of crucial properties of fine-grained objects.

1 Introduction
Fine-grained image retrieval in computer vision aims to retrieve images belonging to multiple subordinate categories of a super-category (aka a meta-category), e.g., different species of animals/plants [36], different models of cars [20], different kinds of retail products [39], etc. Its key challenge therefore lies with understanding fine-grained visual differences that sufficiently discriminate between objects that are highly similar in overall appearance, but differ in fine-grained features. Also, fine-grained retrieval still demands ranking all the instances so that images depicting the concept of interest (e.g., the same sub-category label) are ranked highest based on the fine-grained details in the query.
In particular, with the explosive growth of fine-grained data in real applications [1, 14, 26, 36, 39], fine-grained hashing, as a promising solution for dealing with large-scale fine-grained retrieval tasks, has proven to be able to greatly reduce the storage cost and increase the query speed [8, 18] benefiting from the learned compact binary hash code representations. However, although previous works,
∗Corresponding author. X.-S. Wei, Y. Shen, X. Sun and J. Yang are with PCA Lab, Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, and Jiangsu Key Lab of Image and Video Understanding for Social Security, School of Computer Science and Engineering, Nanjing University of Science and Technology. This work was supported by Natural Science Foundation of Jiangsu Province of China under Grant (BK20210340), the Fundamental Research Funds for the Central Universities (No. 30920041111), and CAAI-Huawei MindSpore Open Fund (CAAIXSJLJJ-2020-022A). The first two authors contributed equally to this work.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Query image
101 1
Encoder
3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET model. We elaborate it in65 the following four aspects, i.e., unsupervised attribute-guided learning, attribute-specific feature66 decorrelation, hash code learning and the optimization learning algorithm.67 3.3.1 Unsupervised Attribute-Guided Learning68 In real-applications, especially for the large-scale and fine-grained tasks, attribute annotations are69 always infeasible, which limits the learning process to be conducted in an unsupervised setting.70 While, in the literature, the main goal of unsupervised learning is to capture regularities in data for the71 purpose of extracting useful representations or for restoring corrupted data [4]. Many unsupervised72 methods explicitly produce internal latent units or codes, from which the data is to be reconstructed.73
Inspired by this, we develop an unsupervised attribute-guided learning component to project the74 holistic representation xi of Ii into a latent space, i.e., the attribute space V . In V , its high-level75 vectors are designed to have certain desirable properties, e.g., corresponding to semantic properties76 of fine-grained objects (a.k.a. “fine-grained attributes”).77
More specifically, in our A2-NET, the unsupervised attribute-guided learning is realized by a recon-78 struction paradigm with an encoder-decoder structure, as shown in Figure ??. In concretely, given79 a batch of n training data Ii, the holistic representations X = {x1; x2; . . . ; xn} 2 Rd⇥n can be80 obtained as aforementioned. By formulation, the encoder projects X into the attribute space V with81 a projection matrix W 2 Rk⇥d to get an internal latent representation V 2 Rk⇥n w.r.t. X . In82 particularly, we set that the dimension of latent representation k equals to the number of hash bits in83 the final binary hash code ui. Furthermore, each column of V , i.e., vi 2 Rk, can derive ui by84
ui = sgn(tanh(vi)) . (5)
Meanwhile, regarding vi, we also consider to reconstruct its input xi by a decoder as a counterpart85 of the encoder. Therefore, on one hand, such a reconstruction paradigm can reduce and further distill86 high-level semantic cues in the attribute space V . While, on the other hand, it can drive the training87 of A2-NET by preserving the similarity between queried hash codes and database points in terms of88 learning to hash (cf. Section 3.3.3).89
In concretely, the learning objective of unsupervised attribute-guided learning is written as follows:90
min W
kX W>WXk2F s.t. WX = V 0 = tanh(V ) , (6)
where the decoder (i.e., a counterpart of the encoder) is realized by W> to simplify the network.91 However, directly minimizing Eq. (6) with a hard constraint is hard to optimize. Therefore, we relax92 the constraint into a soft constraint, and the learning objective can be rewritten by93
min W
kX W>V 0k2F + kWX V 0k2F . (7)
3.3.2 Attribute-Specific Feature Decorrelation94
By conducting the aforementioned unsupervised attribute-guided learning, we can obtain the internal95 latent vectors V 0 as the attribute-specific features. In order to both enhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features, we introduce a feature97 decorrelation constraint upon V 0, which is formulated by98
min V 0
kV 0V 0> kIk2F , (8)
where I is the identity matrix. Such a feature decorrelation constraint is preferable to construct99 independent features and reduce redundant information. Therefore, based on unsupervised attribute-100 guided learning and attribute-specific feature decorrelation, the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102
3.3.3 Hash Code Learning103
In the following, we conduct the hash code learning based on the obtained attribute-specific features.104 Assume that we have n query data points which are denoted as {qi}ni=1, as well as m database points105
3
3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET model. We elaborate it in65 the following four aspects, i.e., unsupervised attribute-guided learning, attribute-specific feature66 decorrelation, hash code learning and the optimization learning algorithm.67 3.3.1 Unsupervised Attribute-Guided Learning68 In real-applications, especially for the large-scale and fine-grained tasks, attribute annotations are69 always infeasible, which limits the learning process to be conducted in an unsupervised setting.70 While, in the literature, the main goal of unsupervised learning is to capture regularities in data for the71 purpose of extracting useful representations or for restoring corrupted data [4]. Many unsupervised72 methods explicitly produce internal latent units or codes, from which the data is to be reconstructed.73 Inspired by this, we develop an unsupervised attribute-guided learning component to project the74 holistic representation xi of Ii into a latent space, i.e., the attribute space V . In V , its high-level75 vectors are designed to have certain desirable properties, e.g., corresponding to semantic properties76 of fine-grained objects (a.k.a. “fine-grained attributes”).77
More specifically, in our A2-NET, the unsupervised attribute-guided learning is realized by a recon-78 struction paradigm with an encoder-decoder structure, as shown in Figure ??. In concretely, given79 a batch of n training data Ii, the holistic representations X = {x1; x2; . . . ; xn} 2 Rd⇥n can be80 obtained as aforementioned. By formulati n, th en oder pr jects X into the attribute sp ce V with81 a projection matrix W 2 Rk⇥d to get an internal latent representation V 2 Rk⇥n w. .t. X . In82 particularly, we set that the dimension of latent representation k equals to the number of hash bits in83 the final binary hash code ui. Furthermore, each column of V , i.e., vi 2 Rk, ca derive ui by84
ui = sgn(tanh(vi)) . (5)
Meanwhile, regarding vi, we also consider to reconstruct its input xi by a decoder as a counterpart85 of the encoder. Therefore, on one hand, such a reconstruction paradigm can reduce and further distill86 high-level semantic cues in the attribute space V . While, on the other hand, it can drive the training87 of A2-NET by preserving the similarity between queried hash cod s and database points in terms of88 learning to hash (cf. Section 3.3.3).89
In concretely, the learning objective of unsupervised attribute-guided learning is written as follows:90
min W
kX W>WXk2F s.t. WX = V 0 = tanh(V ) , (6)
where the decoder (i.e., a counterpart of the encoder) is realized by W> to simplify the network.91 However, directly minimizing Eq. (6) with a hard constraint is hard to optimize. Therefore, we relax92 the constraint into a soft constraint, and the learning objective can be rewritten by93
min W
kX W>V 0k2F + kWX V 0k2F . (7)
3.3.2 Attribute-Specific Feature Decorrelation94
By conducting the aforementioned unsupervised attribute-guided learning, we can obtain the internal95 latent vectors V 0 as the attribute-specific features. In order to both nhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features, we introduce a feature97 decorrelation constraint upon V 0, which is formulated by98
mi V 0
kV 0V 0> kIk2F , (8)
where I is the identity matrix. Such a feature decorrelation constraint is preferable to construct99 independent features and reduce redundant information. Therefore, based on unsupervised attribute-100 guided learning and attribute-specific feature decorrelation, the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102
3.3.3 Hash Code Learning103
In the following, we conduct the hash code learning based on the obtained attribute-specific features.104 Assume that we have n query data points which are denoted as {qi}ni=1, as well as m database points105
3
3.3 Attribute-Aware Hash C es Gen ating64
How to generate attribute-aware hash codes is the key of our A2-NET model. We elaborate it in65 the following four aspec s, i.e., unsup rvised attribute-guided learning, attribute-specific feature66 decorrelati n, hash code learning and the optimization learning algorithm.67
3.3.1 Unsupervised Attribute-Guided Learning68
In real-applications, especially for the large-scale and fine-grained tasks, attribute annotations are69 always infeasible, which limits the learning process to be conducted in an unsupervised setting.70 While, in the literature, the main goal of unsupervised learning is to capture regularities in data for the71 purpose of extracting useful representations or for restoring corrupted data [4]. Many unsupervised72 methods explicitly produce internal latent units or codes, from which the data is to be reconstructed.73
Inspired by this, we develop an unsupervised attribute-guided learning component to project the74 holistic representation xi of Ii into a latent space, i.e., the attribute space V . In V , its high-level75 vectors are designed to have certain desirable properties, e.g., corresponding to semantic properties76 of fine-grained objects (a.k.a. “fine-grained attributes”).77
More specifically, in our A2-NET, the unsupervised attribute-guided learning is realized by a recon-78 structi n paradigm with an encoder-decod r structure, as shown in Figure ??. In concretely, given79 a batch of n training data Ii, the holistic representations X = {x1; x2; . . . ; xn} 2 Rd⇥n can be80 obtained as aforementioned. By formulation, the encoder projects X into the attribute space V with81 a projectio matrix W 2 Rk⇥d to get an nternal latent representation V 2 Rk⇥n w.r.t. X . I82 particularly, w set that the dimension of latent repre entation k equals t the number of hash bits in83 the final binary ash c de ui. Furthermore, each column of V , i.e., vi 2 Rk, can derive ui by84
ui = sgn(tanh(vi)) . (5)
Meanwhile, regarding vi, we also consider to reconstruct its input xi by a decoder as a counterpart85 of the encod r. Therefore, on one hand, uch a reconstructio paradigm can reduce and further distill86 high-level semantic cues in the attribute space V . While, on the other hand, it can drive the training87 of A2-NET by preserving the similarity between queried hash codes and database points in terms of88 learning to hash (cf. Section 3.3.3).89
In con retely, the learn ng objective of u supervised attribute-guided learning is written as follows:90
min W
kX W>WXk2F .t. WX = V 0 = tanh(V ) , (6)
where the decoder (i.e., a counterpart f the encoder) is r alized y W> to simplify the network.91 However, irectly minimizi g Eq. (6) with a hard con traint is hard to optimize. Therefore, we relax92 the constraint into a soft constraint, and the learning objective can be rewritten by93
min W
kX W>V 0k2F + kWX V 0k2F . (7)
3.3.2 Attribute-Specific Feature Decorrelation94
By conducting the aforementioned unsupervised attribute-guided learning, we can obtain the internal95 latent vectors V 0 as the attribute-specific features. In order to both enhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features, we introduce a feature97 decorrelation constraint upon V 0, which is formulated by98
min V 0
kV 0V 0> kIk2F , (8)
where I is the identity matrix. Such a feature decorrelation constraint is preferable to construct99 independent features and reduce redundant information. Therefore, based on unsupervised attribute-100 guided learning and attribute-specific feature decorrelation, the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102
3.3.3 Hash Code Learning103
In the following, we conduct the hash code learning based on the obtained attribute-specific features.104 Assume that we have n query data points which are denoted as {qi}ni=1, as well as m database points105
3
3.3 Attribute-Aware Hash Codes Generating64
How to generate attribute-aware hash codes is the key of our A2-NET model. We elaborat it in65 the following four aspects, i.e., unsupervised attribute-guided learning, attribute-specific feature66 decorrelation, hash code learning and the optimization learning algorithm.67
3.3.1 Unsupervised Attribute-Guided Learning68
In real-applications, especially for the large-sca e and fine-grained tasks, attribute an otation re69 always infeasible, which limits the learning process to be nducted in an unsupervised setting.70 While, in the literature, the main goal of unsupervised learning is to capture regularities in data for the71 purpose of extracting useful representations or for restoring corrupted data [4]. Many unsupervised72 methods explicitly produce internal latent units or codes, from which the data is to be reconstructed.73
Inspired by this, we develop an unsupervised attribute-guided learning component to project the74 holistic representation xi of Ii into a latent space, i.e., the attribute sp ce V . In V , its high-lev l75 vectors are designed to have certain desirable properties, e.g., corresponding to semantic properties76 of fine-grained objects (a.k.a. “fine-grained attributes”).77
More specifically, in our A2-NET, the unsupervised attribute-guided learning is realized by a recon-78 struction paradigm with an encoder-decoder structure, as shown in Figure ??. In concretely, given79 a batch of n training data Ii, the holistic representations X = {x1; x2; . . . ; xn} 2 Rd⇥n can be80 obtained as aforementioned. By formulation, the encoder projects X into the attribute space V with81 a projection matrix W 2 Rk⇥d to get an internal latent representation V 2 Rk⇥n w.r.t. X . In82 particularly, we set that the dimension of latent representation k equals to the number of hash bits in83 the final binary hash code ui. Furthermore, each column of V , i.e., vi 2 Rk, can derive ui by84
ui = sgn(tanh(vi)) . (5)
Meanwhile, regarding vi, we also consider to reconstruct its input xi by a decoder as a counterpart85 of the encoder. Therefore, on one hand, such a reconstruction paradigm can reduce and further distill86 high-level semantic cues in the attribute space V . While, on the other hand, it can drive the training87 of A2-NET by preserving the similarity between queried hash codes and database points in terms of88 learning to hash (cf. Section 3.3.3).89
In concretely, the learning objective of unsupervised attribute-guided learning is written as follows:90
min W
kX W>WXk2F s.t. WX = V 0 = tanh(V ) , (6)
where the decoder (i.e., a counterpart of the encoder) is realized by W> to simplify the network.91 However, directly minimizing Eq. (6) with a hard constraint is hard to optimize. Therefore, we relax92 the constraint into a soft constraint, and the learning objective can be rewritten by93
min W
kX W>V 0k2F + kWX V 0k2F . (7)
3.3.2 Attribute-Specific Feature Decorrelation94
By conducting the aforementioned unsupervised attribute-guided learning, we can obtain the internal95 latent vectors V 0 as the attribute-specific features. In order to both enhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features, we introduce a feature97 decorrelation constraint upon V 0, which is formulated by98
min V 0
kV 0V 0> kIk2F , (8)
where I is the identity matrix. Such a feature decorrelation constraint is pref rable to construct99 independent features and reduce redundant information. Therefore, based on unsupervise ttri t -100 guided learning and attribute-specific feature decorrelation, the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102
3.3.3 Hash Code Learning103
In the following, we conduct the hash code learning based on the obtained attribute-sp cific feat res.104 Assume that we have n query data points which are denoted as {qi}ni=1, as well as m database points105
3
Decoder
3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET model. We elaborate it in65 the following four aspects, i.e., unsupervised attribute-guided learning, attribute-specific feature66 decorrelation, hash code learning and the optimization learning algorithm.67 3.3.1 Unsupervised Attribute-Guided Learning68 In real-applications, especially for the large-scale and fine-grained tasks, attribute annotations are69 always infeasible, which limits the learning process to be conducted in an unsupervised setting.70 While, in the literature, the main goal of unsupervised learning is to capture regularities in data for the71 purpose of extracting useful representations or for restoring corrupted data [4]. Many unsupervised72 methods explicitly produce internal latent units or codes, from which the data is to be reconstructed.73 Inspired by this, we develop an unsupervised attribute-guided learning component to project the74 holistic representation xi of Ii into a latent space, i.e., the attribute space V . In V , its high-level75 vectors are designed to have certain desirable properties, e.g., corresponding to semantic properties76 of fine-grained objects (a.k.a. “fine-grained attributes”).77 More specifically, in our A2-NET, the unsupervised attribute-guided learning is realized by a recon-78 struction paradigm with an encoder-decoder structure, as shown in Figure ??. In concretely, given79 a batch of n training data Ii, the holistic representations X = {x1; x2; . . . ; xn} 2 Rd⇥n can be80 obtained as aforementioned. By formulation, the encoder projects X into the attribute space V with81 a projection matrix W 2 Rk⇥d to get an internal latent representation V 2 Rk⇥n w.r.t. X . In82 particularly, we set that the dimension of latent representation k equals to the number of hash bits in83 the final binary hash code ui. Furthermore, each column of V , i.e., vi 2 Rk, can derive ui by84 ui = sgn(tanh(vi)) . (5)
Meanwhile, regarding vi, we lso consider to reco struct its input xi by a decod r as a counterpart85 of the ncoder. Therefore, on one ha d, such a rec nstruction paradigm can reduce and further distill86 high-level semantic cues in the attribute space V . While, on the other hand, it can drive the training87 of A2-NET by preserving the similarity between queried hash codes and database points in terms of88 learning to hash (cf. Section 3.3.3).89
In concretely, the learni g objective of unsupervised attribute-guided learni g is written as follows:90
min W
kX W>WXk2F s.t. WX = V 0 = tanh(V ) , (6)
where the decoder (i.e., a counterpart of the encod r) is realized by W> to simplify the network.91 How ve , directly minimizing Eq. (6) with a hard constraint is hard to optimize. Therefore, we relax92 the constraint into a soft constraint, nd the learning objective can be rewritten by93
min W
kX W>V 0k2F + kWX V 0k2F . (7)
3.3.2 Attribute-Specific Feature Decorrelation94
By conducting the aforementioned unsupervised attribute-guided learning, we can obtain the internal95 latent vectors V 0 as the attribute-specific features. In order to both enhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features, we introduce a feature97 decorrelation constraint upon V 0, which is formulated by98
min V 0
kV 0V 0> kIk2F , (8)
where I is the identity matrix. Such a feature decorrelation constraint is preferable to construct99 independent features and reduce redundant information. Therefore, based on unsupervised attribute-100 guided learning and attribute-specific feature decorrelation, the final learned hash codes are expected101 to be both attribute-aware and hash-bit independ nt.102
3.3.3 Hash Code Learning103
In the following, we conduct the hash code learning based on the obtained attribute-specific features.104 Assume that we have n query data points which are denoted as {qi}ni=1, as well as m database points105
3
which are denoted as {vj}mj=1. Note that, both qi and vi are belonging to the attribute space V . Then,106 the corresponding binary hash codes can be obtained via107 ui = sgn(tanh(qi)) , (9) zj = sgn(tanh(vj)) , (10) where ui, zj 2 { 1, +1}k. The goal of our hash code learning is to learn binary hash codes for both108 query points and database points from {qi}ni=1, {vj}mj=1, and the pairwise supervised information,109 i.e., S 2 { 1, +1}n⇥m. To preserve the pairwise similarity, we adopt the `2 loss between the110 supervised information (a.k.a. similarity) and the inner product of query-database point binary code111 pairs. It can be formulated as follows:112 minU ,Z Pn i=1 Pm j=1 u>i zj kSij 2 s.t. U 2 { 1, +1}n⇥k, Z 2 { 1, +1}m⇥k . (11) Overall, we get the final objective of the proposed A2-NET model by considering Eq. (7), Eq. (8) and113 Eq. (11) together as follows:114 min W ,⇥ L(I) = kX W>V 0k2F + kWX V 0k2F +↵kV 0V 0> nIk2F + nX i=1 mX j=1 u>i zj kSij 2 , (12) where , ↵ and are hyper-parameters as the trade-off.115 In practice, uri g training, it might be only available a set of database points {vj}mj=1 without query116 points. Thus, we randomly sample n data points from database to construct a query set, and denote117 the indices of all the database points as with the indices of the query set as ⌦. Additionally, because118 we cannot back-propagate the gradient to ⇥ due to the sgn(·) function, we omit the sgn(·) function119 and only apply tanh(·) for relaxation in Eq. (10) of the whole optimization process. Therefore, the120 optimization formulation of A2-NET can be rewritten with only database points {vj}mj=1 for training121 as:122 min W ,⇥ L(I) = kX W>V 0k2F + kWX V 0k2F + ↵kV 0V 0> nIk2F + X i2⌦ X j2 ⇣ tanh(W · F (Ii;⇥))>zj kSij ⌘2 . (13) For optimization, we employ the back-propagation algorithm and follow [8] to train the whole123 A2-NET m l in an end-to-end manner.124 3.4 Out-of-Sample Extension125 After training A2-NET, the learned model can be applied for generating binary codes for query points126 including unseen query points in th training phase. Specifically, we can use the following equation127 to generate the binary code for Iq:128
uq = sgn(tanh(W · F (Iq;⇥))) . (14)
4 Experiments129
In this section, we present the implementation details, empirical settings, main results, ablation studies130 and also some interesting qualitative a alyses of the learned attribute-aware binary hash codes.131
4.1 Datasets132
x0i By following ExchNet [4], our experiments are conducted on five fine-grained benchmark datasets,133 i.e., CUB-200-2011 [16], Aircraft [12], Food101 [1], NABirds [15] and VegFru [6]. Specifically,134 CUB-200-2011 is one of the most popular used fine-grained datasets. It contains 11,788 bird images135 from 200 bird species and is officially split into 5,994 images for training with 5,794 images for136 test. Aircraft contains 10,000 images spanning 100 aircraft models with 3,334 for training, 3,333137
4
Fine-grained representation learning
U ns
up er
vi se
d at
tr ib
ut e-
gu id
ed lo
ss (S
ec tio
n 3.
3. 1)
Feature decorrelation loss (S ction 3.3.2)
3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET model. We laborate it in65 the foll wing four aspects, i.e., unsupervised attribute-guided learning, attribute-specifi feature66 decorrelation, hash code learning and the optimization learning algorithm.67 3.3.1 Unsupervised Attribute-Guided Learning68
In real-applications, especially for the large-scale and fine-grained tasks, attribute annotations are69 always infeasibl , which limits the learning process to be conducted in an unsupervised setting.70 While, in the literature, the main goal of unsupervised learning is to capture regularities in data for the71 purp se of xtracting useful representations or for restoring corrupted data [13]. Many unsupervised72 methods explicitly produce internal latent units or codes, from which the data is to be reconstructed.73
Inspired by this, we develop an unsupervised attribute-guided learning component to project the74 holistic r presentation xi of Ii into a latent space, i.e., the attribute space V . In V , its high-level75 vectors are designed to have certain desirable properties, e.g., corresponding to semantic properties76 of fine-grained objects (a.k.a. “fine-grained attributes”).77
More specifically, in our A2-NET, the unsupervised attribute-guided learning is realized by a recon-78 struction paradigm with an encoder-decoder structure, as shown in Figure ??. In concretely, given79 a batch of n training data Ii, the holistic representations X = {x1; x2; . . . ; xn} 2 Rd⇥n can be80 obtained as aforementioned. By formulation, the encoder projects X into the attribute space V with81 a projection matrix W 2 Rk⇥d to get an internal latent representation V 2 Rk⇥n w.r.t. X . In82 particularl , we set that the dimension of lat nt representation k quals to the number of hash bits in83 the final binary hash code ui. Furthermore, each column of V , i.e., vi 2 Rk, can derive ui by84
ui = sgn(tanh(vi)) . (5)
Meanwhile, regarding vi, we als consider to reconstruct its input xi by decoder as a counterpart85 of the encoder. Therefore, o one hand, such a reconstruction paradigm can reduce and further distill86 high-level semantic cues in the attribute space V . While, on the other hand, it can drive the training87 of A2-NET by preserving the similarity between queried hash codes and database points in terms of88 learning to hash (cf. Section 3.3.3).89
In concretely, the learning objective of unsupervised attribute-guided learning is written as follows:90
min W
kX W>WXk2F s. . WX = V 0 = tanh(V ) , (6)
where the decoder (i.e., a counterpart of the encoder) is realized by W> to simplify the network.91 Howev r, directly minimizing Eq. (6) with a hard constraint is hard to optimize. Therefore, we relax92 the constraint into a soft constraint, and the learning objective can be rewritten by93
min W
kX W>V 0k2F + kWX V 0k2F . (7)
3.3.2 Attribute-Specific Feature Decorrelation94
By conducting the aforementioned u supervised attribute-guided learning, we can obtain the internal95 latent vectors V 0 as the attribute-specific features. In order to both enhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features, we introduce a feature97 d correlation constraint upon V 0, which is formulated by98
min V 0
kV 0V 0> nIk2F , (8)
where I is the i entity matrix. Such a feature decorrelation constraint is preferable to construct99 independent features and reduce redundant information. Therefore, b sed on unsupervise attribute-100 guided learning and attribute-specific feature decorrelation, the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102
3.3.3 Hash Code Learning103
In the following, we conduct the hash code lear ing based on the obtained attribute-specific features.104 Assume that we have n query data points which are denoted as {qi}ni=1, s well as m database points105
3
Hash code learning loss (Section 3.3.3)
Attribute-aware hash codes generating“Red head” “On the tree” “Spotted body”
2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26
3 Methodology27
In this section, we introduce the overall framework and notations of the proposed A2-NET model, as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29
3.1 Overall Framework and Notations30
As illustrated in Figure ??, our A2-NET model consists of two crucial modules, i.e., a fine-grain d31 representation learning module and an attribute-aware hash codes generating module. Given an32 input image Ii, based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN, a set of attention guidance A = {Ac} is learned for capturing fine-grained tailored34 local patterns T ci from Ti. To distill semantical cues and further generate the final attribute-aware35 binary hash codes, we propose to transform these appearance-specific features T towards semantic-36 specific representations T̂ by performing a transform network (·). After aggregating T̂ , the obtained37 attentive local-level features xci are associated with the global-level feature x global i to form as a38 holistic feature representation xi. In order to generate attribute-aware binary hash codes, we conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond o40 an attribute vector w.r.t. one kind of namable properties of fine-grained objects. Finally, hash code41 learning is performed upon vi to obtain the final attribute-a are binary codes ui.42
3.2 Fine-Grained Representation Learning43
Attention plays an important role in human perception [3, 7], and humans exploit a sequence of44 partial glimpses and s lective y focus on salient parts of an object or a s ene in order to bet er capture45 visual structure [10]. Inspired by this, we incorporat the attention mechanism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differ nces between these47 subordinate categories.48
In concretely, we extract the deep feature of its input image Ii via a backbone CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Then, based on Ti, C attention guidance Ac 2 RH⇥W is generated as a set of attention maps, i.e., A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obtain the final attribu e-aware binary co es, it is54 desirable to transform these appearance-specific (i.e., low-level) features T to semantic-specific (i.e.,55 mid-level) representations which are closer to the attribute space. Thus, a transforming network (·),56 which is equipped with a stack of convolution layers, is performed on T as follows:57
T̂ ci = (T c i ; ✓local) , (3)
T̂i = (Ti; ✓global) , (4)
where ✓ presents the para eters of the corresponding transfor ing networks w.r.t. T ci and Ti, respec-58 tively. Then, we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . The holistic60 feature representation w.r.t. the input image Ii is achieved by concatenating both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F (Ii;⇥) 2 R . Note that, we hereby abstract the aforementioned fine-grai ed62 feature learning process as a function F (Ii;⇥) with its parameters ⇥.63
2
Attention guidance
2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section, we introduce the overall framework and notations of the proposed A2-NET model, as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29 3.1 Overall Framework and Notations30 As illustrated in Figure ??, our A2-NET model consists of two crucial modules, i.e., a fine-graine31 representation learning module and an attribute-aware hash codes generating module. Given an32 input image Ii, based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN, a set of attention guidance A = {Ac} is learned for capturing fi e-grained tailored34 local patterns T ci from Ti. To distill semantical cues and further generate the final attribute-aware35 binary hash c des, we propose to transform these appearance-specific features T towa ds semantic-36 specific representations T̂ by performing a transform network (·). After aggregating T̂ , the obt ined37 attentive local-level features xci are associated with the global-level feature x global i to form as a38 holistic feature representation xi. In order to generate attribute-aware binary hash codes, we conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond to40 an attribute vector w.r.t. one kind of namable properties of fine-grained objects. Finally, hash code41 learning is performed upon vi o obtain the final attribute-aware binary codes ui.42
3.2 Fine-Grained Representation Learning43
Attention plays a imp rtant role in human perception [3, 7], and humans exploit a sequence of44 artial glimpses d selectively focus on salient parts of an object or a scene in or er to better capture45 visual structure [10]. Inspired by this, we incorporate the attention mechanism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48
In concretely, we extract the deep feature of its input image Ii via a backbone CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Then, based on Ti, C attention guidance Ac 2 RH⇥W is g nerated as a set of attention ma s, i.e., A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obtain the final attribute-aware binary codes, it is54 desirable to transform these appearance-specific (i.e., low-level) features T to semantic-specific (i.e.,55 mid-level) representations which are closer to the attribute space. Thus, a transforming network (·),56 which is equipped with a stack of convolution layers, is performed on T as follows:57
T̂ ci = (T c i ; ✓local) , (3)
T̂i = (Ti; ✓global) , (4)
where ✓ presents the parameters of the corresponding transforming networks w.r.t. T ci and Ti, respec-58 tively. Then, we aggregate T̂ ci and T̂i by conducting global average-pooling d correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . The holistic60 feature representation w.r.t. the input image Ii is achieved by concatenati g both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F (Ii;⇥) 2 Rd. Note that, we hereby abstract the aforementioned fine-grained62 feature learning process as a function F (Ii;⇥) with its parameters ⇥.63
2
2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section, we introduce the overall framework and notations of the proposed A2-NET model, as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29 3.1 Overall Framework and Notations30 As illustrated in Figure ??, our A2-NET model consists of two crucial modules, i.e., a fine-grained31 representation learning module and an attribute-aware hash codes generating module. Given an32 input image Ii, based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN, a set of attention guidance A = {Ac} is learned for capturing fine-grained tailored34 local patterns T ci from Ti. To distill semantical cues and further generate the final attribute-aware35 binary hash codes, we propose to transform these appearance-specific features T towards semantic-36 specific representations T̂ by performing a transform network (·). After aggregating T̂ , the obtained37 attentive local-level features xci are associated with the global-level feature x global i to form as a38
holistic feature representation xi. In order to generate attribute-aware binary hash codes, we conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond to40 an attribute vector w.r.t. one kind of namable properties of fine-grained objects. Finally, hash code41 learning is performed upon vi to obtain the final attribute-aware binary codes ui.42
3.2 Fine-Grained Representation Learning43
Attention plays an important role in human perception [3, 7], and humans exploit a sequence of44 partial glimpses and selectively focus on salient parts of an object or a scene in order to better capture45 visual structure [10]. Inspired by this, we incorporate the attention mechanism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48
In concretely, we extract the deep feature of its input image Ii via a backbone CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Then, based on Ti, C attention guidance Ac 2 RH⇥W is generated as a set of attention maps, i.e., A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obtain the final attribute-aware binary codes, it is54 desirable to transform these appearance-specific (i.e., low-level) f atures T to semantic-specific (i.e.,55 mid-level) representations which are closer to the attrib te space. Thus, a transforming n twork (·),56 which is equipped with a stack of convolution layers, is performed on T as follows:57
T̂ ci = (T c i ; ✓local) , (3) T̂i = (Ti; ✓global) , (4)
where ✓ presents the parameters of the corresponding transfor ing networks w.r.t. T ci and Ti, respec-58 tively. Then, we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x glo al i . Th holistic60 feature representation w.r.t. the input image Ii is achieved by concatenating both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F (Ii;⇥) 2 Rd. Note that, we hereby abstract the aforementioned fine-grained62 feature learning process as a function F (Ii;⇥) with its parameters ⇥.63
2
2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section, we introduce the overall framework and notations of the proposed A2-NET model, as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29 3.1 Overall Framework and Notations30 As illustrated in Figure ??, our A2-NET model consists of two crucial modules, i.e., a fine-grai d31 representation learning module and an attribute-aware hash codes g nerating modul . Given an32 input image Ii, based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN, a set of attention guidance A = {Ac} is learned for capturing fine-grained tailored34 local patterns T ci from Ti. To distill semantical cues and further generate the final attribute-aware35 bi ary ash c des, we propose to transform these appearance-specific features T towards semantic-36 specific representations T̂ by perform ng a transform network (·). After aggregating T̂ , th obtained37 attentive local-level features xci are associated with the global-level f ature x global i to form as a38 holistic feature representation xi. In order to generate attribute-aware binary hash codes, we conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond to40 an attribute vector w.r.t. one kind of namable properties of fine-grained objects. Finally, hash code41 learning is performed upon vi to obtain the final attribute-aware binary codes ui.42 3.2 Fine-Grained Representation Learning43
Attention plays an important role in human perception [3, 7], and humans exploit a sequence of44 partial glimpses and selectively focus on salient parts of an object or a scene in order to b ter capture45 visu l structure [10]. I spired by this, we incorporate th attention mech nism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48
In concretely, we extra t the deep feature of its input image Ii via a backbon CNN model CNN(·)49 s50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Then, based on Ti, C attention guidance Ac 2 RH⇥W is generated as a set of attention maps, i.e., A.51 The attention guidance Ac is designed to ev luate which deep descript rs should be attended or ven52 ov rlooked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obtain the final attribute- ware binary codes, it is54 desirable to transform these appearance-specific (i.e., low-level) features T to semantic-specific (i.e.,55 mid-level) representations which are closer to the attribute space. Thus, a transforming network (·),56 which is equipped with a stack of convolution layers, is performed on T as follows:57
T̂ ci = (T c i ; ✓local) , (3)
T̂i = (Ti; ✓global) , (4)
where ✓ pr sents the parameters of the corresponding tr nsforming networks w.r.t. T ci nd Ti, respec-58 tively. Then, we aggregate T̂ ci and T̂i y conducting global average-pooling and cor espo dingly59 obtain the atte tive local-level f atures xci , as well as the global-level fe tur x global i . The holisti60 fe ture representation w.r.t. the input image Ii is achieved b concatenating both xci and xgl bali , i.e.,61 xi = h xci ; x global i i = F (Ii;⇥) 2 Rd. Note hat, we hereby abstract the aforementioned fin -grained62 featur learning process as a function F (Ii;⇥) with its parameters ⇥.63
2
…
2 Related Work23 2.1 Fine-Grained Imag Retrieval24
2.2 Learning to Hash25
2.3 Visual Attributes26
3 Methodology27
In this secti n, we int oduce the overall framew rk and otations of the p oposed A2-NET model, as28 well as elaborating the detailed modules of A2-NET a its corresp nding ptimiza ion algorithm.29
3.1 Overall Framework and No ations30
As illustrated in Figure ??, our A2-NET model consists of two crucial modules, i.e., a fine-grained31 representation learning module and an attribute-aware hash codes generating module. Given an32 input image Ii, based n its corr po ding deep activa io ten o Ti 2 RC⇥H⇥W extracted by a33 backbone CNN, a set of attention guida ce A = {Ac} i learned for capt ring fine-grained t ilored34 local patterns T ci from Ti. To distill semantical cues and further generate the final attribute-aware35 bi ary has c des, we pro ose to transfo m these ppearan e-specific f atures T owards semantic-36 sp cific representations T̂ by performing a transform etwork (·). Aft r aggre ating T̂ , h obtained37 attentive local-level features xi ar associated with t e gl bal-level feature x global i to form as a38 holistic feature representation xi. In order to ge erate attribute-aware binary hash codes, we conduct39 a reconstructing paradigm to projec xi a vi in an attribute space wh re ts data point correspond to40 an attribute vector w.r.t. one kind of namable properties of fine-grained obj cts. Finally, hash code41 learning is performed upon vi to obtain the final attribute-aware binary codes ui.42
3.2 Fine-Grained Representation Learning43
Att ntion plays an important role in hu an perception [3, 7], nd humans exploit a sequenc of44 partial glimpses and selectively focus on salient parts of an object or a scene in order to better capture45 visual structure [10]. Inspired by this, we incorp rate the attentio mechanism into represent tion46 lear ing to capture fine-grain d local patt rns for distinguishing subtle differences between these47 subordinate categories.48
In concretely, we extract the deep feature of its input image Ii via a backbone CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Then, based on Ti, C attention guida c Ac 2 RH⇥W is gen rat d as set of attention maps, i.e., A.51 The ttent on guidance Ac s designed to ev luate which deep descriptors should be attended or even52 overl ked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obt in the final attribute-aware binary codes, it is54 desirable to transform these appeara ce-specific (i.e., low-level) features T to semantic-specific (i.e.,55 mid-l vel) repres ntations which are closer to the attribute space. Thus, a transforming network (·),56 which is equipped with a stack of convolution layers, is performed on T as follows:57
T̂ ci = (T c i ; ✓local) , (3)
T̂i = (Ti; ✓global) , (4)
where ✓ prese ts the parameters of the corresponding transforming networks w.r.t. T ci and Ti, respec-58 tively. Then, we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentiv local-level f atures xci , as well as the global-level feature x global i . Th holistic60 feature representation w.r.t. the input image Ii is achieved by concatenating both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F (Ii;⇥) 2 Rd. Note that, we hereby abstract the aforementioned fine-grained62 feature learning process as a function F (Ii;⇥) with its parameters ⇥.63
2
2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section, we introduce the overall framework and notations of the proposed A2-NET model, as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29 3.1 Overall Framework and Notations30 As illustrated in Figure ??, our A2-NET model consists of two crucial modules, i.e., a fine-grained31 representation learning module and an attribute-aware hash codes generating module. Given an32 input image Ii, based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN, a set of attention guidance A = {Ac} is learned for capturing fine-grained tailored34 local patterns T ci from Ti. To distill semanti al cu s d further generate the final attribute-aware35 binary hash codes, we propose to transform these appearance-specific features T towards semantic-36 specific representations T̂ by performing a transform network (·). After aggregating T̂ , the obtained37 attentive local-level features xci are associated with the global-level feature x global i to form as a38 holistic feature representation xi. In order to generate attribute-aware binary hash codes, we conduct39 a r constructing paradigm to project xi as vi in an attribute sp ce where its data point correspond to40 an attribute vector w.r.t. one kind of namable properties of fine-grained objects. Finally, hash code41 lear ing is perfor ed upon vi to obtain the final attribute-aware binary codes ui.42 3.2 Fine-Grained Representation Learning43 Attention plays an important role in human percept on [3, 7], and humans exploit a sequence of44 partial glimpses and selectively focus on salient parts of an object or a scene in order to better capture45 visual structure [10]. Inspired by this, we inc rporate t e attention mechanism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48 In concretely, we extract the deep feature f its i put im ge Ii via a ba kbone CNN model CNN(·)49 as50
Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Then, based on Ti, C attention guidance Ac 2 RH⇥W is generated as a set of ttention maps, i.e., A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obtain the final attribut -aware binary codes, it is54 desirable to tran for these appearance-specific (i.e., low-level) features T to semantic-spec fic (i.e.,55 mid-level) representations which are closer to the attribute space. Thus, a transforming network (·),56 which is equipped with a stack of convolution layers, is performed on T as follows:57
T̂ ci = (T c i ; ✓local) , (3)
T̂i = (Ti; ✓global) , (4)
where ✓ pr sents the parameters of he c rrespo ding transforming networks w.r.t. T ci a d Ti, respec-58 tively. Then, we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . Th holistic60 feature rep esentation w.r.t. the input image Ii i achieved by concat ati g both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F (Ii;⇥) 2 Rd. Note that, we hereby abstract the aforem ntioned fine-grained62 f ature le ning pro ess as a function F (Ii;⇥) with its parameters ⇥.63
2
Aggregation …
2 Related Work23 2.1 Fine-Grained Image Retri val24 2.2 Le rning to Hash25 2.3 Visual Attributes26 3 Method logy27
In this section, we introduce the ove ll framework nd notati ns of the propos A2-NET model, as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29
3.1 Over l Fra ework and Notations30
As illustrated in Figur ??, our A2-NET model consist of two crucial modules, i.e., a fin -gr ined31 representation learning module and a attribute-aware hash c des ge erating mo ul . Given a32 input image Ii, based on its corres onding de p activatio tens r Ti 2 RC⇥H⇥W extracted by a33 back o e CNN, a set of ttention guidance A = {Ac} is learn d f capturing fine-grained tailored34 local patterns T ci from Ti. To distill semantical cues and further gen rate the fi al attribute-aware35 binary hash cod s, we propose to tr nsform these ppearance-specific features T towards semantic-36 specific repres ntations T̂ by performing a transform network (·). After aggregating T̂ , the obtained37 attentive local-level features xci are associated with the global-level featur x global i to form as a38 holistic e ture r pr s ntation xi. In order to g nerate attribute-aware binary hash cod s, we conduct39 a econstructing p adigm to project xi as vi i an attribute space wh re it dat point correspond to40 an ttribute tor w.r.t. o e kind f n m ble properties of fin -grained objects. Finally, hash code41 le rning is p rformed pon vi to obtain th final attribute- ware bi ary codes ui.42
3.2 Fi e-Grained Representation Le rning43
Attention plays an important role in human perception [3, 7], and humans exploit a sequence of44 partial glimpses and selectively focus on sali nt parts of an object or a scene in or e to b tt r capture45 visual structur [10]. Inspired by this, we incorporate the attention mechanism into representation46 learning to capture fin -grained local patterns for distinguishing subtle differences between these47 s bordi te tegories.48
In concretely, we extract the deep feature of its input image Ii via a backbone CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Th n, based o i, C attention uidan e Ac 2 RH⇥W is g nerated as a set f att ntion m ps, i.e., A.51 The atte ion i ance Ac is designed to evaluate which deep descriptors should b attended or even52 overlooked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obtain the final attribute-aware binary codes, it is54 desirable to tran form these appearance-specific (i.e., low-level) features T to semantic-specific (i.e.,55 mid-level) representations whi h are closer to the attribute space. Thu , a transforming network (·),56 which is equipped with a stack of convolution layers, is performed on T as follows:57
T̂ ci = (T c i ; ✓local) , (3)
T̂i = (Ti; ✓global) , (4)
wher ✓ presents the parameters of the corresponding transforming networks w.r.t. T ci and Ti, respec-58 tively. Then, we ggrega e T̂ ci and T̂i by c n ucting global average-pooling and correspondingly59 o tain the attentive local-level featur s xci , as well a the global-l vel feature x global i . The holistic60 feature representation w.r.t. the i put mage Ii is chi ved by concatenating both xci and xglobali , i.e.,61 xi = h xci ;
l al = F (Ii;⇥) 2 Rd. Note that, we hereby abstract the aforementioned fine-grained62 feature learnin proc ss as a function F (Ii;⇥) with its parameters ⇥.63
2
2 Rel t d Work23 2.1 Fi -Grained Image Retrieval24
2.2 Learni g to Ha h25
2.3 Visu l Attributes26
3 Methodology27
In this section, we introduce the overall framework and notations of the proposed A2-NET model, as28 w ll as elaborating the detailed modules of A2-NET and its correspon ing optimization algorithm.29
3.1 Overall Fram work and Notation30
As illustrated in Figur ??, our A2-NET model consists of two cruc al modules, i.e., a fi e-grained31 r p tatio l arning modul nd an attribut - ware has codes generating module. Given an32 inp t image Ii, b sed o its respond ng deep a tivation tensor Ti 2 RC⇥H⇥W extracted by a33 ba kbone CNN, a set of attention g idance A = {Ac} is learned for capturing fine-grained tailored34 l l patterns T ci from Ti. To distill semantical cues and further en rate the final attribute-aware35 binary hash odes, we prop se to transform these ppearance-specific features T towards semantic-36 specific representations T̂ by performing a transform netwo k (·). After aggregating T̂ , he obt ined37 attentive local-level features xci re associated with the global-level feature x global i to form as a38 holistic feature representation xi. In order to generate attribute-aware binary hash codes, w conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond to40 an attribute vector w.r.t. one kind of namable properties of fine-grained objects. Finally, has c de41 learning is performed upon vi to obtain the fi al attribute-aware binary codes ui.42
3.2 Fine-Grained Representation Learning43
Attention plays a important role in human perception [3, 7], and humans exploit a sequence of44 partial glimpses and select vely focus on salient parts of an bject or a scene in order to be ter capture45 visual structure [10]. Inspired by this, we incorporate the attention mechanism into representation46 learning to c pture fine-grained local patterns for distinguishing subtle differences b tween th se47 subordinate c tegories.48
In concretely, we extract the deep feature of its input image Ii via a backbone CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Then, based on Ti, C attention guidance Ac 2 RH⇥W is generat d as a set of attention maps, i.e., A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be tte d d or even52 overlooked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obtain the final attribute-aware binary codes, it is54 desirable to transform these appearance-specific (i.e., low-level) features T to semantic-specific (i.e.,55 mid-level) representations which are closer to the attribute space. T us, transf rming netw rk (·),56 which is equipped with a stack of convolution layers, is performed on T as follows:57
T̂ ci = (T c i ; ✓local) , (3)
T̂i = (Ti; ✓global) , (4)
where ✓ presents the parameters of the corresponding transformi g networks w.r.t. T ci and Ti, r spec-58 tively. Then, we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . The holistic60 feature representation w.r.t. the input image Ii is achieved by concatenating both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F (Ii;⇥) 2 Rd. Note that, we her by abstr c the af rementioned fine-gr ined62 feature learning process as a function F (Ii;⇥) with its parameters ⇥.63
2
2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section, we introduce the overall framework and notations of the proposed A2-NET model, as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorith .29 3.1 Overall Framework and Notations30 As illustrated in Figure ??, our A2-NET model cons sts of two crucial modules, i.e., a fine-grain d31 representation learning module and an attribute-aware hash codes gene ating module. Given an32 input image Ii, based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN, a set of attention guidance A = {Ac} is learned for capturing fin -grained tail red34 local patterns T ci from Ti. To distill semantical cues and further generat the final at ribute-aware35 binary hash codes, we propose to transfor th se appearance-specific features T towards semantic-36 specific representations T̂ by performing a transform network (·). Aft r aggregating T̂ , the obtained37 attentive local-level features xci are asso iated with the global-level feature x global i o form as a38 holistic featur representation xi. In ord r to generate attribute-awar binary hash codes, we conduct39 a reconstructing paradigm to project xi as vi in an attribute space wh re its data point corr spond to40 an attribute vector w.r.t. one kind of namable properties of fine-grained objects. Finally, hash code41 learning is performed upon vi to obtain the final attribute-aware binary codes ui.42 3.2 Fine-Grain d Representation Learning43
Attention plays an important role in human perception [3, 7], and humans exploit a sequence of44 partial glimps s and s lectively focus on s lient parts of an object or a scene in order to better capture45 visual structure [10]. Inspired by this, we incorporate the attention mechanism into repr sentation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48
In concretely, we extract the deep feature of its nput image Ii via a backbone CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Then, based on Ti, C attention guidance Ac 2 RH⇥W is generated as a set of attention maps, i.e., A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadama d product. T obtain he final a tribute-aware binary cod s, it is54 desirable to transform these appearance-specific (i.e., low-level) features T to semantic-specific (i.e.,55 mid-level) representations which are closer to the attribute space. Thus, a transforming network (·),56 which is equipped with a stack of convolution layers, is performed on T as follows:57
T̂ ci = ( i ; local) , (3)
T̂i = (Ti; ✓global) , (4)
where ✓ presents the parameters of the corresponding transforming networks w.r.t. T ci and Ti, respec-58 tively. Then, we aggregate T̂ ci and T̂i by onducting glob l average-poolin and correspondingly59 obtain the attentive local-level features xci , as w ll as the gl bal-level feature x global i . Th holistic60 feature representation w.r.t. the input image Ii is achiev d by concat nating both xci a d xglobali , i.e.,61 xi = h xci ; x global i i = F (Ii;⇥) 2 Rd. Note that, we hereby abstract the aforementioned fine-grained62 feature learning process as a function F (Ii;⇥) with its parameters ⇥.63
2
Aggregation
where t e d coder (i.e., a counterpart of the encoder) is realized by W> to simplify the netwo k.91 However, directly minimizing Eq. (6) with a hard constraint is hard to optimize. Therefore, we relax92 the constraint into a soft constraint, and the learning objective can be rewritten by93 min W kX W>V 0k2F + kWX V 0k2F . (7) 3.3.2 Attribute-Specifi F ur Decorrelation94 By conducting the aforementioned unsupervised attribut -guided learning, we can obtain the internal95 l tent vectors V 0 as the attribute-specific features. I order to both enhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features, we introduce a feature97 decorrelation c n r int upon V 0, which is formulat d by98
mi V 0 kV 0V 0> nIk2F , (8) where I is the identity matrix. Such a feature ecorrelation co straint is pref rable to construct99 independent features and reduce redundant information. Therefore, based on unsupervised attribute-100 guided learning and attribute-specific feature decorrelation, the final learned hash codes are expected101 to be bo h attribute-aware and hash-bit ind pend nt.102
3.3.3 H sh Co e L arning103
In the f llowing, we conduct the hash code learning based on the obtained attribute-specific features.104 Assume that w have n qu ry data p ints which are denoted as {qi}i=1, as w ll a m database points105 which a noted as {vj}j=1. Note hat, both qi and vi are belo gin to the attribute space V . Then,106 the corresponding binary has codes can be obt ined via107
ui = sgn(tanh(qi)) , (9) zj = sgn(tanh(vj)) , (10)
where ui, zj 2 { 1, +1}k. The goal of our hash code learning is to learn binary hash codes for both108 qu ry poin s and database points from {qi}ni=1, {vj}mj=1, and the pairwise supervised information,109 i.e., S 2 { 1, +1}n m. To pr se ve the pairwise similarity, adopt the `2 loss between the110 supervised inf rmati ( .k.a. similarity) d t inner product f qu ry-database point binary code111 pairs. It can be formulated as follows:112
minU ,Z Pn
i=1 Pm j=1 u>i zj kSij 2
s.t. U 2 { 1, +1}n⇥k, Z 2 { 1, +1}m⇥k . (11)
Overall, e g t the fina obj tiv of the propo ed A2-NET model by considering Eq. (7), Eq. (8) and113 Eq. (11) toge a follows:114
min W ,⇥
L(I) = kX W>V 0k2F + kWX V 0k2F +↵kV 0V 0> nIk2F + nX
i=1
mX
j=1
> i zj kSij 2 ,
(12) where , ↵ and are hyper-parameters as the trade-off.115
In practice, during training, it might be only available a set of database poin s {vj}mj=1 without query116 points. Thus, we randomly sample n data points from database to construct a query set, and denote117 the indices of all the database points as with the indices of the query set as ⌦. Additionally, because118 we cannot back-propagate the gradient to ⇥ due to the sgn(·) function, we omit the sgn(·) function119 and o ly apply t nh(·) for elaxation in Eq. (10) of the whole optimization process. Therefore, the120 optimization formulation of A2-NET can be rewritten with only database points {vj}mj=1 for training121 as:122
min W ,⇥
L(I) = kX W>V 0k2F + kWX V 0k2F + ↵kV 0V 0> nIk2F
+ X
i2⌦
X
j2
⇣ tanh(W · F (Ii;⇥))>zj kSij ⌘2 . (13)
For optimization, our A2-NET does not require co plicated two-stage learning algorithms, e.g., the123 alternative optimization strategy. We employ the back-propagation algorithm and follow [8] to train124 the whol A2-NET model in a u ified end-to-end manner.125
4
“Blue he d”
Figure 1: Overall framewo k of the proposed A2-NET model, which consists of two crucial modules, i.e., fine-grained representation learning and attribute-aware hash codes generating. The whole netw rk c n be end-to-e d trainabl , d is generally riven by the nsupervi ed attribute-guided loss, the feature decorrelation loss and he hash cod learn g loss, cf. S ction 3.3.
has th potenti l to help with large-sc le fine- ra ed d ta t o. Therefore, targeting the ig data35 challenge, fine-grained hashing [7, 13] is a promisi g direction orthy of further explorations.36
low-level to mid-level appearance-specific to se antic-specific37
obtain corresponding fine-grai ed obj ct attributes (e.g., xx), without significant annotation cost38
2 Relat d Work39
In this section, we briefly review t e r lated work i the following t ree aspects, i.e., n -g ained40 image retrieval, learning to hash and the studies about visual attributes in computer vision problems.41
2.1 Fine-Grained Image Retrieval42
Fine-grained image retrieval as an integral part of fine-grained image analysis [34] has gained more43 and more tractio i recent years [7, 20, 23, 33, 35, 37, 38]. What makes it challenging is that44 objects of fine-grained images have only subtle differences, and often largely vary in pose, scale, and45 orientation or can exhibit cross-modal differences (e.g., sketch-based retrieval [20]).46
Depending on the type of query image, the most studied areas of fine-grained image retrieval47 can be separated into two groups: fine-grained content-based image retrieval (FG-CBIR) and fine-48 grained sketch-based image retrieval (FG-SBIR). More specifically, in FG-CBIR, unsupervised49 learning based [33] and supervised learning based methods [36–38] were developed from different50 perspectives for dealing with fine-grained retrieval tasks, e.g., localizing fine-grained parts [33],51 enhancing intra-class separability with inter-class compactness [38], and reducing the confidence52 of the fine-grained predictions [36], etc. While, FG-SBIR needs to not only capture fine-grained53 characteristics present in the sketches, but also possess the ability to traverse the sketch and image54 domain gap. In the literature of FG-SBIR, the earlier works, e.g., [17, 27, 35], were mostly based55 on Siamese-triplet networks [3] to tackle the aforementioned challenges. Recently, some works56 tried to incorporate the advances of recent progress in self-supervised learning [5] and attention57 mechanisms [6] for further improving the retrieval accuracy of FG-SBIR, e.g., [23, 25].58
However, although these fine-grained retrieval methods achieved good results, they still show the59 limitations in the face of large-scale data, i.e., the searching time for exact nearest neighbor is typically60 expensive or even impossible for the given queries. To alleviate this issue, fine-grained hashing, which61 aims to generate compact binary codes to represent fine-grained images, as a promising direction62 has attracted the attention in the fine-grained community very recently [7, 14]. More specifically,63
2
R d_head?
Blue_tail?
On_ he_ r e?
Spotted_body?
YES
NO
YES
YES
1
0
1
1
Att ibute-aware hash code of
Retrieval
Top-k retrieved images
1
0
1
1
1
0
1
1
1
0
1
0 Query image
101 1
Encoder
3.3 ttribute-Aware Hash Codes G nerating64 How to generate attribute-aware hash codes is the key of our A2-NET model. We elaborate it in65 the following four aspects, i.e., u supervised attribute-guided learning, attribute-specifi feature66 decorrelation, hash code learning and the optimization learning algorithm.67 3.3.1 Unsupervised Attribute-Guided Le rning68 In r al-applications, especially for the larg -scale and fine-grained tasks, attribut annotations are69 always infeasible, which limits the learning process to be conducted in an unsupervised setting.70 While, in the literature, th main goal of nsupervised learning is to capture regularities in data for the71 purpose of xtracting useful rep es n tio s or for restoring c rupted data [4]. Many unsupervi ed72 methods explicitly produce int rnal lat t units or codes, from which the data is to be reconstructed.73
Inspired by this, we develop an unsu rvised attribute-guided learni component to proj ct the74 holistic representation xi of Ii into a latent space, i.e., the attribute space V . In V , its high-level75 v ctors are designed to have certain desirable properties, e.g., corresp nding to semantic properti s76 of fine-grained objects (a.k.a. “fine-grained attributes”).77
More specifically, in our A2-NET, the unsupervised attribute-guided learning is realized by a recon-78 struction paradigm with n encoder-de oder structure, as shown in Figure ??. In concretely, given79 a batch of n tr ining data Ii, the holis ic representations X = {x1; x2; . . . ; xn} 2 Rd⇥n can b80 btained as aforemention d. By formulation, the encoder projects X into the attribute space V with81 a projection matrix W 2 Rk⇥d to get an internal latent representation V 2 Rk⇥n w.r.t. X . In82 particul rly, we set tha the dimension of latent representation k equals to th number of hash bits in83 the final binary hash code ui. Furt rmore, each lumn of V , i. ., vi 2 Rk, can derive ui by84
ui = sgn(tan (vi)) . (5)
Meanwhile, regarding vi, we also consider to reconstruct its input xi by a decoder as a counterpart85 of the encoder. Therefore, o one hand, such a reconstruction paradigm can reduce and further distil86 high-lev l semantic cues in the attribute space V . While, on the other ha d, it can drive the training87 of A2-NET by preserving the similarity between queried hash codes and database points in terms of88 learning to hash (cf. Section 3.3.3).89
In concretel , the learning objective of unsuperv sed attribut -guided l arning is written as follows:90
min W
kX W>WXk2F s.t. WX = V 0 = tanh(V ) , (6)
where the decoder (i.e., a counterpart of the encoder) is realized by W> to simplify the network.91 However, directly minimizing Eq. (6) with hard constraint is hard to ptimize. Therefore, we r lax92 the constraint into a soft constraint, and the lear ing objective can be rewritten by93
min W
kX W>V 0k2F + kWX V 0k2F . (7)
3. .2 Att ibute-Specific Fe tu e Decorrel ti n94
By conducting the aforementioned unsupervised attribut -guided le rning, we can obtain the internal95 latent vectors V 0 as the attribute-specific features. In order to both enhance the discriminative ability96 and re ove the redundant correl tion amo g these attribute-specific features, we introduce a feat r97 correlation co straint upo V 0, which is formulated by98
min V 0
kV 0V 0> kIk2F , (8)
whe e I is the identity matrix. Such feature decorr lation constraint is pref rable to construct99 independent features and reduce r dundant information. Therefore, based on un upervised attribu -100 guided learning and attribute-specific feature decorrelation, the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102
3.3.3 Hash Cod Le rning103
In the following, we conduct the hash code learning based on the obtained attribute-specific features.104 Assume that we have n query data points which are denoted as {qi}n=1, as well as m database points105
3
3.3 Attribute-Aware Hash Codes Ge rating64 How to generate attribute-aware hash codes is he key of our A2-NET model. We elaborate it in65 the f llowing four aspects, i.e., unsupervised attribute-guided learning, a tribute-spec fic feature66 decorrelation, hash code learning and the optimizatio le rning algorithm.67 3. .1 Unsupervised Attr bute-Guided Learning68 In real-applica ions, especially for the larg -scal and fine-grained tasks, attribute annotations are69 always infeasible, which limits th learnin p ocess to be conducted in an unsupervised setting.70 While, in the litera ure, the main goal of unsuperv sed learning is to capture regu arities in data for the71 purpose of extracting us ful representations or for restoring corrupted data [4]. Many unsupervised72 meth ds explicitly produce internal latent units or codes, from which the data is to be reconstr ct d.73
Inspired by this, we develop an unsupervised attribute-guided learning component to project the74 holistic rep sentation xi of Ii int a latent space, i.e., the attribute sp ce V . In V , i s high-level75 vectors are designed to have certain desirable properties, e.g., corresponding to semantic properties76 of fine-grained objects (a.k.a. “fi e-gr ined attribute ”).77
More specifically, in our A2-NET, the unsupervis d attribute-gu d learning is realized by a recon-78 struction paradigm with an encoder-decoder structure, as shown in Figure ??. In concretely, given79 batch of n training data Ii, the holistic representations X = {x1; x2; . . . ; xn} 2 Rd⇥n can be80 obtained as aforementioned. By formulati n, th en oder pr jects X into the attribute sp ce V with81 a projection matrix W 2 Rk⇥d to e an internal l tent repre entation V 2 Rk⇥ w. .t. X . I82 parti ularly, we set that the dimension of latent representation k equals to the nu ber of hash bit in83 the final binary hash co e ui. Furthermore, ach column of V , i.e., vi 2 Rk, ca d rive ui by84
ui = sgn(tanh(vi)) . (5)
Meanw ile, regarding vi, we also consider to reconstruct i s input xi by a decoder s a counterpart85 of the encoder. Therefore, on one h nd, such a reconstructi n para igm can redu e and furth r distill86 high-level semantic cues in the attribute space V . While, on the other h nd, it can driv t e training87 of A2-NET by preserving the similarity between queried hash cod s and database points in terms of88 learning to hash (cf. S ction 3.3 3)89
In co cretely, the learning objective of unsupervised attribute-guided learning is written as foll ws:90
min W
kX W>WXk2F s.t. WX = V 0 = tanh(V ) , (6)
where the decoder (i.e., a counterpart of the encoder) is realized by W> to simplify the network.91 How ver, directly minimizing Eq. (6) with a ha d constraint i hard t optimize. Therefore, we relax92 the constraint into soft constraint, and the lear ing objective can be rewritten by93
min W
kX W>V 0k2F + kWX V 0k2F . (7)
3. .2 Attribute-Specific Feature Decorrelation94
By conducting the aforementioned unsupervised attribute-guided learning, we can obtain the internal95 latent vectors V 0 as the attribute-specific features. In order to bo nhance th discri inative ab lity96 and remov the r dundant cor elation amo thes ttribute-specific features, we introduce a feature97 decorrelation constraint up V 0, which is formulated by98
mi V 0
kV 0V 0> kIk2F , (8)
where I is the i entity matrix. Such a f ature d correlation constraint is prefer ble to construct99 independent features and reduce redundant information. Therefore, based on unsupervised attribute-100 guided learning and attribute-specific feature decorrelation, the final learned hash codes are expected101 to be both attribute-aware and hash- it independent.102
3.3.3 Has Co e Learning103
In the following, we conduct the hash code learning based on the obtained attribute-specific features.104 Assume that we h ve n query dat poi ts which are denoted as {qi}ni=1, as well as m database points105
3
3.3 Attribute-Aware Hash C s Gene ating64
How to generate attribute-aware hash codes is the key of our A2-NET model. We elaborate it in65 the following fou asp c s, .e., unsup rvis attribute-guided le rni g, at ribute-specific feature66 decorrelati n, h sh code learning and the o timization earni g algorithm.67
3.3.1 Unsupervised Attr bute-Gui ed Learn ng68
In real- pplications, specially for the large-scale and fine-grai ed tasks, attribute annotations are69 always infeasible, which limits t l arning process to be conducted in an nsupervised setti g.70 While, in the literature, the main goal of unsupervised learn ng is to capture regularities in d ta for th71 purpose of extracting useful representations or for restoring corrupted data [4]. Many unsupervised72 methods explicitly rodu e internal latent units or codes, from which the data is to e reconstructed.73
Inspired by this, we develop an unsupervised attribute-guided learning component to project the74 holistic prese tation xi of Ii into a latent space, i.e., the attribute space V . I V , its high-level75 vectors are designed to have certain desirable properties, e.g., corres onding to semantic properties76 of fine-grained jects (a.k.a. “fine-grained attribut s”).77
More specifically, in our A2-NET, the unsupervised attribute-guided learning is realized by a recon-78 structi n paradigm with an encoder-decod r structure, as shown in Figure ??. In concretely, given79 a batch of n trai ing data Ii, the h listic representations X = {x1; x2; . . . ; xn} 2 Rd⇥n ca be80 obtained as aforementio ed. By formulation, the encoder pr jects X into the attribute sp ce V with81 a projectio mat ix W 2 Rk⇥d to get an nternal latent repres ntation V 2 Rk⇥n w.r.t. X . I82 particularly, w set that the dimension of lat nt repre entation k equals t the number of hash bits in83 the final binary ash c e i. Furthermore, each c lumn of V , i.e., vi 2 Rk, c n deriv ui b84
ui = sgn(tanh(vi)) . (5)
Meanwhile, regarding vi, we also consider to r co st uct its input xi by a decoder as a counterpart85 of the encod r. Therefore, on one hand, uch a reconstruction paradigm can duce and further distill86 high-level semantic cues in the attribute space V . While, on the other hand, it can drive the training87 of A2-NET by preserving the similarity between queried hash codes and database points in terms of88 learning to hash (cf. Section 3.3.3).89
In con retely, the learn ng obj ctive of u sup rvised at ribute-guided learning is written as foll ws:90
min W
kX W>WXk2F .t. WX = V 0 = tanh(V ) , (6)
where the decoder (i.e., a counterpart f he encoder) is r alized y W> to simplify the network.91 However, directly minimizing Eq. (6) with a hard con traint is hard to optimize. Therefore, we re ax92 the c nstraint into a soft co straint, and the learning objectiv can be re ritt n by93
min W
kX W>V 0k2F + kWX V 0k2F . (7)
3.3.2 Attribute-Specific Feature Decorrela ion94
By conducting the aforementioned unsupervised attr bute-gu ded learning, we can obtain the inte nal95 l tent v ctors V 0 as th ttribute-specific features. In order to both enhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features, we introduce a feature97 decorrelation constraint upon V 0, which is formulated by98
min V 0
kV 0V 0> kIk2F , (8)
where I is the identity matrix. Such a feature decorrelation constraint is preferable to construct99 independent features and reduce redundant information. Therefore, based on unsupervised attribute-100 guided learning and attribute-specific feature decorrelation, the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102
3.3.3 Hash Code Learni g103
In the following, we c nduct the hash code learning base n the btained attribute-spe ific features.104 Assume that we ve query ata points which are denoted as {qi}ni=1, as well as m database points105
3
3.3 Attribute-Aware Hash Codes Generating64
How to generate attribute-aware hash codes is the key of our A2-NET model. We elaborat it in65 the following four aspects, i.e., unsupervised attribute-guided learning, attribute-specific feature66 decorrel tion, h sh code learning and the optimization learning algorithm.67
3.3.1 Unsupe vised At ribute-Guided Learnin68
In real-applications, specially for the large-sca e and fine-gr ined tasks, attribute an tation re69 always infeasible, which limits the learning process to be nducted in an unsuperv sed setting.70 While, i the literature, the ma goal of unsupervis d learning is to capture regularities in data for the71 purp se of extracting useful representations or f r restoring corrupted data [4]. Many unsupervise72 methods explicitly oduce int r al latent units or codes, from which the d ta is to be reconstructed.73
Inspired by this, we develop an unsupervised attribute-guided learning component to project the74 holistic representation xi of Ii into a latent space, i.e., the attribute sp ce V . In V , its high-lev l75 vectors are designed to have certain desirable properti s, .g., corresponding to semantic properties76 of fine-grained objects (a.k.a. “fine-grained attributes”).77
More specifically, in our A2-NET, the unsupervis attribute-guided l arn ng is realized by a recon-78 structio paradigm with an enco er-decoder structure, as shown in Figure ??. I concretely, given79 batch f n training data Ii, the holistic representatio s X = {x1; x2; . . . ; xn} 2 Rd⇥n ca be80 obtained as forem ntioned. By formulation, the encoder projects X into the attribute space V with81 a pr jection matrix W 2 Rk⇥d to get an intern l l tent represent on V 2 Rk⇥ w.r.t. X . In82 particularly, we set that the dimension of latent representation k equals t the number of hash bits in83 the final binary hash code ui. Furthermore, each column of V , i.e., vi 2 Rk, can derive ui by84
ui = sgn(tanh(vi)) . (5)
Meanwhile, regarding vi, w also consider to reconstruct its input xi by a decoder s a counterpart85 of the encoder. Therefore, on one hand, such a reconstruction paradigm can reduce and further distill86 high-level semantic cues in the attribute space V . While, on th other hand, it can drive the training87 of A2-NET by preserving the similarity between qu ried hash codes and database oints in terms of88 learning to hash (cf. Section 3.3.3).89
In concretely, the learning objective f unsupervis d attribut -guided learning is written s follows:90
min W
kX W>WXk2F s.t. WX = V 0 = tanh(V ) , (6)
where the decoder (i.e., a counterpart of the encoder) is realized by W> to simplify the network.91 However, directly minimizing Eq. (6) with a hard constraint is hard to optimize. Therefore, we relax92 the constraint into a soft constraint, and the learning objective ca be rewritten by93
min W
kX W>V 0k2F + kWX V 0k2F . (7)
3.3.2 Attribute-Specific Feature Decorrelation94
By con ucting the aforementioned unsupervised attribute-guided learning, we ca obtain th internal95 latent vectors V 0 as the attribute-specific features. In order to both enhance the discriminative bility96 and remove the redundant correlation among these attribute-specific features, we introduce a feature97 decorrelation constraint upon V 0, which is formulated by98
m V 0
kV 0V 0> kIk2F , (8)
where I is the identity matrix. Such a feature decorrelation constraint is pref rable to construct99 independent features and reduce redundant inf rmation. T erefo e, based on unsupervise ttri t -100 guided learning and attribute-specific feature decorr lation, the final learned hash codes are expected101 to be both attribute-aware and hash-bit indep ndent.102
3.3.3 Hash Cod Learning103
In the f llowing, we conduct the hash code learning based on the btained attribute-sp cific feat res.104 Assume that we have n query data points which are denoted as {qi}ni=1, as well as m database points105
3
Dec der
3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET model. We elaborate it in65 the following four a pects, i.e., unsupervised attribute-guided learning, attribute-spec c feature66 decorrelation, hash code learning and the optimization learning algorithm.67 3.3.1 Unsupervised Attribute-Guided Lear ing68 In real-applications, especially for the large-scale and fine-grained tasks, attribute annotations are69 always infeasible, which limits the lear ing proc ss to be conducted in an u supervised setting.70 While, in the literatur , the main goal of u supervised l arning is to captu e regularities in data f r the71 purpose of extra ting useful repre entat ons or for rest ring orrupted da a [4]. Many unsupervised72 methods explicitly produce internal latent units or codes, from which the data is to be rec nstructed.73 Inspired by this, we develop an unsupervised attribute-guided learning component to project the74 holistic representation xi of Ii into a latent spac , i.e., the att ibut space V . In V , its high-level75 vectors are designed to have certain d irable properties, e.g., corresponding to semanti properties76 of fine-grained objects (a.k.a. “fin -grained attributes”).77 More specifically, in our A2-NET, the unsupervised attribute-guided learning is re lized by a recon-78 struction paradigm wi h an encoder-decoder structure, as shown in Figure ??. In concretely, given79 a batch of n ra ning data Ii, th holistic repres ntatio s X = {x1; x2; . . . ; xn} 2 Rd⇥n can be80 obtained as aforementioned. By formulation, the encoder projects X into the attribute sp ce V with81 a projection matrix W 2 Rk⇥d to get an internal atent representation V 2 Rk⇥n w.r.t. X . In82 particularly, we set that the dimension of latent representation k equals to the number of hash bits in83 the final binary hash code ui. Furthermore, each col mn of V , i.e., vi 2 Rk, can derive ui by84
ui = sgn(tanh(vi)) . (5)
Meanwhile, regard ng vi, we lso consi r to reco struct its input xi by a decod r as a counterpart85 of the nco er. Therefore, on one ha d, such a rec nstruction paradigm can reduce and furthe distill86 high-level semantic cues in the attribute sp ce V . While, on the other hand, it can drive the training87 of A2-NET by preserving the similarity between queried hash codes and database points in terms of88 learning to hash (cf. Sectio 3.3.3).89
In concretely, th lear i g objective of unsupervised attribute-guided learni g is written as follows:90
min W
kX W>WXk2F s.t. WX = V 0 = tanh(V ) , (6)
where the decoder (i.e., count rpart of t e encod r) s realize by W> to simplify the net k.91 How ve , directly minimizing Eq. (6) with hard constraint is hard to optimize. Therefore, we relax92 the con traint into a oft constraint, nd the learning objective can be rewritten by93
mi W
kX W>V 0k2F + kWX V 0k2F . (7)
3.3.2 Attribute-Specific Feature Decorrelation94
By con ucting he aforementioned unsupervis attribut -guided lear ing, we can obtain the internal95 lat nt vectors V 0 as the ttribute-specific features. In order to both enhance the discriminative bility96 and remove the redundant correlation among these attribute-specific features, we introduce a feature97 decorrelation constraint upon V 0, which s formulated by98
min V 0
kV 0V 0> kIk2F , (8)
where I is the dentity matrix. S ch a featur decorrelation constraint is pref able to construct99 independent f tures a d r duce redundant informa ion. Therefore, based on unsupervised attr bute-100 guided learning and attribute-specific feature decorrelation, the final learned hash codes are expected101 to be both attribute-aware and hash-bit independ n .102
3.3 3 Hash C e Lear ing103
In the following, we conduct the ash code l a ning based on the obtain attribute-specific fe tures.104 Assume that we have n query data poi ts which are denoted as {qi}ni=1, as well as m database poi ts105
3
which are denoted as {vj}mj=1. Note that, both qi and vi are belonging to the attribute space V . Then,106 the corresponding binary hash codes can be obtained via107 ui = sgn tanh(qi)) , (9) zj = sgn(tan (vj)) , (10) where ui, zj 2 { 1, +1}k. The goal of our hash code learning is to learn binary hash codes for both108 query points and database points from {qi}ni=1, {vj}mj=1, and the pairwise supervised information,109 i.e., S 2 { 1, +1}n⇥m. To preserve the pairwise similarity, we adopt the `2 loss between the110 supervised information (a.k.a. similarity) and the inner product of query-database point binary code111 pairs. It can be formulated as follows:112 minU ,Z Pn i=1 Pm j=1 u>i zj kSij 2 s.t. U 2 { 1, +1}n⇥k, Z 2 { 1, +1}m⇥k . (11) Overall, we get the final objective of the proposed A2-NET model by considering Eq. (7), Eq. (8) and113 Eq. (11) together as follows:114 min W ,⇥ L(I) = kX W>V 0k2F + kWX V 0k2F +↵kV 0V 0> nIk2F + nX i=1 mX j=1 u>i zj kSij 2 , (12) where , ↵ and are yper-param ters as th trade-off.115 In practi e, uri g training, it might be only available a set of atabase points {vj}mj=1 without query116 points. Thus, we randomly sample n data points from database to construct a query set, and denote117 th indices of all the database points as with the indices of the query set as ⌦. Additionally, because118 we cannot back-propagate the gradient to ⇥ due to the sgn(·) function, we omit the sgn(·) function119 and only apply tanh(·) for relaxation in Eq. (10) of the whole optimization process. Therefore, the120 optimization formulation of A2-NET can be rewritten with only database points {vj}mj=1 for training121 as:122 min W ,⇥ L(I) = kX W>V 0k2F + kWX V 0k2F + ↵kV 0V 0> nIk2F + X i2⌦ X j2 ⇣ tanh(W · F (Ii;⇥))>zj kSij ⌘2 . (13) For optimization, we employ the back-propagation algorithm and follow [8] to train the whole123 2-NET m l in an e d-to-end manner.124 3.4 Out-of-Sample Extension125 After traini g A2-NET, the learned model c n be applied for generating binary codes for query points126 in luding uns en query p ints in th training phase. Sp ifically, we can use the following equation127 to generate the binary code for Iq:128
uq = sgn(tanh(W · F (Iq;⇥))) . (14)
4 Experime ts129
In this section, we present the implementation details, empirical settings, main resu ts, ablation studies130 and also s m interesting qualitative a alyses of the learned ttribute-aware binary hash cod s.131
4.1 Datasets132
x0i By following ExchNet [4], our experiments a e conducted on five fine-grained b nchmark datas ts,133 i. ., CUB-200-2011 [16], Aircraft [12], Food101 [1], NABird [15] and VegFru [6]. Specifically,134 CUB-200-2011 is on of t e most p pular us d fine-graine datasets. It contains 11,788 bird images135 from 200 bird pe i s and is officially split into 5,994 image for training with 5,794 images for136 test. Aircraf contains 10,000 im ges spanning 100 aircraft m dels with 3,334 for training, 3,333137
4
Fine-grained repr sentatio learning
Un su
pe rv
is ed
a ttr
ib ut egu id ed lo ss (S ec tio n 3. 3.
1)
Feature decorrelation loss (S ction 3.3.2)
3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET model. We laborate it in65 the foll wing four aspects, i.e., unsupervised attribute-guided learning, attribute-specifi feature66 decorrelation, hash code learning and the optimization learning algorithm.67
3.3.1 Unsupervised Attribute-Guided L arning68
In real-applications, especially for the large-scale and fine-grained tasks, attribu e annotations are69 always infeasibl , which limits the learning process to be conducted in an unsupervised setti g.70 While, in the literature, the main goal of unsupervised learning is to capture regularities in data for the71 purp se of xtracting useful representations or for restoring corrupted data [13]. Many unsupervised72 methods explicitly produce internal latent units or codes, from which the data is to be recon tructed.73 Inspired by this, w develop an unsupervised attr bute-guided learning component to project the74 holistic r presentati n xi of Ii i to a latent space, i.e., the attribute space V . In V , its high-level75 vec ors are designed to h ve certain desirable properties, e.g., cor espo ding to semantic properties76 of fine-grained objects (a.k.a. “fine-grained attributes”).77 More specifically, in our A2-NET, the unsupervised attribute-guided learning is realized by a recon-78 struction paradigm with an encoder-decoder structure, as shown in Figure ??. In concretely, given79 a batch of n training data Ii, the holistic r presentations X = {x1; 2; . . . ; xn} Rd⇥n ca be80 obtained as aforementioned. By formulation, th encoder projects X into the attribute space V with81 a projection matrix W 2 Rk⇥d to get an internal latent representation V 2 Rk⇥n w.r.t. X . In82 particularl , we set that the dimension of lat nt representation k quals to the number of hash bits i83 the final binary hash code ui. Furthermore, each column of V , i.e., vi 2 Rk, can derive ui by84
ui = sgn(tanh(vi)) . (5)
Meanwhile, regarding vi, we als consider to reconstruct its input xi by decoder as a c unterpart85 of the encoder. Therefore, o one hand, such a reconstruction paradigm can reduce and further distill86 high-level s mantic cues in the attribute space V . While, on the other hand, it an drive the training8 of A2-NET by preserving the similarity between queried hash codes and database points in terms of88 learning to hash (cf. Section 3.3.3).89 In concretely, the le rning objectiv of unsupervised attribute-guided learning is written as follows:90
min W
kX W>WXk2F s. . WX = V 0 = tanh(V ) , (6)
where the decoder (i.e., a counterpart of the encoder) is realized by W> to simplify the network.91 Ho ev r, dir ctly minimizing Eq. (6) with a hard constraint is hard to optimize. Therefore, we relax92 the constraint into a soft constraint, and the learning objective can be rewritten by93
min W
kX W>V 0k2F + kWX V 0k2F . (7)
3.3.2 Attribute-Specific Feature Decorrelation94
By conduc ing the af rementioned u supervised attribute-guided learning, we can obtain the internal95 latent vectors V 0 as the attribute-specific features. In order to both nhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features, we introduce a feature97 d correlation constraint upon V 0, which is formulated by98
min V 0
kV 0V 0> nIk2F , (8)
where I is the i entity matrix. Such a feature decorrelation co straint is pref rable t construct99 independent features and reduce redundant information. Therefore, b sed on unsupervise attribute-100 guided learning and attribute-specific feature decorrelation, the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102
3.3.3 Hash Code Learning103
In the foll wing, we conduct the hash code lear ing based on the obtained attribute-specific features.0 Assume that we have n query data points which are denoted as {qi}ni=1, s well as m database points105
3
Hash code learning loss (Section 3.3.3)
Attrib te-aware hash codes generating“Red head” “On the tree” “Spotted body”
2 Related Work23 2.1 Fine-Grained Im ge Retrieval24
2.2 Learning to Hash25
2.3 Visual Attributes26
3 Methodology27
In this section tr duce the overall framework and notations of the proposed A2-NET model, as28 well as elaborating the detailed modul f A2-NET and its corre ponding optimization algorit m.29
3.1 Overall Framework and Notations30
As illustrated in Figure ??, our A2-NET model consists of two crucial modules, i.e., a fine-grain d31 represent tion learni g module and attribute-aware h sh codes generating module. Given a32 input image Ii, based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN, a set of attention guidanc A = {Ac} is learned for capturing fine-grained t ilored34 local patterns T ci from Ti. To distill semantical cues and further generate the fin l attribute-aware35 bi ary hash c des, we prop se to transform these appear nce-specifi features T towards sema tic-36 specific representations T̂ by performing a transform network (·). After aggregating T̂ , the obtained37 attentive lo al-l vel features xci are ssociated with the global-lev l feature x global i to form as a38 holistic feature representation xi. In rder to generate attribute-aware binary hash codes, we conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond o40 an attribute vector w.r.t. one kind of namable properties of fine-grained objects. Fi ally, hash code41 learning is performed upon vi to obtain the final attribute-a are binary codes ui.42
3.2 Fine-Grained Representation L arning43
Attention plays an important role in human perception [3, 7], and humans exploi sequence of44 partial glim ses and s lective y focus on salient parts of an object or s ene in order to bet r cap ure45 visual structure [10]. Inspired by this, we incor orat the attention m chanism into represent ion46 learning to capture fine-grained local patter s for distinguishing subtle differ nces be ween these47 ubordi a e categories.48
In concretely, we extract the de p feature of its input image Ii via a backbo e CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 C⇥H⇥W . (1) Then, based on Ti, C attention guidance Ac 2 RH⇥W is generated as a set of attention maps, i.e., A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be ttend d or even52 overlooked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obtain the final attribu e-aware binary co es, it is54 desirable to transform these appearance-specific (i.e., low-level) f atures T to semantic-specific (i.e.,55 mid-level) representations which are closer to the attribute space. Thus, a transforming network (·),56 which is equipped with a stack of convolution layers, is performed on T as follows:57
T̂ ci = (T c i ; ✓local) , (3) T̂i = (Ti; ✓global) , (4)
where ✓ presents the para eters of the corresponding transfor ing networks w.r.t. T ci and Ti, respec-58 tively. Then, we aggregate T̂ ci and T̂i by conducting global average-pooling and correspon ingly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . The holistic60 feature representation w.r.t. the input image Ii is achieved by concatenating both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F (Ii;⇥) 2 R . Note that, we hereby abstract the aforementioned fine-grai ed62 feature learning process as a function F (Ii;⇥) with its parameters ⇥.63
2
Att ntion guidance
2 Rela ed Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section, we introduce the overall framework and notations of the proposed A2-NET model, as28 well as el borating the detailed modules of A2-NET and its corresp nding optimization algorithm.29
3.1 Overall Framework and Notations30
As illustrated in Figure ??, our A2-NET model consists of two crucial modules, i.e., a fine-graine31 repres ntation learning module and an ttribute-aw re hash codes generating m dule. Given an32 input image Ii, based on its corresponding de p activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN, a set of attention guidance A = {Ac} is le rned for capturing fi -grained tailored34 local patterns T ci from Ti. To distill semantical cues and further generate the final attribute-aware35 binary hash c des, we propos t transform th se app arance-specific feature T towa ds semantic-36 specific representations T̂ by performing a transform network (·). After aggregating T̂ , the obt ined37 attentive local-level f atures xci are associated with the global-level feature x global i to form as38 holistic feature representation xi. In order to gen ra e attribute-aware bin ry hash c des, we conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond to40 an attribute vector w.r.t. one kind of nam bl properties of fine-grained objects. Finally, hash code41 learn ng is perform upon v o obtain the fi al attribute-aware inary cod s ui.42
3.2 Fine-Grained Representation Learning43
Attention plays a imp rtant role in human perception [3, 7], and humans exploit a sequence of44 artial gl mpses d selectively focus on salient part of an bj ct or a scene in or er to b tter capture45 visual structure [10]. Inspired by this, we incorporate the attention mechanism into representation46 learn ng to capture fine-grained local patterns for distingu shing subtle diff rences betw en these47 subordinate categories.48 In concretely, we extract the deep feature of its input image Ii via a backbone CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Then, based on Ti, C attention guidance Ac 2 RH⇥W is g nerated as a set of attention ma s, i.e., A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obtain the final attribute-aware binary codes, it is54 desirable to transform these appearance-specific (i.e., low-level) features T to semantic-specific (i.e.,55 mid-level) representations which are clos r to the attrib te space. Thus, a transforming network (·),56 which is equipped with a stack of convolution layers, is perf rmed on T as follows:57
T̂ ci = (T c i ; ✓local) , (3) T̂i = (Ti; ✓global) , (4)
where ✓ presents the paramet rs of the corresponding transforming networks w.r.t. T ci and Ti, respec-58 tive y. Then, we aggregate T̂ ci and T̂i by condu ting global average-pooling d correspondin ly59 obtain the attentive local-level features xci , as well as the global-level feature x lobal i . The holistic60 feature representation w.r.t. the input image Ii is achieved by concatenati g both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F (Ii;⇥) 2 Rd. Note that, we hereby abstract the aforementioned fine-grai ed62 feature learning proc ss as a function F (I ;⇥) with its parameters ⇥.63
2
2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section, we introduce the overall framework and notati ns of the proposed A2-NET model, as28 well as elaborating the detailed odules of A2-NET and its corresponding optimization algorithm.29 3.1 Overall Framework and Notat ons30
As illustrated in Figure ??, our A2-NET model consists of two crucial modules, i.e., a fine-grained31 representation learning module and an attribute-aware hash codes g nerating module. Given an32 input image Ii, based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN, a set of attention guidance A = {Ac} is learned for capturing fine-gr i ed tailored34 local patterns T ci fr m Ti. To distill semantic l cues and further generat the fin l ttribute-aware35 binary hash codes, we propose to transform these appearance-specific features T towards semantic-36 specific representations T̂ by performing a transform network (·). After aggregating T̂ , the obtained37 attentive local-level features xci are associated with the global-level feature x global i to form s a38 holistic feature representation xi. In order to generate attribute-aware binary hash codes, we conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond to40 an attribute vector w.r.t. one kind of namable properties of fine-grained obj cts. Finally, hash code41 lear ing is performed upon vi to obtain the final attribut -awar binary codes ui.42
3.2 Fine-Grained Represe tatio Learning43
Attention plays an important role in human p rception [3, 7], and humans exploit a sequence of44 partial glimpses and se ectively focus on salient parts of an object or a scene i order to better capture45 visual structure [10]. Inspired by this, we incorporate the att ntion mechanism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48 In concretely, we extract the dee feature of its input image Ii via a backbone CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Then, based on Ti, C attention guidance Ac 2 RH⇥W is generated as a set of attention maps, i.e., A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obtain the final attribute-aware binary codes, it is54 desirabl to transform these a pearance-specific (i.e., low-level) f atures T to semantic-sp cific (i.e.,55 mid-level) representations which are closer to the attrib te space. Thus, a transforming n twork (·),56 which i equipped with a stack of convolution layers, is p rformed on T as follows:57
T̂ ci = (T c i ; ✓local) , (3) T̂i = (Ti; ✓global) , (4)
where ✓ presents the parameters of the corresponding transfor ing networks w.r.t. T ci and Ti, respec-58 tively. Then, we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentive local-level features xci , as w l as the global-level feature x glo al i . Th holistic60 feature representation w.r.t. the input image Ii is achi ved by conc tenating both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F (Ii;⇥) 2 Rd. Note that, we hereby abstract the af rementioned fine-grained62 feature learning process as a function F (Ii;⇥) with its parameters ⇥.63
2
2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section, we introduce the overall framework and notations of the proposed A2-NET mod l, as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29 3.1 Overall Framework and Notations30 As illustrated in Figure ??, our A2-NET model consists of two crucial modules, i.e., a fine-grai d31 representation learning module and an attribute-aware hash odes g ne ating modul . Given an32 input image Ii, based n its corresponding de activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN, a set of attention guidance A = {Ac} is learned for capturing fine-grained tailored34 local patterns T ci from Ti. To distill semantical cues and further generate the final attribute-aware35 bi ary ash c d s, we propose to trans r the e ppearance-specific features T towards semantic-36 s ecific repr s nt t ons T̂ by perform ng transform network (·). After aggregating T̂ , th o tained37 attentive local-level features xci are asso iated with the global-level f ature x global i to form as a38 holistic f ature representation xi. In order to ge erate attribute-aware binary hash codes, we conduct39 a reconstructing paradigm to project xi as vi in an attribute space wher its data point correspond to40 an a tribute vect r w.r.t. one kind of amabl properties of fine-grained bjects. Finally, has code41 le rning is performed up n vi to obtain the final attribute-awar binary c des ui.42 3.2 Fine-Grained Representation Learning43
Atte tion plays an important role in hum n perception [3, 7], and hum ns exploit a sequence of44 partial glimpses and selectively focus salient parts of an object or a scene in order to b ter capture45 visu l st ucture [10]. I spir d by this, we in orporate th attention mech nism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differences between thes47 ubordinate categories.48 In concretely, we extra t the deep feature of its input image Ii via a backbon CNN model CNN(·)49 s50 Ti = CNN(Ii) 2 RC⇥H⇥W (1) Then, based on Ti, C attenti n guidance Ac 2 RH⇥W is generated as a set of attention maps, i.e., A.51 The attention guidance Ac is desi ned to ev luate which deep descript rs should be attended or ven52 ov rlooked by condu ting53
T ci = A c Ti , (2)
w re is the elem nt-wise Hadamard product. T obtain the final attri ute- ware binary codes, it is54 desirable to transform these appearanc -specific (i.e., low-level) features T to semantic-specific (i.e.,55 mid-level) repr sentations which are closer to the attribute space. Thus, a transforming net ork (·),56 which is equipped with a stack of convolution layers, is performed on T as follows:57
T̂ ci = (T c i ; ✓local) , (3) T̂i = (Ti; ✓global) , (4)
where ✓ pr sents he parameters of the correspo ding r nsforming networks w.r.t. T ci nd Ti, respec-58 tively. Then, we aggregate T̂ ci and T̂i y conducting global average-pooling and co espo dingly59 obtain the atte tive local-level f atures xci , as well as the global-level fe tur x global i . The holisti60 fe ture representation w.r.t. the input image Ii is achieved b concatenating both xci and xgl bali , i.e.,61 xi = h xci ; x lobal i i = F (Ii;⇥) 2 Rd. Note hat, we hereby abstract the aforementioned fin -grained62 featur learning process as a function F (Ii;⇥) with its paramet rs ⇥.63
2
…
2 Related Work23
2.1 Fi e-Grained Imag Retrieval24
2.2 Learning to Hash25
2.3 Visual Attribu es26
3 Methodology27
In thi secti n, we int oduce the verall framew rk and otation f the p oposed A2-NET model, as28 well as elaborating the detailed modules of A2-NET a its corresp nding pti iza ion algorithm.29
3.1 Overall Framework and No ations30
As illustrated in Figure ??, our A2-NET model consists of two crucial modules, i.e., a fine-grained31 representation learning modul nd an attribute-aware hash codes generati g module. Given an32 input image Ii, based n it corr po ding de p activa i t n o Ti 2 RC⇥H⇥W extracted by a33 backbone CNN, a set of attention guida ce A = {Ac} i learned for capt ring fine-grained t ilored34 local patterns T ci from Ti. To distill semantical cues and further generate th final attribute-aware35 bi ary has c des, we pro ose to transfo m these ppearan e-specific f atures T owards semantic-36 sp cific representations T̂ by perf rming a transform etwork (·). Aft r aggre ating T̂ , h obtained37 a tentive local-level features xi ar as ociat d with t e gl bal- evel feature x global i to form as a38 holistic feature representation xi. In order to generate attribute-aware binary h sh codes, we conduct39 a reconstr cting paradigm to projec xi a vi in an attribute space wh re ts data point correspond to40 an attrib te vector w.r.t. one kind of namable properties of fine-grained obj cts. Finally, hash code41 learning s performed upon vi to obtain the final attribute-aw re binary codes ui.42
3.2 Fine-Grained Representation Learning43
Att ntion plays an important role in hu an perception [3, 7], nd humans exploit a sequenc of44 partial glimpses and selectively focus on salient parts of an object or a scene in order to bet er c pture45 visual structure [10]. Inspired by this, we incorp r te the attentio mec anism into represent tion46 lear ing to capture fine-grain d local patt rns for distinguishing subtl differences between these47 subordi ate categories.48 In concretely, we extract the deep feature of its input image Ii via a backbone CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 RC⇥ ⇥W . (1) Then, based on Ti, C attention guida c Ac 2 RH⇥W is gen rat d as s t of at nti n maps, i.e., A.51 Th ttent on guidance Ac s designed to ev luate which deep descriptors should be attended or even52 overl ked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obt in the final attribute-aware bin ry cod s, it is54 desirable to transform these appeara ce-specific (i.e., low-level) features T to semantic-specific (i.e.,55 mid-l vel) repres ntations which are closer to the attribute space. Thus, a transforming network (·),56 which is equipped with a stack of co volution layers, is p rformed on T s follows:57
T̂ ci = (T c i ; ✓local) , (3) T̂i = (Ti; ✓global) , (4)
where ✓ prese ts the paramet rs of the corresponding transforming networks w.r.t. T ci and Ti, respec-58 tively. Then, we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentiv local-level f atures xci , as well as the global-level feature x global i . Th holistic60 f ature representation w.r.t. the input image Ii is a hieved by concatenating both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F (Ii;⇥) 2 Rd. Note that, we hereby abstract the forementioned fine-grained62 feature learning process as a function F (Ii;⇥) with its parameters ⇥.63
2
2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section, we introduce the overall fra ework and notations of the proposed A2-NET model, as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization lgorithm.29 3.1 Overall Framework nd Notations30 As illustrated in Figure ??, our A2-NET model consists of two crucial modules, i.e., a fine-grained31 representation learning module and an attribute-aware hash codes generati g module. Given an32 input image Ii, based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN, a set of attention guidance A = {Ac} is learned for capturing fine-grained tailored34 local patterns T ci from Ti. To distill semanti al cu s d further generate the final attribut -aware35 binary hash cod s, we propose to transform these a pearance-specific eatures T towards semantic-36 specific representations T̂ by performing a transform etwork (·). After aggregating T̂ , the obtained37 attentive local-lev l features xci are associated with th global-level feature x global i to form as a38 holistic feature representation xi. In order to generate attribute-aware binary hash codes, we conduct39 a r constructing paradigm to project xi as vi in an ttribu e sp c where its data point correspond to40 an attribute vector w.r.t. one kind of namable properties f fine-grained objects. Finally, hash code41 lear ing is perfor ed upon vi to obtain the final attribute-aware binary codes ui.42 3.2 Fine-Grained Repres ntatio Learn g43 Attention plays an important role in human percept on [3, 7], and humans expl it a s quence of44 partial glimpses and selectively focu on salient parts of an object or a scene in order to better capture45 visua structure [10]. Inspir d by this, we i c rporate t e attention mechanism into representation46 learning to capture fine-grained local pa terns for distinguishing subtle differences between these47 subordinate categories.48 In concretely, we extract the deep feature f its i put im ge Ii v a a ba kbone CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Then, based on Ti, C attention guidance Ac 2 RH⇥W is generated as a set of ttention maps, i.e., A.51 The attention guidance Ac is des gned to valuate which de p descriptors should be attend d or eve52 overlooked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obtain the final attribut -aware binary codes, it is54 desirable to tran for these appearance-specific (i.e., lo -level) features T to semantic-spec fic (i.e.,55 mid-level) represe tations which are closer to the attribute space. Thus, a transforming network (·),56 which is equipped with a stack of convolution layers, is performed on T as follows:57
T̂ ci = (T c i ; ✓local) , (3) T̂i = ( i; ✓global) , (4)
where ✓ pr sents the parameters of he c rrespo ding transforming networks w.r.t. T ci d Ti, resp c-58 tively. Then, we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . Th holistic60 feature rep es ntation w.r.t. the inp t image I is achieve by concat ati g bot xci and xglobali , i.e.,61 xi = h xci ; x global i i = F (Ii;⇥) 2 Rd. Note that, we hereby abstract the aforem ntioned fine-grained62 f ature l ning pro ess a a function F (Ii;⇥) with its parameters ⇥.63
2
Aggre ation …
2 Related Work23 2.1 Fi e-Grained Image Retri val24 2.2 Le r ing o Hash25 2.3 Visual Attributes26
3 Method logy27
In this section, we introduce the ve ll framework nd notati ns of the propos A2-NET model, as28 well s elaborating the d t iled modules f A2-NET and its corresponding optimization algorithm.29
3.1 Over l Fra ework and Notations30
As ill str ted in Figur ??, our A2-NET model consist of two crucial modules, i.e., a fin -gr ined31 repr sentation learning module and a attribute-aware hash c des ge erating mo ul . Given a32 input i age Ii, based on its co re nding de activatio tens r Ti 2 RC⇥H⇥W xtr cted by a33 back o e CNN, a set of ttention guidance A = {Ac} is learn d f capturing fine-grained tail red34 lo al patterns T ci fr m Ti. To dis ill semantical cues and further gen rate the fi al attribute-aware35 binary hash cod s, we propose to tr nsfo m these ppearance-specific features T towards semantic-36 specifi r pres tations T̂ by performing a transform network (·). Afte aggrega ing T̂ , th obtaine37 atten ive l cal-level f t r s xci are ssociated with th global-level featur x global i o f m as a38 holistic e ture r pr s ntation xi. In order to g nerate attribute-aware binary hash cod s, we conduct39 a econstructing p adigm to project xi as vi i an attribute space wh re it dat point correspo d to40 an ttribute tor w.r.t. o e kind f n m ble properties of fi -grained objects. Finally, hash code41 le rning is p rformed pon vi to obtai th final attribute- ware bi ary codes ui.42
3.2 Fi e-Grained R pres ntation Le rning43
A tentio plays an important role in human pe ception [3, 7], and humans exploit a sequence of44 partial glimpses d selectiv ly focus on sali t parts of an object or a s ene in or e to b tt r capture45 visual stru tur [10]. I sp red by this, w incorporat the at entio mechanism into representation46 learning to ca u e fi -grained local patter s for distinguishing subtle differences between these47 s bordi te tegories.48
In concretely, we extract the deep feature of its input image Ii via a backbone CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1 Th n, based o i, C attention uidan e Ac 2 RH⇥W is g nerated as a set f att ntion m ps, i.e., A.51 The atte ion i a ce Ac is designed to evaluate which deep descriptors should b attended or even52 overlooked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obtain the final attribute-aware b nary codes, it is54 desirable t tran form th se appeara ce-specific (i.e., low-level) features T to semantic-specific (i.e.,55 mid-level) representa ions whi h are closer to the attribute space. Thu , a transforming network (·),56 which is equipped with a stack of convolution layers, is performed n T as follows:57
T̂ ci = (T c i ; ✓local) , (3)
T̂i = (Ti; ✓global) , (4)
wher ✓ presents the parameters of the corresponding transforming n tworks .r.t. T ci and Ti, respec-58 tively. Th n, we ggrega e T̂ ci and T̂i by c n ucting global average-pooling and correspondingly59 o tain the attentive local-level featur s xci , as well a the global-l vel feature x global i . The holistic60 feature representation w.r.t. the i put mage Ii is chi ved by concatenating both xci and xglobali , i.e.,61 xi = h xci ;
l al = F (Ii;⇥) 2 Rd. Note that, we hereby abstract the aforementioned fine-grained62 feature learnin proc ss as a function F (Ii;⇥) with its parameters ⇥.63
2
2 Rel t d Work23
2.1 Fi -Grain d Ima e Retrieval24
2.2 Lear i g to Ha h25
2.3 Visu l Attributes26
3 Methodology27
In this section, we introduce the overall framework and notations of the proposed A2-NET model, as28 w ll as elaborating th detailed m dules of A2-NET and its correspon ing optimization algorithm.29
3.1 Overall Framework and Notation30
As illustrated in Fig r ??, our A2-NET m del consists f two cruc al modules, i. ., a fi e-grained31 r p tatio l arning modul d an attribut - ware hash codes generating module. Given an32 inp t image Ii, b sed o its respond ng deep a tivation tensor Ti 2 RC⇥H⇥W extracted by a33 ba kbone CNN, a set of attention g id nce A = {Ac} is learned f r capturing fine grained tailored34 l l atterns T c from Ti. To distill semantical cues and further en r te the final att ibute-aware35 binary hash des, w pr p se to transform these ppeara ce-specific features T towards semantic-36 specific representations T̂ by perfor ing a transform netwo k (·). After aggregating T̂ , he obt ined37 attentive local-lev l features xci re associ te with the glob l-lev l feature x global i to form as38 holistic feature representation xi. In order to g nerate attribute-aware binary h sh cod s, w conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond to40 an attribute v ctor w.r.t. one kind of namable properties of fine-gra ned o jects. Finally, has c de41 learning is performed upon vi to btain the fi al attribute-aware binary codes ui.42
3.2 Fine-Grained Representation Lear ing43
At ention plays a important role in human perception [3, 7], and humans exploit a sequence of44 pa tial glimpses and select vely f cus on salient parts of an bject or scene in ord r to be ter capture45 visual structure [10]. Inspired by this, we incorporate the attention mechanism into representatio46 lear ing to c pture fin -grain d local patterns for distinguishing ubtle differences b twe n th s47 subordinate c tegories.48
In concretely, we xtract the deep feature of its input image Ii via a backbone CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Then, based on Ti, C attention guidance Ac 2 RH⇥W is generat d as a set of attention maps, i.e., A.51 The attention guidance Ac is designed to evaluate hich deep descriptors should be tte d d or eve52 overlooked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obtain the final attribute-aware binary codes, it is54 desirable to transform these appearance-specific (i.e., low-level) features T to sema tic-specific (i.e.,55 mid-level) representatio s which are clo er to the attribute sp c . T us, transf rming etw rk (·),56 which is equipped with a stack of convolution layers, is performed on T as follows:57
T̂ ci = (T c i ; ✓local) , (3) T̂i = (Ti; ✓global) , (4)
where ✓ presents the parameters of the corresponding transformi g networks w.r.t. T ci and Ti, r spec-58 tively. Then, we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . The holistic60 feature representation w.r.t. the input image Ii is achieved by concaten ting both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F (Ii;⇥) 2 Rd. Note that, we her by abstr c the af rementioned fine-gr ined62 feature learning process as a function F (Ii;⇥) with its parameters ⇥.63
2
2 R lat d Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section, we introduce the verall framework and notations of the proposed A2-NET model, as28 well as elaborating the detailed odules of A2-NET and its corr sponding optimization algorith .29 3.1 Overall Framework and Notations30 As illustrated in Figure ??, our A2-NET model cons sts of two crucial modules, i.e., a fine-grain d31 representation learning module and an attribute-aware hash codes gene ating module. Given an32 input image Ii, based on its corresponding de p c ivation t nsor Ti 2 RC⇥H⇥W extracted by33 backbone CNN, a set of at ention guidance A = {Ac} is l arned for capturing fin -grained tail red34 local patterns T ci from Ti. To distill semantical cues and furt er generat the final at ribute-aware35 binary hash codes, we propose to transfor th se appearance-specific features T towards semantic-36 specific re resentations T̂ by performing a transform network (·). Aft r aggregating T̂ , the obtained37 attentive local-level features xci are asso iated with the global-level feature x global i o form as a38 holistic featur representation i. In ord r to generate attribute-awar binary hash codes, we conduct39 a rec nstructing parad gm to project xi as vi in an attribute space wh re its data point corr spond to40 an attribute vector w.r.t. one kind of namable properties of fine-grained objects. Finally, hash code41 learning is performed up n vi to obtain the fin l attribute-aware bi ary codes ui.42
3.2 Fine-Grain d Representation Learning43
Attention plays an important role in human perception [3, 7], and ans exploit a sequ nce of44 partial glimps s and s lectively focus on s lient parts of an object or a e in orde to bette captur45 visual structure [10]. Inspired by this, we incorporate he attenti n ec anism into repr sentation46 learning to capture fine-grained local patterns for distinguishi g subtle ifferences between these47 subordinate categori s.48 In concretely, we extract the deep feature of its nput image Ii via a backbone CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Then, based on Ti, C attention guidance Ac 2 RH⇥W is generated as a set of attention maps, i.e., A.51 The att ntion guidance Ac is designed to evaluate which d ep escriptors should be attended or even52 overlooked by conducting53 T ci = A Ti , (2) where is the element-wise Hadama d product. T obtain he final a tribute-awar binary cod s, it is54 desirable to transform these appearance-specific (i.e., low-level) features T to semantic-specific (i.e.,55 mid-level) representations which are closer t the attribute space. Thus, transforming netwo k (·),56 which is equipped with a stack of conv lution layers, is performed on T as follows:57
T ci = ( i ; local) , (3) T̂i = (T ; ✓globa ) , (4)
where ✓ presents the parameters of the corresponding transforming networks w.r.t. T ci and Ti, respec-58 tively. Then, we aggregate T̂ ci and T̂i by onducting glob l average-poolin and correspondingly59 obtain the attentive local-level features xci , as w ll as the gl b l-level featur x global i . Th holistic60 f ature r presentation w.r.t. t input image Ii is achiev d by concat nating both xci a d xglobali , i.e.,61 xi = h xci ; x global i i = F (Ii;⇥) 2 Rd. Note that, we hereby bst ct the aforementione fin -grai d62 feature learning process as a function F (Ii;⇥) with its parameters ⇥.63
2
Aggregation
where t e d coder (i.e., a counterpart of the encoder) is r alized by W> to simplify the netw k.91 However, directly minimizing Eq. (6) with a hard constraint is hard to optimize. Therefore, we relax92 the constraint into a soft constraint, and the learning objective can be rewritten by93 min W kX W>V 0k2F + kWX V 0k2F . (7) 3.3.2 Attribute-Specifi F ur Decorrelation94 By conducting the aforementioned unsupervised attribute-guided learning, we can obtain the internal95 l tent vec rs V 0 as the attribut -sp cific features. I order to both en ance the discriminativ ability96 and remove the redundant correlation among these attribute-specific features, we introduce a featur97 decorrelation c n r int upon V 0, which is formulat d by98
mi V 0 kV 0V 0> nIk2F , (8) whe e I is t e i entity matrix. Such a feature ecorrelation co straint is pref rable to construct99 independent features and reduce redundant information. Therefore, based on unsupervised attribute-100 guided lear ing an ttribute-specific feature de orrelation, the final learne hash code are expected101 to be bo h ttribute-aware and hash-bit ind pend nt.102
3.3.3 H sh Co e L arning103
In the f llowing, we conduct the h sh code learni g ased on the obtained attribute-spe ific features.104 Assum that w have n q ry data p ints which ar d noted as {qi}i=1, as w ll a m database points105 which a enoted as {vj}j=1. Note hat, both qi and vi are belo gin to the attribute space V . Then,106 the corr sponding binary has codes ca be obt ined via107
ui = sgn(tanh(qi)) , (9) zj = sg (tanh(vj)) , (10)
where ui, zj 2 { 1, +1}k. The goal of our hash code learning is to learn binary hash c des for both108 qu ry poin s and database points from {qi}ni=1, {vj}mj=1, and the pairwise supervised information,109 i.e., S 2 { 1, +1}n m. To pr se ve the pairwise similarity, ad pt the `2 loss between the110 supervised inf rmati ( .k.a. similarity) d t inner product f qu ry-database point binary code111 pairs. I can be fo mulated s foll ws:112
minU ,Z Pn
i=1 Pm j=1 u>i zj kSij 2
s.t. U 2 { 1, +1}n⇥k, Z 2 { 1, +1}m⇥k . (11)
Overall, g t the fi a obj tiv of the propo ed A2-NET model by considering Eq. (7), Eq. (8) and113 Eq. (11) toge a follows:114
min W ,⇥
L(I) = kX W>V 0k2F + kWX V 0k2F +↵kV 0V 0> nIk2F + nX
i=1
mX
j=1
> i zj kSij 2 ,
(12) where , ↵ and are hyper-param ters as the trade-off.11
In practice, during training, it might be only available a set of database poin s {vj}mj=1 without query116 points. Thus, we randomly sample data poi ts from database to cons uct a query set, and denote117 the indices of all the databa e points as with the indices of the query set as ⌦. Additionally, because11 we cannot back-propagate the gradient to ⇥ due to the sgn(·) function, we o it the sgn(·) function119 and o ly apply t nh(·) for elaxation in Eq. (10) of the whole optimization process. Therefore, the120 optimization formulation of A2-NET can be rewritten with only database points {vj}mj=1 for training121 as:122
min W ,⇥
L(I) = kX W>V 0k2F + kWX V 0k2F + ↵kV 0V 0> nIk2F
+ X
i2⌦
X
j2
⇣ tanh(W · F (Ii;⇥))>zj kSij ⌘2 . (13)
For opti ization, our A2-NET does not require co plicat d two-stage le rning algorithms, e.g., the123 altern tive optimization strategy. We emplo the back-propagation algorithm and follow [8] to train124 the whol A2-NET mo el in a u ified end-to-end anner.125
4
“Blue he d”
Figure 1: Overall fr ewo k of the propos d A2-NET m el, which consists of two crucial modules, i.e., fin -grained representa ion l arning and attribute-aware hash c d s generating. The whole etw rk c n be end-to-e d trai abl , d is generally riven by the nsupervi ed attribute-guided loss, the feature decorrelation loss and he hash cod learn g loss, cf. S ction 3.3.
has th potenti l to help with large-sc le fine- ra ed d ta t o. Therefo , targeting the ig data35 challenge, fine-grained hashing [7, 13] is a promisi g direction ort y of furth r explorations.36
low-lev to mid-lev l appe ra ce-specific t s a tic-specific37
obtain corresponding fine-grai ed obj ct attributes (e.g., xx), without significant annotati n cost38
2 Relat d ork39
In this section, we briefly review t e r lated work i the following t r e aspects, i.e., n -g ained40 i age retrieval, learning to hash and the studies b ut visual attribu es in computer visi n problems.41
2.1 Fine-Grained Image Retrieval42
Fine-grained image retrieval as an integr l part of fine-grained image analysis [34] has gained more43 and more tractio i recent years [7, 20, 23, 33, 35, 37, 38]. What makes it challenging is that44 objects of fine-grained images ave only subtle differences, and often largely vary in pose, scale and45 orientation or can exhibit cross-modal d ff rences (e.g., sketch-based retrieval [20]).46
Depending on the type of query image, the most studied areas of fine-grained im ge retrieval47 can be s parated into two groups: fine-grained con ent-base image retrie al (FG-CBIR) and fine-8 grained sketch-based image retrieval (FG-SBIR). More specifically, in FG-CBIR, u s pervised49 learning based [33] nd supervised learning based methods [36–38] were dev loped from different50 perspectives for dealing with fine-grained retrieval tasks, e. ., localizing fin -gr ined parts [33],51 hancing intra-class s parability with inter-class compactness [38], and reducing the confidence52 of the fine-grained predictions [36], etc. While, FG-SBIR needs to ot only capture fine-grained53 ch racteristics pres nt in the sketches, but also possess the ability to traverse the sketch and image54 domain gap. In the literature of FG-SBIR, the earlier works, e.g., [17, 27, 35], were mostly based55 on Siamese-triplet networks [3] to tackle the aforementioned challenges. Recently, some works56 tried to incorporate the advances of recent progress in self-supervised learning [5] and attention57 mecha isms [6] f further improving the retrieval ccuracy of FG-SBIR, e g., [23, 25].58
However, although these fine-grained retrieval methods achieved good results, they still show the59 limitations in the face of large-scale data, i.e., the searching time for exact nearest neighbor is typically60 expe siv or ven impossible for the given queries. To alleviate this issue, fine-grained hashing, which61 aims to generate compact binary codes to represent fine-grained images, as a promising direction62 has attracted the attention in the fine-grained community very recently [7, 14]. More specifically,63
2
R d_head?
Blue_tail?
O _the_tr e?
Spotted_body?
YES
NO
YES
YES
1
0
1
1
Att ibute-aware has codes
Retrieval
Top-k retrieved images
1
0
1
1
1
0
1
1
1
0
1
0
Figure 1: K y idea f t propos A2-NET mo l, s w ll th ain proces of fine- rained hashi g ased o our attribute-a are h s c des. In co cr tely, for a query image Iq of Red bellied Woodp cke , aft r r tur ing all the correct results, th fine-grained image (belonging to Red aded Wo pe k ) cl s s to t e qu ry ima e is al o etriev d.
A2-NET: Le r i g Attr bu e-Aw re Has Codes for L g -Scal Fi -Gr Im ge R ieval
Anonymous Author(s) Affiliation Addr ss email
Abstract
The task of fine-grained image retrieval aims to rank all the insta c s so that images1 depicting the concept of interests (i.e., the same sub-category labels) are ranked2 highest based on the fine-grained details in the query. It is challenging due to the3 subtle differences between sub-category, as well as the fine-grained nature of small4 inter-clas variations d l rge intra- lass va iations. Espec ally for large-scale5 data, it is desirable to be ha dled in a bo h effective and efficient way. In this paper,6 we an Attribute-Aware hashing Network, i. ., A2-NET, for generating an attribute-7 awar binary cod s to not only speed up the retrieval process, but also corresponds8 to fine-grained attributes. Specifically, based on the learned attention maps, we9 design an auto-encoder to project the visual feature vector into an attribute space10 by following a reconstructing paradigm. Furthermore, A2-NET is also equipped11 with an orthogonal constraint upon the generated attributes to enhance their own12 representation abilities. Qualitative experiments on five benchmark fine-grained13 datasets show our superiority over c mpeting methods. Moreover, quantitative14 results r veal the obtained hash cod s corresp nd to the attributes of fine-g ained15 patterns.16
1 Introduction17
a other barrier also occurs because of xxx18
suffer from slow query speed and redundant storage costs due to both the explosive growth of massive19 fin -grain d data nd high-dimensional r al-valued features.20
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.
Query image
101 1
Enc de
3.3 Attribute-Aware Hash Codes Ge erating64 How to generate attribute-awar h sh codes is the key of our A2-NET model. We laborate it i65 the following four aspects, i.e., unsupervised attri ute- uided learning, attribute-specific feature66 decorrelation, hash code learni g and the opti izatio lea n ng algo it m.67 3.3.1 Unsupervised Attribute-Guided L r ing68 In r al-applic tio s, esp ially f r the large-scale d fi e-grained tasks, attribute annotations are69 always infea ible, whic limits the l ar ing p oces t be conducted in an unsupervised setting.70 Whil , i he literatur , th main go l of unsup rvis d learning is to capture regularities in data for the71 purp s f extracti g useful r presentations or for restoring corrupted d ta [4]. Many uns pervised72 m tho s explicitly r uce internal la e t n ts or c d s, fr m which the data is to be reconstru ted.73
Inspired by this, w d velop an u supervised att ibute-guided l rning component to pr ject the74 holistic representation xi of Ii into a latent space, i.e., the attribute space V . In V , its high-level75 vectors are designed to have certain sirable pr per s, e.g., o responding to se an ic properti s76 of fine-grained objects (a.k.a. “fine-gr ine at ributes”).77
More s ecifically, in our A2-NET, the u supervise attribute-guid d learning is r alized by a rec n-78 struction paradi m wi a coder-d coder structur , s s own in F g e ??. I co cret ly, given79 a b tch of n tra i g d ta Ii, th h listic r p es ta ion X = {x1; x2; . . . ; xn} 2 Rd⇥n can be80 obtained as af remention d. By rmulation, the n ode projects X into the ttrib te space V with81 projection m trix W 2 Rk⇥d to get an inter al lat nt r pr se tation V Rk⇥n w.r.t. X . In82 particularly, we set that the dim nsion of atent represe tation k quals to th n ber of has its in83 the final binary hash code ui. Fu th r ore, each c lumn of V , i.e., vi 2 Rk, an d riv i by84
= sg (ta h v )) . (5)
Meanwhil , regarding vi, w also co side to rec stru t its inpu xi by a de oder as a count rpart85 of the encoder. Therefor , on o hand, such reconstruction paradigm can reduce nd further i till86 high-level sem tic cues in the attribute space V . While, on the other h nd, it can drive the training87 of A2-NET by prese ving he s il rity etween queried hash co es and databas poi ts i terms of88 learning to hash (cf. Section 3.3.3).89
In concretely, the le rni g bjective of unsupervised att ibut -guided lear ing i written s follows:90
min W
kX W>WXk2F s.t. WX = V 0 = t h(V ) , (6)
wh re the de oder (i.e., unter ar f the e coder) is r alized by W> to simplify th etw rk.91 However, directly minimizing Eq. (6) with a hard constrai t s hard to optimize. Therefor , w r l x92 the constr int into a soft con traint, and th learning objective can be rewritten by93
min W
kX W>V 0 2 + kWX V 0k2F . (7)
3.3.2 Attribute-Specific Feature D correlation94
By conducting the aforementioned unsupervised attribute-guided learni g, w ca obtain he internal95 latent vect rs V 0 as the attribute-specific features. In order to both e han e the di cr minativ bility96 and remove the redundant correlation among these a tribute-specific features, w introduce fea ur97 decorrelation constrain upon V 0, which is formulated by98
min V 0
kV 0V 0> kIk2F , (8)
where I is the d ntity mat ix. Such a featur decorrelatio constraint is pr f rable to construct99 independent f atures and redu e red ndant informati n. Therefore, based on unsupervised attri t100 guided learning and attribute-specific feature decorrelation, the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102
3.3.3 Hash Code Learning103
In the following, we conduct the hash code learning based on t e obtained attribute-specific featu .104 Assume that we have n que y data points which are denoted as {qi}ni=1, as well as m database point105
3
3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET mo el. We elaborate it in65 the following four aspects, i.e., unsupervised attribute-guided learning, attribute-specific feature66 decorrelation, hash code learning and the optimization learning algorithm.67 3.3.1 Unsupervised A tribut -Guid d Lea ing68 In real-applications, especially for the large-scale a d fine-grained asks, attribute annotations are69 always infeasible, which limits the ea ning proc ss to be conduct d in a unsup rvi d setting.70 While, in the literature, the main goal of un upervised learning i to capture re ulari ies i ata for the71 purpose of extracting useful r presentations r for restoring corrupted d a [4]. Many unsup rvised72 methods explicitly produce internal latent units or codes, from whi h the data i to be r constructed.73 Inspired by this, we develop an unsupervised attribute-guided learning component to project the74 holistic representation xi of Ii into a latent space, i.e., the attribute spa V . In V , i s high-l vel75 vectors are designed to have certain desirable properties, e.g., corresponding to semantic properties76 of fine-grained objects (a.k.a. “fine-grai ed attributes”).77 More specifically, in our A2-NET, th un upervised tt ibute-gui d l arn ng is r lized by recon-78 struction paradigm with a encoder-decoder structure, as show in Figure ??. In concretely, giv n79 a batch of n training data Ii, the h listic e esentations X = {x1; x2; . . . ; xn} 2 Rd⇥n an be80 obtained as aforementioned. By formulati n, th e o er pr jects X i to the attribute p ce V wi h81 a projection matrix W Rk⇥d to g t an int rnal latent epr se tatio V 2 Rk⇥ w. .t. X . In82 particularly, we set that the d en ion of l t nt r pr sent tio k equ ls to the number of hash bit in83 the final binary hash code ui. Furthermore, each colum of V , i. ., v 2 Rk, ca derive ui by84
ui = sgn(tanh(vi)) . (5)
Meanwhile, regarding vi, we also consider to re onstruct its in t xi by dec der as a counterpart85 of the encoder. Therefore, on one hand, such a reconstructio paradi m c n r duc and fur h d still86 high-level semantic cues in the attribute sp c V . While, on th other hand, t c n rive the rai ing87 of A2-NET by pres ving he similarity b twe n queri d h h cod a d ataba o nts in t m of88 learning to hash (cf. Section 3.3.3).89
In concretely, the learning objective of unsupervised ttribut u d learning is writt s follows:90
min W
kX W>WXk2F s.t. WX = V 0 = anh(V ) , (6)
where the decoder (i.e., a counterpart of the ncod r) is realiz d by W> t implif th network.91 However, directly minimizing Eq. (6) with a hard constr int is hard to optimize. Therefore, we r lax92 the constraint into a soft constraint, nd the learning obj ctive can be rewrit by93
mi W
kX W>V 0k2F + kWX V 0k2F . (7)
3.3.2 Attribute-Specific Feature Decorre a ion94
By conducting the afor mentio ed unsupervised attribute-guided le ning, we can obtain the internal95 latent vectors V 0 as the attribute-specific features. In ord r to both nhan e the discri inative ability96 and remove the redundant correlation among these attribute-specific features, we introduce a fe ture97 decorrelation constraint upon V 0, which is formulated by98
mi V 0
kV 0V 0> kIk2F , (8)
where I is the identity matrix. Such a feature decorrelation constraint is preferable to construct99 independent features and reduce redundant information. Therefore, b sed on unsupervised attribute-100 guided learning and attribute-specific feature dec r elation, the final learned hash codes ar expected101 to be both attribute-aware and hash-bit independent.102
3.3.3 Hash Code Learning103
In the following, we conduct the hash code learning based on the obtained attribute-s c features.104 Assume that we have n query data points which are denote s {qi}ni=1, as well as m database points105
3
3.3 Attribute-Awar H sh Codes G n ati g64
How to generate attribut -aware hash codes is the key of our A2-NET model. We elaborate it in65 the following four aspec s, i.e., un u e vised attribu e-gui ed lear i g, attribute-s ecific feat re66 d c relati n, hash code learning and the ptimization learn g algorith .67
3.3.1 Unsup rvised Att ibute-Guided Learning68
In real-a plications, esp cially for the large-scale and fine-gr ined tasks, attribute an ot tions are69 always infeasible, which limits the learning proc ss to be con ucted in an u supervised setting.70 While, in the literature, the main goal of u supervised learning is to capture regul rities in data for the71 purpo e of extracti g useful repr s nt tions or for restoring orrupted d ta [4]. Many unsup rvi d72 methods explicitly produc internal lat nt units r c des, fr w ic t e dat is o be r constructed.73
Inspired by this, we develop an un upervise attribut -g ided learn ng compon t t r ject t74 holistic representatio xi of Ii into a latent space, i.e., the attribut space V . In V , its high-level75 vectors are designed to have certain desirable propertie , e.g., corresponding to s mantic prope ties76 of fine-grained objects (a.k.a. “fine-grained attribut s”).77
More specifically, in our A2-NET, h un up rvised attribute-guided arning is re lized by a r co -78 structi n aradigm it an encoder-decod r structure, as shown in Figu e ??. In c ncre ely, given79 a batc of n tr in ng data Ii, the holi ic r pr se tation X = {x1; x2; . . . ; xn} 2 Rd⇥ n be80 obtained as aforemention d. By form lation, the enco er projects X into the t ribu e space V with81 a projectio matrix W 2 Rk⇥d to t an nternal lat nt pr ent tion V 2 Rk⇥n w.r.t. X . I82 particularly, w s t tha the dim nsio of late t rep se tati n k quals t the numb r of hash bits in83 the final binary ash c de ui. Furthermore, each lum of V , i. ., vi 2 Rk, can derive ui by84
ui = sgn(tanh(vi)) . (5)
Meanwhile, regarding vi, we also consider to econstruct i s input xi by a decoder as a counterpart85 of the ncod . Th refore, on one hand, uch reconstructio par digm can reduce and further distil86 high-level semantic cues in the attribut space V . Whil , on the other hand, it can drive the training87 of A2-NET by preservin the similarity betwee queried hash codes and database points in terms of88 learning to hash (cf. Section 3.3.3).89
In con retely, the learn ng objective f supervised attribute-guided learni g is written as foll ws:90
min W
kX W>WXk2F .t. WX = V 0 = anh(V ) , (6)
where the decoder (i.e., a counterpart f the encoder) is r alized y W> to simplify the network.91 However, irectly minimizi g Eq. (6) with a hard con traint is hard to optimize. Therefore, we r lax92 the constraint into a soft constraint, and the l ar ing objective can be rewritte by93
min W
kX W>V 0k2F + kWX V 0k2F . (7)
3.3.2 Attribute-Specific Featur Decorr lation94
By conducting the aforementioned unsupervised attribute-guided learning, we can obtain the internal95 latent vectors V 0 as the attr bute-specific features. In order to both enhance the discriminative ability96 and remove the redundan correlation among these attribute-specific features, we introduce a feature97 decorrelation constraint upon V 0, which is formulated by98
min V 0
kV 0V 0> kIk2F , (8)
where I is the identity matrix. Such a feature decorrel tio constraint is prefer ble to construct99 independent features and reduce redundant information. Therefore, based on unsup rvised attribute-100 guided learning an attribute- p ific feature ecorrelation, the final l arned hash codes are expect101 to be both attribute-aware and hash-bit independent.102
3.3.3 Hash Code Learning103
In the following, we conduct the hash code learning based on the obtained attribut -spec fic featur s.104 Assume that we have n query data points which are denoted as {qi}ni=1, as well as m da base points105
3
3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET model. W laborat it in65 the following four aspects, i.e., unsupervised attribute-guided learning, attribute-specific feature66 decorrelation, hash code learning and the optimization learning algorithm.67
3.3.1 Unsupervised Attribute-Guided Learning68
In real-applications, especially for the large-sca e and fine-grained task , ttr bute an otation r69 always infeasible, which limits the learning process to be nducted in an unsu ervised setting.70 While, in the literature, the main goal of unsupervised learning is to capture regularities in dat for the71 purpose of extracting useful representations or for restoring corrupted data [4]. Many unsupervised72 methods explicitly produce internal latent units or codes, from which the data is to be reconstructed.73
Inspired by this, we develop an unsupervised attribute-guided learning compo ent to project the74 holistic representation xi of Ii into a latent space, i.e., t tribute sp ce V . In V , its high-l75 vectors are designed to have certain desirable properties, e.g., corresponding to semantic properties76 of fine-grained objects (a.k.a. “fine-grained attributes”).77
More specifically, in our A2-NET, the unsupervised attribut -guided learning s r alize by a reco -78 struction paradigm with an encoder-decoder structure, as shown in Figure ??. In concretely, given79 a batch of n training data Ii, the holistic representations X = {x1; x2; . . . ; xn} 2 Rd⇥n can be80 obtained as aforementioned. By formulation, the encoder projects X into the attribute space V with81 a projection matrix W 2 Rk⇥d to get an internal latent representation V 2 Rk⇥n w.r.t. X . In82 particularly, we set that the dimension of latent representation k equals to the number of hash bits in83 the final binary hash code ui. Furthermore, each column of V , i.e., vi 2 Rk, can derive ui by84
ui = sgn(tanh(vi)) . (5)
Meanwhile, regarding vi, we also consider to reconstruc its input xi by decoder as a counterpart85 of the encoder. Therefore, on one hand, such a reconstruction paradigm can reduce nd further distill86 high-level semantic cues in the attribute space V . While, on the other hand, it can drive the training87 of A2-NET by preserving the similarity between queried hash codes and database points in terms of88 learning to hash (cf. Section 3.3.3).89
In concretely, the learning objective of unsupervised attribute-guided learning is written as follows:90
min W
kX W>WXk2F s.t. WX = V 0 = tanh(V ) , (6)
where the decoder (i.e., a counterpart of the encoder) is realized by W> to simplify the network.91 However, directly minimizing Eq. (6) with a hard constraint is hard to optimize. Therefore, we relax92 the constraint into a soft constraint, and the learning objective can be rewritten by93
min W
kX W>V 0k2F + kWX V 0k2F . (7)
3.3.2 Attribute-Specific Feature Decorrelation94
By conducting the aforementioned unsupervised attribute-guided learning, we can btain the int rnal95 latent vectors V 0 as the attribute-specific features. In order to both enhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features, we introduce a feature97 decorrelation constraint upon V 0, which is formulated by98
min V 0
kV 0V 0> kIk2F , (8)
where I is the identity matrix. Such a feature decorrelation constraint is pr f rable to construct99 independent features and reduce redundant information. Therefore, based on unsupervised ttri t -100 guided learning and attribute-specific feature decorrelation, the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102
3.3.3 Hash Code Learning103
In the following, we conduct the hash code learning based on the obtained attribute-sp cific feat res.104 Assume that we have n query data points which are denoted as {qi}ni=1, as well as m database points105
3
D coder
3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET model. We elaborate it in65 the following four aspects, i.e., unsupervised ttribute-guided learning, attribute-specific feature66 decorrelation, hash code learning and the optimization learning algorithm.67 3.3.1 Unsupervised Attribute-Guided Learning68 In real-applications, especially for the large-scale and fine-grained tasks, attribute annotations are69 always infeasible, which limits the learnin proc ss to be conducted in an unsupervised setting.70 While, in the literature, the main goal of unsupervised learning is to capture regularities in data for the71 purpose of extracting useful representations or for restoring corrupted data [4]. Many unsupervised72 methods explicitly produce internal latent units or codes, from which the data is to be reconstructed.73 Inspired by this, w develop an unsupervised attribute-guided learn ng component to project th74 holistic repr sentation xi of I into a late t space, i.e., th attribute space V . In V , its high-level75 vectors are desig ed to have certain d sirable pr perties, e.g., corresponding to semantic properties76 of fine-grain d objects (a.k.a. “fine-grained attributes”).77 More specifically, in our 2-NET, the unsupervised attribute-guided learning is realized by a reco -78 s ruction paradigm with an ncode -decoder structure, as shown in Figure ??. In concret ly, give79 a batch of n training d ta Ii, the holistic represe tatio s X = {x1; x2; . . . ; xn} 2 Rd⇥n can be80 obtained as afor menti ned. By form atio , th enco r projects X into the attribute spac V with81 a proj ction matrix W 2 Rk⇥d to get an internal latent representation V 2 Rk⇥n w.r.t. X . In82 p rticularly, we se that the dim nsion of latent repr sentation k equals t th number of hash bits in83 the final binary hash c de i. Furthermore, each colum of V , i.e., vi 2 Rk, ca derive ui by84 ui = sg (tanh(vi)) . (5) Meanwhile, regard ng vi, we l o conside to reco struct it input xi by a decod r s a counterpart85 f the ncoder. Ther fore, o o e ha , ch a rec nstruction paradigm can re uce and further distill86 high-level semantic cues in the attribute space V . While, on the ther hand, it can drive the training87 of A2-NET by p eserving the imila ity between queried h sh cod s and database po nts in t rms of88 lear ing to hash (cf. Section 3.3.3).89
In concretely, the learni g objec ive of nsupervised t ibute-guid d learni g is w itten as follows:90
min W
kX W>WXk2F s.t. WX = V 0 = tan (V ) , (6)
where the decoder (i.e., a count rpart f the encod r) is realized by W> to s mplify the network.91 How ve , di tly minimizing Eq. (6) with a h rd onstraint is hard to optimize. T erefore, rel x92 the constr i t int a soft constraint, and th l arning obj ctiv can b r written by93
mi W
kX W>V 0k2F + kWX V 0k2F . (7)
3.3.2 A tribut -Specific F ure Decorrela ion94
By c ducting the a re e tioned unsupervis d attribute-g id d l arni g, we can obtain the in r l95 latent vectors V 0 as the attribute-specific fe ures. In rd r to both e hanc the iscrimi ative a il y96 and r m ve the re un nt co relation among these attribute-sp cific features, we introd ce a feat e97 decorrela ion co str int upon V 0, which is formulat d by98
min V 0
kV 0V 0> kIk2F , (8)
where I is the identity matrix. Such a f ature d c rr lati n constraint is pr f rabl to onstruct99 independent features and reduce redundant information. Therefore, based on unsupervised attribute-100 uided l arning and attribute-specific feature decorrelat on, th fi al learned h sh c des are xpected101 to be both attribute-aw re and hash-bit ndepend nt.102
3.3.3 Hash Code Learning103
In the following, we conduct the hash code learning based on the obtained attribute-specific features.104 Assume that w hav query d ta poi ts which ar den ted as {qi}ni=1, as w ll as m d tabase points105
3
which are denoted as {vj}mj=1. Note that, both qi and vi are belonging to the attribute space V . Then,106 the corresponding binary hash codes can be obtained via107 ui = sgn(tanh(qi)) , (9) zj = sgn(tanh(vj)) , (10) where ui, zj 2 { 1, +1}k. The goal of our hash code learning is to learn binary hash codes for both108 query points and database points from {qi}ni=1, {vj}mj=1, and the pairwise supervised information,109 i.e., S 2 { 1, +1}n⇥m. To preserve the pairwise similarity, we adopt the `2 loss between the110 supervised information (a.k.a. similarity) and the inner product of query-database point binary code111 pairs. It can be formulated as follows:112 minU ,Z Pn i=1 Pm j=1 u>i zj kSij 2 s.t. U 2 { 1, +1}n⇥k, Z 2 { 1, +1}m⇥k . (11) Overall, we get the final objective of the proposed A2-NET model by considering Eq. (7), Eq. (8) and113 Eq. (11) together as follows:114 min W ,⇥ L(I) = kX W>V 0k2F + kWX V 0k2F +↵kV 0V 0> nIk2F + nX i=1 mX j=1 u>i zj kSij 2 , (12) where , ↵ an are hyper-parameters as the trade-off.115 In practice, uri g training, it might be only available a set of database points {vj}mj=1 without qu ry116 points. Thus, we randomly sample n data points from database to construct a query set, and denote117 th indices of all the database points as with the indices of the query set as ⌦. Additionally, because118 we cannot b ck-propagate the gradient to ⇥ due to the sgn(·) fu ction, we omit the sgn(·) function119 and only apply tanh(·) for relaxation in Eq. (10) of the wh le optimization process. T erefore, th120 optimization formul tion of A2-NET can be rewritten with only atabase points {vj}mj=1 for training121 as:122 min W ,⇥ L(I) = kX W>V 0k2F + kWX V 0k2F + ↵kV 0V 0> nIk2F + X i2⌦ X j2 ⇣ tanh(W · F (Ii;⇥))>zj kSij ⌘2 . (13) For optimization, we e ploy the back-propagat on lgorithm and follow [8] t train the whole123 A2-NET m l in an e d-t - d m nn r.124 3.4 Out-of-Sample Extension125 After tr ining A2-NET, the learned model can be applie for genera ing binary codes f r query points126 in lud ng unseen query points in th training phase. Specifically, we can use the followi g equation127 t gen rate the binary code for Iq:128
uq = sg (ta h(W · F (Iq;⇥))) . (14)
4 Exp riments129
In this cti , we pr sent th impl men atio d tails, empiri al settings, main results, blati n studi s130 and al o some i t rest ng qualitative a lyses of the le rned attribute-aware binary hash codes.131
4.1 Datasets132
x0i By fol owi g ExchNet [4], our expe iments are conducted on five fine-grained benchmark datasets,133 i.e., CUB-200-2011 [16], Aircraft [12], Food101 [1], NAB ds [15] and V gFru [6]. Specifically,134 CUB-200-2011 is on of th most pop lar used fi e-gr ined d tas ts. It contains 11,788 bird images135 from 200 bird sp i s and is officially s lit into 5,994 i g s f r training with 5,794 imag s f r136 test. Ai cr ft con a s 10,000 imag s nnin 100 aircr ft models with 3,334 for training, 3,333137
4 Fine-grained representation learning Un
su pe
rv is
ed a
ttr ib
ut e-
gu id
ed lo
ss (S
ec tio
n 3.
3. 1)
Feature decorrelation loss (S ction 3.3.2)
3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET model. We elaborate it i65 the following four aspects, i.e., unsupervised attribute-guided learning, attribute-specifi feature66 deco relation, hash code learning and the optimization lea ning algorithm.67 3.3.1 Unsupervised Attribute-Guided Learning68
In real-applications, especially for the large-scale and fine-grained tasks, attribute annotations are69 always infeasibl , which limits the learning process to be onducted in an unsupervise setti g.70 While, in the literature, the main goal of unsupervised learning is to capture regularities in data for the71 pu p se of xtracting useful re esentations or for restoring c rrup ed data [13]. Many unsupervised72 methods explicitly produc intern l latent units or codes, fr m which the da is to b reconstructed.73 Inspired by this, we develop an unsupervised attribute-guided learning c mponent to roject the74 holistic r presentation xi f Ii into a latent space, i.e., the attribute space V . In V , its high-level75 vectors are designed to have certain desirable properties, e.g., cor sponding to semantic properties76 of fine-grained objects (a.k. . “fine-grained att ibutes”).77 Mor specifically, in our A2-NET, the unsupervised attribute-gu ded l arning is realized by a recon-78 struction paradigm with an encoder-decoder structure, as shown in Figure ??. In concretely, given79 a batch of n training data Ii, the holistic representations X = {x1; x2; . . . ; xn} 2 Rd⇥n ca be80 obtained as aforementione . By formulation, the encoder projects X into the attribute space V with81 a projection matrix W 2 Rk⇥d to get an internal lat nt r presentat on V 2 Rk⇥n w.r.t. X . In82 particularl , we set that the dime sion of latent representation k quals to the number of hash bits in83 the final binary h sh code ui. Furthermore, each column of V , i.e., vi 2 Rk, can derive ui by84
ui = sgn(tanh(vi)) . (5)
Meanwh le, regardi g vi, we als consider to reconstruct its input xi by decoder as a counterpart85 of th enco er. Th efore, o one hand, such a r construction paradigm can reduce an further distill86 high-level semantic cues in the attribute space V . While, on the other hand, it can driv the training87 of A2-NET by preserving e similarity betwee queried ha h codes and databas points in t rms of88 learning to hash (cf. Section 3.3.3).89 In concretely, the learning object v of unsupervised at ribute-guided learni g i written as follows:90
mi W
kX W>WXk2F s. . WX = V 0 = tanh(V ) , (6)
where the decoder (i.e., a count rpart of the encoder) is realized by W> to simplify the network.91 Howev , directly minimizing Eq. (6) with a hard constraint is hard to optimize. Therefore, we relax92 the constraint into a soft cons rai t, and the lea ning objective can be rewr tt n by93
min W
kX W>V 0k2F + kWX V 0k2F . (7)
3.3.2 Attribute-Specific Feat re Dec rrelation94
By conducting the aforementioned u supervis d attribute-guided l arning, we can obtain the intern l95 l tent vectors V 0 as t e attribute-spe ific featur s. In r r to both enhance the discrimin tive ability96 and remove the redunda t correlatio among these attri ute-sp cific fe tu es, we introduce a f ature97 d correlation constraint upon V 0, hich is formulated by98
min V 0
kV 0V 0> nIk2F , (8)
where I is the i entity matrix. Such a feature decorrelation constraint is preferable to construct99 independent features and reduce redundant information. Therefore, b sed on unsupervise attribute-100 guided learning and attribute-specific feature decorrelation, the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102
3.3.3 Hash Code Learni g103
In the following, we conduct the hash code lear ing based on the obtained attribute-specific features.104 Assume that we have n query data points which are denoted as {qi}ni=1, s well as m database points105
3
H h code learning loss (Section 3.3.3)
Attribute-aware has c des g ne tin“Red head” “On the tre ” “Spotted b dy”
2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26
3 Methodology27
In this section, we introduce the overall framework and notations of the proposed A2-NET mod l, as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29
3.1 Overall Framework and Notations30
As illustrated in Figure ??, our A2-NET model consists of two crucial modules, i.e., a fine-grain d31 representation learning module and an attribute-aware hash codes generating module. Given an32 input image Ii, based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN, a set of attention guidance A = {Ac} is learned for capturing fine-grained tailored34 local patterns T ci from Ti. To distill semantical cues and further generate the final attribute-aware35 binary hash codes, we propose to transform these appearance-specific features T towards semantic-36 specific representations T̂ by performing a transform network (·). After aggregating T̂ , the obtained37 attentive local-level features xci are associated with the global-level feature x global i to form as a38 holistic feature representation xi. In order to generate attribute-aware binary hash codes, we conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point corr spond o40 an attribute vector w.r.t. one kind of namable properties of fine-grained objects. Finally, hash code41 learning is performed upon vi to obtain the final attribute-a are binary codes ui.42
3.2 Fine-Grained Representation Learning43
Attention plays an important role in human perception [3, 7], and humans exploit a sequence of44 partial glimpses and s lective y focus on salient parts of an object or a s ene in order to bet r capture45 visual structure [10]. Inspired by this, we incorporat the attention mechanism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differ nces between these47 subordinate categories.48
In concretely, we extract the deep feature of its input image Ii via a backbone CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Then, based on Ti, C attention guidance Ac 2 RH⇥W is generated as a set of attention maps, i.e., A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or eve52 overlooked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obtain the final attribu e-aware binary co es, it is54 desirable to transform these appearance-specific (i.e., low-level) features T to semantic-specific (i.e.,55 mid-level) representations which are closer to the attribute space. Thus, a transforming network (·),56 which is equipped with a stack of convolution layers, is performed on T as follows:57
T̂ ci = (T c i ; ✓local) , (3)
T̂i = (Ti; ✓global) , (4)
where ✓ presents the para eters of the corresponding transfor ing networks w.r.t. T ci and Ti, respec-58 tively. Then, we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . The holistic60 feature representation w.r.t. the input image Ii is achieved by concatenating both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F (Ii;⇥) 2 R . Note that, we hereby abstract the aforementioned fine-grai ed62 feature learning process as a function F (Ii;⇥) with its parameters ⇥.63
2
Attention guidance
2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section, we introduce the overall framework and notations of the proposed A2-NET model, as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29 3.1 Overall Framework and Notations30
As illustrated in Figure ??, our A2-NET model consists of two crucial modul s, i.e., a fine-graine31 representation learning module and an attribute-aware hash codes generating module. Given an32 input image Ii, based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN, a set of attention guidance A = {Ac} is le rned for capturing fi -grained tailored34 local patterns T ci from Ti. To distill semantical cues and further generate the final attribute-aware35 binary hash c des, we propose to transform these appearance-specific features T t wa ds semantic-36 specific representations T̂ by performing a transform network (·). After aggregating T̂ , the obt ined37 attentive local-level features xci are associated with the global-level feature x global i to form as a38 holistic feature representation xi. In order to generate attribute-aware bi ary hash codes, e conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond to40 an attribute vector w.r.t. one kind of namable properties of fine-grained objects. Finally, hash code41 learning is performed upon vi o obtain the final attribute-aware binary codes ui.42
3.2 Fine-Grained Representation Learning43
Attention plays a imp rtant role in human perception [3, 7], and humans exploit a sequence of44 artial glimpses d selectively focus on salient parts of an object or a scene in or er to better capture45 visual structure [10]. Inspired by this, we incorporate the attention mechanism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48 In concretely, we extract the deep feature of its input image Ii via a backbone CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Then, based on Ti, C attention guidance Ac 2 RH⇥W is g nerated as a set of attention ma s, i.e., A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obtain the final attribute-aware binary codes, it is54 desirable to transform these appearance-specific (i.e., low-level) features T to semantic-specific (i.e.,55 mid-level) representations which are closer to the attribute space. Thus, a transformi g n twork (·),56 which is equipped with a stack of convolution layers, is performed on T as follows:57
T̂ ci = (T c i ; ✓local) , (3) T̂i = (Ti; ✓global) , (4)
where ✓ presents the parameters of the corresponding transforming networks w.r.t. T ci and Ti, respec-58 tively. Then, we aggregate T̂ ci and T̂i by conducting global average-pooling d correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . The holistic60 feature representation w.r.t. the input image Ii is achieved by concatenati g both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F (Ii;⇥) 2 Rd. Note that, we hereby abstract the aforementioned fine-gra ned62 feature learning process as a function F (Ii;⇥) with its parameters ⇥.63
2
2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section, we introduce the overall framework and notations of the proposed A2-NET model, as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29 3.1 Overall Framework and Notations30 As illustrated in Figure ??, our A2-NET model consists of two crucial modules, i.e., a fine-grained31 representation learning module and an attribute-aware hash codes generating module. Given an32 input image Ii, based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN, a set of attention guidance A = {Ac} is learned for capturing fine-grained tailored34 local patterns T ci from Ti. To distill semantical cues and further generate the final attribute-aware35 binary hash codes, we propose to transform these appearance-specific features T towards semantic-36 specific representations T̂ by performing a transform network (·). After aggregating T̂ , the obtained37 attentive local-level features xci are associated with the global-level feature x global i to form as a38 holistic feature representation xi. In order to generate attribute-aware binary hash codes, we conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond to40 an attribute vector w.r.t. one kind of namable properties of fine-grained objects. Finally, hash code41 learning is performed upon vi to obtain the final attribute-aware binary codes ui.42
3.2 Fine-Grained Representation Learning43
Attention plays an important role in human perception [3, 7], and humans exploit a sequence of44 partial glimpses and selectively focus on salient parts of an object or a scene in order to better capture45 visual structure [10]. Inspired by this, we incorporate the attention mechanism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48 In concretely, we extract the deep feature of its input image Ii via a backbone CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Then, based on Ti, C attention guidance Ac 2 RH⇥W is generated as a set of attention maps, i.e., A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obtain the final attribute-aware binary codes, it is54 desirable to transform these appearance-specific (i.e., low-level) f atures T to semantic-specific (i.e.,55 mid-level) representations which are closer to the attrib te space. Thus, a transforming n twork (·),56 which is equipped with a stack of convolution layers, is performed on T as follows:57
T̂ ci = (T c i ; ✓local) , (3) T̂i = (Ti; ✓global) , (4)
where ✓ presents the parameters of the corresponding transfor ing networks w.r.t. T ci and Ti, respec-58 tively. Then, we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x glo al i . Th holistic60 feature representation w.r.t. the input image Ii is achieved by concatenating both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F (Ii;⇥) 2 Rd. Note that, we hereby abstract the aforementioned fine-grained62 feature learning process as a function F (Ii;⇥) with its parameters ⇥.63
2
2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section, we introduce the overall framework and notations of the proposed A2-NET model, as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29 3.1 Overall Framework and Notations30 As illustrated in Figure ??, our A2-NET model consists of two crucial modules, i.e., a fine-grai d31 representation learning module and an attribute-aware hash codes g nerating ul . Given an32 input image Ii, based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN, a set of attention guidance A = {Ac} is learned for capturing fine-grained tailored34 local patterns T ci from Ti. To distill semantical cues and further g nerate the final attribute-aware35 bi ary ash c des, we propose to transform these appe rance-specific features towar s semantic-36 specific representations T̂ by perform ng a transform network (·). Aft r aggregating T̂ , th obtained37 attentive local-level features xci are associated with the global-level f ature x g obal i to for as a38 holistic feature representation xi. In order to generate attribute-aware binary hash codes, we conduct39 a reconstructing paradigm to project xi as vi in an attribute space wher its dat point corresp nd to40 an attribute vector w.r.t. one kind of namable propertie of fine-grained objects. Finally, hash co41 learning is performed upon vi to obtain the final attribute-aw e binary codes ui.42 3.2 Fine-Grained Representation Learning43
Attention plays an important role in human perception [3, 7], and humans exploit a sequence of44 partial glimpses and selectively focus on salient parts of an object or a scene in order to be ter capture45 visu l structure [10]. I spired by this, we incorporate th attention mech nism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48
In concretely, we extra t the deep feature of its input image Ii via a backbon CNN model CNN(·)49 s50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Then, based on Ti, C attention guidance Ac 2 RH⇥W is generated s a set of attention maps, i.e., A.51 The attention guidance Ac is designed to ev luate which deep descript rs should be attended or ven52 ov rlooked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obtain the final attribute- ware binary codes, it is54 desirable to transform these appearance-specific (i.e., low-level) features T to semantic-specific (i.e.,55 mid-level) representations which are closer to the attribute space. Thus, a transforming network (·),56 which is equipped with a stack of convolution layers, is performed on as follows:57
T̂ ci = (T c i ; ✓local) , (3) T̂i = (Ti; ✓global) , (4)
where ✓ pr sents the parameters of the corresponding tr nsforming networks w.r.t. T ci d Ti, resp c-58 tively. Then, we aggregate T̂ ci and T̂i y conducting global average-pooling and cor spo dingly59 obtain the atte tive local-level f atures xci , as well as the global-level fe tur x global i . The holisti60 fe ture representation w.r.t. the input image Ii is achieved b concatenating both xci and xgl b li , i.e.,61 xi = h xci ; x global i i = F (Ii;⇥) 2 Rd. Note hat, we hereby abstract the aforem ntion d fin -grain d62 featur learning process as a function F (Ii;⇥) with its parameters ⇥.63
2
…
2 Related Work23 2.1 Fine-Grained Image Retrieval24
2.2 Learning to Hash25
2.3 Visual Attributes26
3 Methodology27
In this secti n, we int oduce the over ll framew rk and otations f the p oposed A2-NET ode , as28 well as elaborating the detailed modules of A2-NET a its corresp nding ptimiza io algorithm.29
3.1 Overall Frame ork and No ations30
As illustrated in Figure ??, our A2-NET model consists of two crucial modules, i.e., a fi e-grained31 representation learning module and an attribute-aware ha h codes generating modul . G v n an32 input image Ii, based n its corr po d ng deep activa io ten o Ti 2 RC⇥H⇥W ex rac ed by a33 backbone CNN, a set of attention gui a ce A = {Ac} i learned for capt g fine-grained t ilored34 local patt ns T ci from Ti. To distill semantical cues and further generate the final attribute-aware35 bi ary has c des, we pro ose to transfo m these ppearan e-s ecific f atures T owards semantic-36 sp cific representations T̂ by performing a transform etwork (·). Aft r aggre ating T̂ , h obta37 attentive local-level featur s xi ar associated with t e gl bal-level f ture x global i to for s a38 holistic featu representation xi. I order to ge erate attribute-awar binary hash codes, w conduct39 a reconstructing paradigm to projec xi a vi in an attribute space wh re ts data point co respond to40 an attribute vector w.r.t. one kind of amable properties of fine-grai ed obj cts. F n lly, hash cod41 learning is performed upon vi to obtain the final attribute-aw re binary codes ui.42
3.2 Fine-Grained Representation Learning43
Att ntion lays a important rol in hu an perce tion [3, 7], nd h mans ex loit a seque c44 partial glimpses and selectively focus on salient parts of an bj ct or a scene in d r to better captu e45 visual structure [10]. Inspired by this, we incorp rate th attentio mechanism into present t on46 lear ing to c pture fine-grain d local patt rns for distinguishin subtle differences betw en th se47 subordinate categories.48
In concretely, we extract the deep feature of its input image Ii via a backbone CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Then, based on Ti, C attention guida c Ac 2 RH⇥W is gen rat d as set of attention maps, i.e., A.51 The ttent on guidance Ac s designed to ev luate which deep descriptors should be attended or even52 overl ked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obt in t e final attribute-aw re binary codes, it is54 desirable to transform these appeara ce-specific (i.e., low-level) features T to semantic-specific (i.e.,55 mid-l vel) repres ntations which are closer to the attribute space. Thus, a transforming network (·),56 which is equipped with a stack of convolution layers, is performed n T as follows:57
T̂ ci = (T c i ; ✓local) , (3) T̂i = (Ti; ✓global) , (4)
where ✓ prese ts the parameters of the corresponding transforming networks w.r.t. T ci and Ti, respec-58 tively. Then, we aggregat T̂ ci a d T̂i by conducting global average-po ling nd c rr spo dingly59 obtain the attentiv local-level f atures xci , as well as the global-level feature x global i . Th holistic60 feature representation w.r.t. the input imag Ii i achieved by concatenating both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F (Ii;⇥) 2 Rd. Note that, we h reby abstract the aforementioned fine-grained62 feature learning process as a function F (Ii;⇥) with its parameters ⇥.63
2
2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section, we introduce the overall framework and notations of the proposed A2-NET model, as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29 3.1 Overall Framework and Notations30 As illustrated in Figure ??, our A2-NET model consists of two crucial modules, i.e., a fine-grained31 representation learning module and an attribute-aware hash codes generating module. Given an32 input image Ii, based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN, a set of attention guidance A = {Ac} is learned for capturing fine-grained tailored34 local patterns T ci from Ti. To distill semanti al cu s d further generate the final attribute-aware35 binary hash codes, we propose to transform these appearance-specific features T towards semantic-36 specific representations T̂ by performing a transform network (·). After aggregating T̂ , the obtained37 attentive local-level features xci are associated with the global-level feature x global i to form as a38 holistic feature representation xi. In order to generate attribute-aware binary hash codes, we conduct39 a r constructing paradigm to project xi as vi in an attribute sp ce where its data point correspond to40 an attribute vector w.r.t. one kind of namable properties of fine-grained objects. Finally, hash code41 lear ing is perfor ed upon vi to obtain the final attribute-aware binary codes ui.42 3.2 Fine-Grained Representation Learning43 Attention plays an important role in human percept on [3, 7], and humans exploit a s quence of44 partial glimpses and selectively focus on salient parts of an object or a scene in order to better capture45 visual structure [10]. Inspired by this, we inc rporate t e attention mechanism into repres ntation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48 In concretely, we extract the deep feature f its i put im ge Ii via a ba kbone CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Then, based on Ti, C attention guidance Ac 2 RH⇥W is generated as a set of ttention maps, i.e., A.51 The attention guidance Ac is designed to evaluate which deep descrip ors sh uld be attended or even52 overlooked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obtain the final attribut -aware binary codes, t is54 desirable to tran for these appearance-specific (i.e., low-level) features T to semantic-spec fic (i. .,55 mid-level) representations which are closer to the attribute space. Thus, a transforming network (·),56 which is equipped with a stack of convolution layers, is performed on T as follows:57
T̂ ci = (T c i ; ✓local) , (3) T̂i = (Ti; ✓global) , (4)
where ✓ pr sents the parameters of he c rrespo ding transforming networks w.r.t. T ci a d Ti, respec-58 tively. Then, we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondi gly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . Th h lis ic60 feature rep esentation w.r.t. the input image Ii i achieved by concat ati g both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F (Ii;⇥) 2 Rd. Note that, we hereby abstract the aforem ntioned fine-grained62 f ature le ning pro ess as a function F (Ii;⇥) with its parameters ⇥.63
2
Aggregation …
2 Related Work23 2.1 Fine-Grained Image Retr val24 2.2 Le rning to Hash25 2.3 Visual Attribut s26 3 Method logy27
In this section, we introduc the ove ll framework nd n t ti ns of the prop s A2-NET model, as28 well as elaborating the detailed mo ules f A2-NET and it rr spo ding ptimization algorithm.29
3.1 Over l Fra work and Notations30
As illustrated in Figur ??, o r A2-NET model onsist of two crucial m dul s, i.e., a fin -gr ined31 representation learning modul and a attribute-awar h sh c des ge e ating mo l . Given a32 input image Ii, based on its cor es onding d p activatio tens r Ti 2 RC⇥H⇥W extracted by a33 back o e CNN, a set of ttention guidance A = {Ac} l rned f c pt i fine-gra ned t i or d34 local patterns T ci from Ti. To d still semantical c es and further ge rate the fi al a tribut -aw re35 binary hash cod s, we propose to tr nsform these ppearance-specific f atures T tow rds se ntic-36 specific repres ntations T̂ erforming a tra sform n twork (·). Aft aggr ga i g T̂ , the obtai d37 attentive local-level feat res xci are associated with the global-level featur x global i t f rm as a38 holistic e ture r pr s ntation xi. I order to g nerate a tribute- war binary hash codes, we condu t39 a econstructing p adigm to project xi as vi i an attri te space w re it da p i t co resp d t40 an ttribute tor w.r.t. o kin f n m ble r perti o fin in d bject . Fi ally, ash cod41 le rning is p rformed pon vi to obtain th fin l at t - war bi ry co es ui.42
3.2 Fi e-Grained Representation L rning43
Attention plays an important role in human perceptio [3, 7], and hu ans exploit a s qu nce of44 partial glimpses and selectiv ly focus on sali nt parts of an object or a scene in or e to b tt r capture45 visual structur [10]. Inspired by this, we incorporate he att ntion mech ism int represent tion46 learning to capture fin - rained l cal patterns f r disti guishing subtle differ n es betw e these47 s b rdi te ategories.48
In concr tely, w extract the de p feature of i s input image Ii vi a backbone CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Th n, based o i, C attention uidan e Ac 2 RH⇥W is g nerated as a set f att ntion m ps, i.e., A.51 The atte ion i ance Ac is designed to evaluate which deep descriptors should b attended or ven52 overlooked by conducting53
T ci = c Ti , (2)
where is the element-wise Hadamard product. To o tain the fina attribute-aware binary odes, it is54 desirable to tran form these appearance-specific (i.e., low-level) features T to emantic-specific (i. .,55 mid-level) representations whi h are clos r to the attribute space. Thu , a tr nsforming network (·),56 which is equipped with a stack of convolution layers, is performed on T as follows:57
T̂ ci = (T c i ; ✓local) , (3)
T̂i = (Ti; ✓global) , (4)
wher ✓ presents the parameters of the correspo ding transforming network w.r.t. T ci and i, respec-58 tively. Then, we ggrega e T̂ ci and T̂i by c ucting global av rage-pooli g nd corr pondingly59 o tain the attentive local-level featur s xci , as well a the global-l vel feature x global i . The holistic60 feature representation w.r.t. the i put mage Ii is chi ved by concatenating both xci and xgl bali , .e.,61 xi = h xci ;
l al = F (Ii;⇥) 2 Rd. Note that, we hereby abstract the aforementioned fine-grained62 feature learnin proc ss as a function F (Ii;⇥) with its p ram t rs ⇥.63
2
2 Rel t d Work23
2.1 Fi -Grained Image Retrieval24
2.2 Learni g to Ha h25
2.3 Visu l Attributes26
3 Methodology27
In this section, we introduce the overall framework and notations of the proposed A2-NET model, as28 w ll as elaborating the detailed modules of A2-NET and its correspon ing optimization algorithm.29
3.1 Overall Fram work and Notation30
As illustrated in Figur ??, our A2-NET model consists of two cruc al mo ules, i.e., fi e-grai ed31 r p tatio l arning modul nd an attribut - ware has codes generating module. Given an32 inp t image Ii, b sed o its respond ng deep a tivation tensor Ti 2 RC⇥H⇥W extracted by a33 ba kbone CNN, a set of attention g idance A = {Ac} is learned for captu ing fine-gr ined tailor d34 l l patterns T ci from Ti. To distill semantical cues and further en rate the final ttribute- ware35 binary hash odes, we prop se to transform these ppearance-specific features T towards semantic-36 specific representations T̂ by performing a transform netwo k (·). After aggregat ng T̂ , he obt ined37 attentive local-level features xci re associated with the global-level feature x global i to form as a38 holistic feature representation xi. In order to generate attribute-aware binary hash codes, we onduct39 a reconstructing paradigm to project xi as vi in an attribute space where i s data po nt correspo d to40 an attribute vector w.r.t. one kind of namable properties of fine-grained obj cts. Finally, has c de41 learning is performed upon vi to obtain the fi al attribute-aware binary codes ui.42
3.2 Fine-Grained Representation Learning43
Attention plays a important role in human perception [3, 7], and humans exploit a sequence of44 partial glimpses and select vely focus on salient parts of an bject or a scene in order to be ter capture45 visual structure [10]. Inspired by this, we incorporate the attention mechanism into representation46 learning to c pture fine-grained local patterns for distinguishing subtle difference b tween th se47 subordinate c tegories.48
In concretely, we extract the deep feature of its input image Ii via a backbone CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Then, based on Ti, C attention guidance Ac 2 RH⇥W is generat d as a set of attention maps, i.e., A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be tte d d or even52 overlooked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obtain the final attribute-aware binary codes, it is54 desirable to transform these appearance-specific (i.e., low-level) features T to semantic-specific (i.e.,55 mid-level) representations which are closer to the attribute space. us, transf rming netw rk (·),56 which is equipped with a stack of convolution layers, is performed on T as follows:57
T̂ ci = (T c i ; ✓local) , (3)
T̂i = (Ti; ✓global) , (4)
where ✓ presents the parameters of the corresponding transformi g networks w.r. . T ci and Ti, r spec-58 tively. Then, we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . The holistic60 feature representation w.r.t. the input image Ii is achieved by concatenating both xci and xglobali , i. .,61 xi = h xci ; x global i i = F (Ii;⇥) 2 Rd. Note that, we her by abstr c the af rementioned fine-gr ined62 feature learning process as a function F (Ii;⇥) with its parameters ⇥.63
2
2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section, we introduce the overall framework and notations of the proposed A2-NET model, as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorith .29 3.1 Overall Framework and Notations30 As illustrated in Figure ??, our A2-NET model cons sts of two crucial modules, i.e., a fine-grain d31 representation learning module and an attribute-aware hash codes gene ating module. Given an32 input image Ii, based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN, a set of attention guidance A = {Ac} is learned for capturing fin -grained tail red34 local patterns T ci from Ti. To distill semantical cues and further generat the final at ribute-aware35 binary hash codes, we propose to transfor th se appearance-specific features T towards semantic-36 specific representations T̂ by performing a transform network (·). Aft r aggregating T̂ , the obtained37 attentive local-level features xci are asso iated with the global-level feature x global i o form as a38 holistic featur representation xi. In ord r to generate attribute-awar binary hash codes, we conduct39 a reconstructing paradigm to project xi as vi in an attribute space wh re its data point corr spond to40 an attribute vector w.r.t. one kind of namable properties of fine-grained objects. Finally, hash code41 learning is performed upon vi to obtain the final attribute-aware binary codes ui.42
3.2 Fine-Grain d Representation Learning43
Attention plays an important role in human perception [3, 7], and humans exploit a sequence of44 partial glimps s and s lectively focus on s lient parts of an object or a scene in order to better capture45 visual structure [10]. Inspired by this, we incorporate the attention mechanism into repr sentation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48
In concretely, we extract the deep feature of its nput image Ii via a backbone CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Then, based on Ti, C attention guidance Ac 2 RH⇥W is generated as a set of attention maps, i.e., A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadama d product. T obtain he final a tribute-aware binary cod s, it is54 desirable to transform these appearance-specific (i.e., low-level) features T to semantic-specific (i.e.,55 mid-level) representations which are closer to the attribute space. Thus, a transforming netwo k (·),56 which is equipped with a stack of convolution layers, is performed on T as follows:57
T̂ ci = ( i ; local) , (3) T̂i = (Ti; ✓global) , (4)
where ✓ presents the parameters of the corresponding transforming networks w.r.t. T ci and Ti, respec-58 tively. Then, we aggregate T̂ ci and T̂i by onducting glob l average-poolin and correspondingly59 obtain the attentive local-level features xci , as w ll as the gl bal-level feature x global i . Th holistic60 feature representation w.r.t. the input image Ii is achiev d by concat nating both xci a d xglobali , i.e.,61 xi = h xci ; x global i i = F (Ii;⇥) 2 Rd. Note that, we hereby abstract the aforementioned fine-grain d62 feature learning process as a function F (Ii;⇥) with its parameters ⇥.63
2
Aggregation
where t e d coder (i.e., a counterpart of the encoder) is realized by W> to simplify the netwo k.91 However, directly minimizing Eq. (6) with a hard constraint is hard to optimize. Therefore, we relax92 the constraint into a soft constraint, and the l rning objective ca be rewritten by93 min W kX W>V 0k2F + kWX V 0k2F . (7) 3.3.2 Attribute-Specifi F ur D orrelatio94 By conducting the afo em ntio d unsupervi ed att ibut -g ided learning, we an obtain the int rn l95 l tent vecto s V 0 as the ttribute-specific feature . I ord to b h e h nce th discrimi ativ bility96 and r move the r dundan correlation among thes ttribute-sp cific features, we introduce a feature97 decorrelation c n r int upon V 0, which is formulat d by98
m n V 0 kV 0V 0> Ik2F , (8) where I is the identity matrix. Such a featur ecor lation co strai t is pref rable to const uct99 independent features and reduce redu da t information. Therefore, ba ed on u supervised ttribut -100 guided learning d attribute-specific featur decorrelation, th fi a learn d hash codes a e expected101 to be bo h attribute-awar a d hash-bi ind pe d nt.102
3.3.3 H sh Co e L arning103
In the f ll wing, we conduct the hash code learning bas d on the obt ned at ribute-sp cific featur s.104 Assume that w have n qu ry data p ints which re enoted as {qi}i=1, as w ll m databa points105 which a noted a {vj}j=1. Note ha , both qi and vi ar belo gin o th att ibute space V . Th n,106 the corresponding bi ary has codes ca b obt ined via107
ui = sgn(tanh(qi)) , (9) zj = sgn(tanh(vj)) , (10)
where ui, zj 2 { 1, +1}k. The g al f our h cod learning is to le r binary ash cod s f r t108 qu ry poin s and database points from {qi} =1, {vj}mj=1, and the pairwise sup vised information,109 i. ., S 2 { 1, +1}n m. To pr se ve the pair simil ri y, adopt t `2 loss b tween the110 supervised inf rmati ( .k.a. similarity) d t inn r product f query-dat base point binary cod111 pairs. It can be formulated as follows:112
inU ,Z P
=1 P j=1 u>i zj kSij 2
s.t. U 2 { 1, +1}n⇥k, Z 2 { 1, +1}m⇥k . (11)
Overall, e g t the fina obj tiv f t e propo ed A2-NET mod l by co sidering Eq. (7), Eq. (8) and113 Eq. (11) toge a follows:114
min W ,⇥
L(I) = kX W>V 0k2F + kWX V 0k2F +↵kV 0V 0> nIk2F + nX
i=1
mX
j=1
> i zj kSij 2 ,
(12) where , ↵ and are hyper-parameters as the tr de-of .115
In practice, during training, it ght b only ava labl et of at ase poin s {vj}mj=1 with ut query116 points. Thus, we randomly sample n ata point from database to construct a query set, and d te117 the indices of all the database points as with the indices of th query set as ⌦. Additionally, because118 we cannot back-propagate the gradient to ⇥ due to the sgn(·) function, w omit the sgn(·) function119 and o ly apply t nh(·) f r elaxation in Eq. (10) of the whole optimization process. Th refore, he120 optimization formulation of A2-NET can be rewritten with only database points {vj}mj=1 for rai ing121 as:122
min W ,⇥
L(I) = kX W>V 0k2F + kWX V 0k2F + ↵kV 0V 0> nIk2F
+ X
i2⌦
X
j2
⇣ tanh(W · F (Ii;⇥)) zj kSij ⌘2 . (13)
For optimization, our A2-NET does not require co plicated two-stage learning algorithms, e.g., th123 alternative optimization strategy. W employ the back-propagation algorith and follow [8] to rain124 the whol A2-NET model in a u ified end-to-end a ner.125
4
“Blue head”
Figure 1: Overall fra ewo k of the roposed A2-NET m el, whic consis s f t o crucial modules, i.e., fine-grained representation learni g and attribute-aware hash codes ge er ting. The whole netw rk c n be end-to-e d trainabl , d is gener lly riv n b th sup rvi d tr bu e-guided loss, the feature decorrelation loss and he hash cod le rn g loss, cf. S tio 3.3.
has th potenti l to help with large-sc le fine- rai ed d ta t o. Therefore, targeting the ig data35 challenge, fine-grained hashing [7, 13] is a promisi g direction orthy f further explorations.36
low-level to mid-level appearance-specific to s antic-specific37
obtain corresponding fine-grai ed obj ct attribut s (e.g., xx), ithout signifi ant annotati cost38
2 Relat d Work39
In this section, we briefly review t e r ated work i t following t ree aspects, i.e., n -g ained40 image retrieval, learning to hash and the studies about visual attributes in co puter vision p oblems.41
2.1 Fine-Grained Image Retrieval42
Fine-grained image retrieval as an integral part of fine-grained image analysis [34] has gained more43 and more tractio i recent years [7, 20, 23, 33, 35, 37, 38]. What makes it challenging is that44 objects of fine-grained images have only subtle differences, and often largely vary n po e, scale, and45 orientation or can exhibit cross-modal differences (e.g., sketch-based retrieval [20]).46
Depending on the type of query image, the most studied areas of fine-grain d imag retrieval47 can be separated into two groups: fine-grained content-based image retrieval (FG-CBIR) and fine-48 grained sketch-based image retrieval (FG-SBIR). More sp cifically, in FG-CBIR, unsupervis d49 learning based [33] and supervised l rning bas d methods [36–38] were developed from differe t50 perspectives for dealing with fine-grained retriev tasks, e.g., localizing fine-grained parts [33],51 enhancing intra-class separability with inter-class compact ess [38], and re uci g th confidence52 of the fine-grained predictions [36], etc. While, FG-SBIR needs to not only capture ne-graine53 characteristics present in the sketches, but also possess the ability to traverse the sketc and image54 domain gap. In the literature of FG-SBIR, the earlier works, e.g., [17, 27, 35], were mostly based55 on Siamese-triplet networks [3] to tackle the aforementioned c allenges. Recently, some works56 tried to incorporate the advances of recent progress in self-supervised learning [5] and attention57 mechanisms [6] for further improving the retrieval accuracy of FG-SBIR, e.g., [23, 25].58
However, although these fine-grained retrieval methods achieved good results, they till show the59 limitations in the face of large-scale data, i.e., the searching time for exact nearest neighbor is typically60 expensive or even impossible for the given queries. To alleviate this issue, fine-gr ined hashing, which61 aims to generate compact binary codes to represent fine-grained images, as a promising direction62 has attracted the attention in the fine-grained community very recently [7, 14]. More sp cifically,63
2
R d_head?
Blue_tail?
O _the_tr e?
Spotted_body?
YES
NO
YES
YES
1
0
1
1
Att ibute- ware has cod s
R t ieval
Top-k retriev d images
1
0
1
1
1
0
1
1
1
0
1
0
Figure 1: K y idea of t e propos A2-NET mo l, s w ll th ain proces of fine rai d ha hing ased o our attribute-aware has c des. In co cr t ly, for a qu y i Iq of R b llied Woodpecker, aft r r tur ing all the correct re ults, t fin -grained i age (b lon i g to R d h d Wo dpecker) closest to t e qu ry image is al o r triev .
A2-NET: Le r ing Attri u -A re Has C es or L g -Scal Fi -Gr I ge Ret i v l
Anonymous Author(s) Affiliation Address email
Abst ct
The task of fine-grained image retrieval aims to r nk all th instances so that images1 depicting the concept of interests (i.e., the same sub-category labels) are ranked2 highest based on the fine-grained details in the query. It is challenging due to the3 subtle differences between sub-category, as well as the fine-grained nature of small4 inter-class variations and large intra-class variations. Especially for large-scale5 data, it is desirable to be ha dled in a bo h effective and efficient way. In this paper,6 we an Attribute-Aware hashing Network, i. ., A2-NET, for generating an attribute-7 aware binary codes to not only speed up the retrieval process, but also corresponds8 to fine-grained attributes. Specifically, based on the learned attention maps, we9 design an auto-encoder to project the visual feature vector into an attribute space10 by following a reconstructing paradigm. Furthermore, A2-NET is also equipped11 with an orthogonal constraint upon the generated attributes to enhance their own12 representation abilities. Qualitative experiments o five be c mark fine-grained13 datasets show our superiority over competing methods. Moreov r, quantitative14 results reveal the obtained hash codes correspond t the attribut s of fine-grained15 patterns.16
1 Introduction17
another barrier also occurs because of xxx18
suffer from slow query speed and redundant storage costs due to both the explosive growth of massive19 fine-grained data and high-dimensional real-valued features.20
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.
F gur 1: K y id a f ur A2-NET, as w ll s th mai oc s of fine-grained hashing based on ur t ibut - w h sh c des. I conc t ly, regar ing a query image Iq of Red bellied Wo d ck , af er ret r i g all the corr ct results, a fine-gra n d image b longing to Red headed Wo dp ck r closes to th qu y ag in erms of H mm distance s also retriev d.
.g., [8, 18], chi v g o retri val perf a c , the bit f th ir hash codes correspond to no seman ics, i.e., fi -gra ned at ributes. While, such t ributes, e.g., head color, ail color, male, f ale, living habits, a e great means of des ribing fine-grained objects, in a way both humans and computers understand. In this paper, to st blish an explicit correspondence between hash codes and visu l attri ut s for not only fu the impr vi g rge-scal fine-grained retrieval accuracy, but more importa tly integrati g interpr tation into deep l arning based hash methods, we propose a unified Attribute-Aw r hashing Ne work, te med s A2-NET (cf. Figure 1), for achieving these goals.
In ou A2-NET, on d ring huge l b r cost f supervised attribute annotations, we restrict ourselves in an unsup rvised setting to automatically c pture di criminative visual attributes from still images an the correspond the final learned hash code representatio s to these attributes. Therefore, a has bit of l ar ed hash co s could be both d scriminative and intuitive. Additionally, thanks to h u supervis d setting, the attributes derived from A2-NET will be not restricted to pre-defined
attributes like supervise -based attribut learning methods [15, 21, 42, 45]. Moreover, it can distill the most useful properties of fine-grained objects as attribute-aware hash codes in such an end-to-end trai able m ner for accuracy retrieval among multiple similar subordinate categories.
More specific lly, as the overall framework shown in Figure 2, our A2-NET consists of a finegrai ed represe tation learni g module and an a tribute-aware hash codes generating module. It first leverages attention mechanisms to model fine-grained tailored patterns in terms of both global-level deep featur s Ti and local-level ues T ci from inp t imag Ii. Then, the appearance-specific features of th se vis l p t rns T are aggr gated and tran lat d into semantic-specific representations xi. After that, we formulate the aforementioned unsupervised attribute learning as a reconstruction task of projecting xi to an attribute vector vi by performing an encoder-decoder structure network. T erefor , it can be xpected that in the high-level at ribute space, vi could correspond to certain kinds of nameable properties of fine-grained objects. Moreover, a feature decorrelation constraint is further introduced upon vi to both enhance t e discriminative bility and remove the redundant correlation among these dimensions of attribute-specific features. Finally, our attribute-aware hash codes ui are enerated from vi by conducting the hash code learning procedure.
To evaluate our model, we conduct extensive experiments using five benchmark fine-grained retrieval datasets for both accuracy and interpretability. Quantitative results of retrieval accuracy on these datasets show that the proposed A2-NET model obviously and consistently outperforms existing stateof-the-art methods. Qualitative visualization of the obtained attribute-aware hash codes demonstrates that these hash bits have strong correspondences to visual attributes of fine-grained objects (cf. Figure 4), even without employing attribute supervisions or part-level annotations. In addition, the ablation studies of these crucial components in A2-NET also validate their own effectiveness.
2 Related Work
2.1 Fine-Grained Image Retrieval
Fine-grained image retrieval as an integral part of fine-grained image analysis [41] has gained more and more traction in recent years [8, 25, 29, 40, 43, 47, 48]. What makes it challenging is that
101 1
Encoder
3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET model. We elaborate it in65 the following four aspects, i.e., unsupervised attribute-guided learning, attribute-specific feature66 decorrelation, hash code learning and the optimization learning algorithm.67
3.3.1 Unsupervised Attribute-Guided Learning68
In real-applications, especially for the large-scale and fine-grained tasks, attribute annotations are69 always infeasible, which limits the learning process to be conducted in an unsupervised setting.70 While, in the literature, the main goal of unsupervised learning is to capture regularities in data for the71 purpose of extracting useful representations or for restoring corrupted data [4]. Many unsupervised72 methods explicitly produce internal latent units or codes, from which the data is to be reconstructed.73
Inspired by this, we develop an unsupervised attribute-guided learning component to project the74 holistic representation xi of Ii into a latent space, i.e., the attribute space V . In V , its high-level75 vectors are designed to have certain desirable properties, e.g., corresponding to semantic properties76 of fine-grained objects (a.k.a. “fine-grained attributes”).77
More specifically, in our A2-NET, the unsupervised attribute-guided learning is realized by a recon-78 struction paradigm with an encoder-decoder structure, as shown in Figure ??. In concretely, given79 a batch of n training data Ii, the holistic representations X = {x1; x2; . . . ; xn} 2 Rd⇥n can be80 obtained as aforementioned. By formulation, the encoder projects X into the attribute space V with81 a projection matrix W 2 Rk⇥d to get an internal latent representation V 2 Rk⇥n w.r.t. X . In82 particularly, we set that the dimension of latent representation k equals to the number of hash bits in83 the final binary hash code ui. Furthermore, each column of V , i.e., vi 2 Rk, can derive ui by84
ui = sgn(tanh(vi)) . (5)
Meanwhile, regarding vi, we also consider to reconstruct its input xi by a decoder as a counterpart85 of the encoder. Therefore, on one hand, such a reconstruction paradigm can reduce and further distill86 high-level semantic cues in the attribute space V . While, on the other hand, it can drive the training87 of A2-NET by preserving the similarity between queried hash codes and database points in terms of88 learning to hash (cf. Section 3.3.3).89
In concretely, the learning objective of unsupervised attribute-guided learning is written as follows:90
min W
kX W>WXk2F s.t. WX = V 0 = tanh(V ) , (6)
where the decoder (i.e., a counterpart of the encoder) is realized by W> to simplify the network.91 However, directly minimizing Eq. (6) with a hard constraint is hard to optimize. Therefore, we relax92 the constraint into a soft constraint, and the learning objective can be rewritten by93
min W
kX W>V 0k2F + kWX V 0k2F . (7)
3.3.2 Attribute-Specific Feature Decorrelation94
By conducting the aforementioned unsupervised attribute-guided learning, we can obtain the internal95 latent vectors V 0 as the attribute-specific features. In order to both enhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features, we introduce a feature97 decorrelation constraint upon V 0, which is formulated by98
min V 0
kV 0V 0> kIk2F , (8)
where I is the identity matrix. Such a feature decorrelation constraint is preferable to construct99 independent features and reduce redundant information. Therefore, based on unsupervised attribute-100 guided learning and attribute-specific feature decorrelation, the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102
3.3.3 Hash Code Learning103
In the following, we conduct the hash code learning based on the obtained attribute-specific features.104 Assume that we have n query data points which are denoted as {qi}ni=1, as well as m database points105
3
3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET model. We elaborate it in65 the following four aspects, i.e., unsupervised attribute-guided learning, attribute-specific feature66 decorrelation, hash code learning and the optimization learning algorithm.67 3.3.1 Unsupervised Attribute-Guided Learning68
In real-applications, especially for the large-scale and fine-grained tasks, attribute annotations are69 always infeasible, which limits the learning process to be conducted in an unsupervised setting.70 While, in the literature, the main goal of unsupervised learning is to capture regularities in data for the71 purpose of extracting useful representations or for restoring corrupted data [4]. Many unsupervised72 methods explicitly produce internal latent units or codes, from which the data is to be reconstructed.73
Inspired by this, we develop an unsupervised attribute-guided learning component to project the74 holistic representation xi of Ii into a latent space, i.e., the attribute space V . In V , its high-level75 vectors are designed to have certain desirable properties, e.g., corresponding to semantic properties76 of fine-grained objects (a.k.a. “fine-grained attributes”).77
More specifically, in our A2-NET, the unsupervised attribute-guided learning is realized by a recon-78 struction paradigm with an encoder-decoder structure, as shown in Figure ??. In concretely, given79 a batch of n training data Ii, the holistic representations X = {x1; x2; . . . ; xn} 2 Rd⇥n can be80 obtained as aforementioned. By formulati n, th en oder pr jects X into the attribute sp ce V with81 a projection matrix W 2 Rk⇥d to get an internal latent representation V 2 Rk⇥n w. .t. X . In82 particularly, we set that the dimension of latent representation k equals to the number of hash bits in83 the final binary hash code ui. Furthermore, each column of V , i.e., vi 2 Rk, ca derive ui by84
ui = sgn(tanh(vi)) . (5)
Meanwhile, regarding vi, we also consider to reconstruct its inp t xi by a decoder as a cou terpart85 of the encoder. Therefore, on one hand, such a reconstruction paradigm can reduce and further distill86 high-level semantic cues in the attribute space V . While, on the other hand, it can drive the training87 of A2-NET by preserving the similarity between queried hash cod s and database points in terms of88 learning to hash (cf. Section 3.3.3).89
In concretely, the learning objective of unsupervised attribute-guided learning is written as follows:90
min W
kX W>WXk2F s.t. WX = V 0 = tanh(V ) , (6)
where the decoder (i.e., a counterpart of the encoder) is realized by W> to simplify the network.91 However, directly minimizing Eq. (6) with a hard constraint is hard to optimize. Therefore, we relax92 the constraint into a soft constraint, and the learning objective can be rewritten by93
min W
kX W>V 0k2F + kWX V 0k2F . (7)
3.3.2 Attribute-Specific Feature Decorrelation94
By conducting the aforementioned unsupervised attribute-guided learning, we can obtain the internal95 latent vectors V 0 as the attribute-specific features. In order to both nhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features, we introduce a feature97 decorrelation constraint upon V 0, which is formulated by98
mi V 0
kV 0V 0> kIk2F , (8)
where I is the identity matrix. Such a feature decorrelation constraint is preferable to construct99 independent features and reduce redundant information. Therefore, based on unsupervised attribute-100 guided learning and attribute-specific feature decorrelation, the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102
3.3.3 Hash Code Learning103
In the following, we conduct the hash code learning based on the obtained attribute-specific features.104 Assume that we have n query data points which are denoted as {qi}ni=1, as well as m database points105
3
3.3 Attribute-Aware Hash C es Gen ating64
How to generate attribute-aware hash codes is the key of our A2-NET model. We elaborate it in65 the following four aspec s, i.e., unsup rvised attribute-guided learning, attribute-specific feature66 decorrelati n, hash code learning and the optimization learning algorithm.67
3.3.1 Unsupervised Attribute-Guided Learning68
In real-applications, especially for the large-scale and fine-grained tasks, attribute annotations are69 always infeasible, which limits the learning process to be conducted in an unsupervised setting.70 While, in the literature, the main goal of unsupervised learning is to capture regularities in data for the71 purpose of extracting useful representations or for restoring corrupted data [4]. Many unsupervised72 methods explicitly produce internal latent u its or codes, from which the data is to be reconstructed.73
Inspired by this, we develop an unsupervised attribute-guided learning component to project the74 holistic representation xi of Ii into a latent space, i.e., the attribute space V . In V , its high-level75 vectors are designed to have certain desirable properties, e.g., corresponding to semantic properties76 of fine-grained objects (a.k.a. “fine-grained attributes”).77
More specifically, in our A2-NET, the unsupervised attribute-guided learning is realized by a recon-78 structi n paradigm with an encoder-decod r structure, as shown in Figure ??. In concretely, given79 a batch of n training data Ii, the holistic representations X = {x1; x2; . . . ; xn} 2 Rd⇥n can be80 obtained as aforementioned. By formulation, the encoder projects X into the attribute space V with81 a projectio matrix W 2 Rk⇥d to get an nternal latent representation V 2 Rk⇥n w.r.t. X . I82 particularly, w set that the dimension of latent repre entation k equals t the number of hash bits in83 the final binary sh de ui. Furthermore, each column of V , i.e., vi 2 Rk, can derive ui by84
ui = sgn(tanh(vi)) . (5)
Meanwhile, regarding vi, we also consider to reconstruct its input xi by a decoder as a counterpart85
of the encod r. Therefore, on one hand, uch a reconstructio paradigm can reduce and further distill86
high-level semantic cues in the attribute space V . While, on the other hand, it can drive the training87
of A2-NET by preserving the similarity between queried hash codes and database points in terms of88
learning to hash (cf. Section 3.3.3).89
In con retely, the learn ng objective of u supervised attribute-guided learning is written as follows:90
min W
kX W>WXk2F .t. WX = V 0 = tanh(V ) , (6)
where the decoder (i.e., a counterpart f the encoder) is r alized y W> to simplify the network.91 However, irectly minimizi g Eq. (6) with a hard con traint is hard to optimize. Therefore, we relax92 the constraint into a soft constraint, and the learning objective can be rewritten by93
min W
kX W>V 0k2F + kWX V 0k2F . (7)
3.3.2 Attribute-Specific Feature Decorrelation94
By conducting the aforementioned unsupervised attribute-guided learning, we can obtain the internal95 latent vectors V 0 as the attribute-specific features. In order to both enhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features, we introduce a feature97 decorrelation constraint upon V 0, which is formulated by98
min V 0
kV 0V 0> kIk2F , (8)
where I is the identity matrix. Such a feature decorrelation constraint is preferable to construct99 independent features and reduce redundant information. Therefore, based on unsupervised attribute-100 guided learning and attribute-specific feature decorrelation, the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102
3.3.3 Hash Code Learning103
In the following, we conduct the hash code learning based on the obtained attribute-specific features.104 Assume that we have n query data points which are denoted as {qi}ni=1, as well as m database points105
3
3.3 Attribute-Aware Hash Codes Generating64
How to generate attribute-aware hash codes is the key of our A2-NET model. We elaborat it in65 the following four aspects, i.e., unsupervised attribute-guided learning, attribute-specific feature66 decorrelation, hash code learning and the optimization learning algorithm.67
3.3.1 Unsupervised Attribute-Guided Learning68
In real-applications, especially for the large-sca e and fine-grained tasks, attribute an otation re69 always infeasible, which limits the learning process to be nducted in an unsupervised setting.70 While, in the literature, the main goal of unsupervised learning is to capture regularities in data for the71 purpose of extracting useful representations or for restoring corrupted data [4]. Many unsupervised72 methods explicitly produce internal latent units or codes, from which the data is to be reconstructed.73
Inspired by this, we develop an unsupervised attribute-guided learning component to project the74 holistic representation xi of Ii into a latent space, i.e., the attribute sp ce V . In V , its high-lev l75 vectors are designed to have certain desirable properties, e.g., corresponding to semantic properties76 of fine-grained objects (a.k.a. “fine-grained attributes”).77
More specifically, in our A2-NET, the unsupervised attribute-guided learning is realized by a recon-78 struction paradigm with an encoder-decoder structure, as shown in Figure ??. In concretely, given79 a batch of n training data Ii, the holistic representations X = {x1; x2; . . . ; xn} 2 Rd⇥n can be80 obtained as aforementioned. By formulation, the encoder projects X into the attribute space V with81 a projection matrix W 2 Rk⇥d to get an internal latent representation V 2 Rk⇥n w.r.t. X . In82 particularly, we set that the dimension of latent representation k equals to the number of hash bits in83 the final binary hash code ui. Furthermore, each column of V , i.e., vi 2 Rk, can derive ui by84
ui = sgn(tanh(vi)) . (5)
Meanwhile, regarding vi, we also consider to reconstruct its input xi by a decoder as a counterpart85
of the encoder. Therefore, on one hand, such a reconstruction paradigm can reduce and further distill86
high-level semantic cues in the attribute space V . While, on the other hand, it can drive the training87
of A2-NET by preserving the similarity between queried hash codes and database points in terms of88
learning to hash (cf. Section 3.3.3).89
In concretely, the learning objective of unsupervised attribute-guided learning is written as follows:90
min W
kX W>WXk2F s.t. WX = V 0 = tanh(V ) , (6)
where the decoder (i.e., a counterpart of the encoder) is realized by W> to simplify the network.91 However, directly minimizing Eq. (6) with a hard constraint is hard to optimize. Therefore, we relax92 the constraint into a soft constraint, and the learning objective can be rewritten by93
min W
kX W>V 0k2F + kWX V 0k2F . (7)
3.3.2 Attribute-Specific Feature Decorrelation94
By conducting the aforementioned unsupervised attribute-guided learning, we can obtain the internal95 latent vectors V 0 as the attribute-specific features. In order to both enhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features, we introduce a feature97 decorrelation constraint upon V 0, which is formulated by98
min V 0
kV 0V 0> kIk2F , (8)
where I is the identity matrix. Such a feature decorrelation constraint is pref rable to construct99 independent features and reduce redundant information. Therefore, based on unsupervise ttri t -100 guided learning and attribute-specific feature decorrelation, the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102
3.3.3 Hash Code Learning103
In the following, we conduct the hash code learning based on the obtained attribute-sp cific feat res.104 Assume that we have n query data points which are denoted as {qi}ni=1, as well as m database points105
3
Decoder
3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET model. We elaborate it in65 the following four aspects, i.e., unsupervised attribute-guided learning, attribute-specific feature66 decorrelation, hash code learning and the optimization learning algorithm.67 3.3.1 Unsupervised Attribute-Guided Learning68 In real-applications, especially for the large-scale and fine-grained tasks, attribute annotations are69 always infeasible, which limits the learning process to be conducted in an unsupervised setting.70 While, in the literature, the main goal of unsupervised learning is to capture regularities in data for the71 purpose of extracting useful representations or for restoring corrupted data [4]. Many unsupervised72 methods explicitly produce internal latent units or codes, from which the data is to be reconstructed.73 Inspired by this, we develop an unsupervised attribute-guided learning component to project the74 holistic representation xi of Ii into a latent space, i.e., the attribute space V . In V , its high-level75 vectors are designed to have certain desirable properties, e.g., corresponding to semantic properties76 of fine-grained objects (a.k.a. “fine-grained attributes”).77 More specifically, in our A2-NET, the unsupervised attribute-guided learning is realized by a recon-78 struction paradigm with an encoder-decoder structure, as shown in Figure ??. In concretely, given79 a batch of n training data Ii, the holistic representations X = {x1; x2; . . . ; xn} 2 Rd⇥n can be80 obtained as aforementioned. By formulation, the encoder projects X into the attribute space V with81 a projection matrix W 2 Rk⇥d to get an internal latent representation V 2 Rk⇥n w.r.t. X . In82 particularly, we set that the dimension of latent representation k equals to the number of hash bits in83 the final binary hash code ui. Furthermore, each column of V , i.e., vi 2 Rk, can derive ui by84
ui = sgn(tanh(vi)) . (5)
Meanwhile, regard ng vi, we lso consider to reco struct i s input xi by a decod r as a counterpart85 of the ncoder. Therefore, on one ha d, such a rec nstruction paradigm can reduce and further distill86 high-level semantic cues in the attribute space V . While, on the other hand, it can drive the training87 of A2-NET by preserving the similarity between queried hash codes and database points in terms of88 learning to hash (cf. Section 3.3.3).89
In concretely, the learni g objective of unsupervised attribute-guided learni g is written as follows:90
min W
kX W>WXk2F s.t. WX = V 0 = tanh(V ) , (6)
where the decoder (i.e., a counterpart of the encod r) is realized by W> to simplify the network.91 How ve , directly minimizing Eq. (6) with a hard constraint is hard to optimize. Therefore, we relax92 the constraint into a soft constraint, nd the learning objective can be rewritten by93
min W
kX W>V 0k2F + kWX V 0k2F . (7)
3.3.2 Attribute-Specific Feature Decorrelation94
By conducting the aforementioned unsupervised attribute-guided learning, we can obtain the internal95 latent vectors V 0 as the attribute-specific features. In order to both enhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features, we introduce a feature97 decorrelation constraint upon V 0, which is formulated by98
min V 0
kV 0V 0> kIk2F , (8)
where I is the identity matrix. Such a feature decorrelation constraint is preferable to construct99
independent features and reduce redundant information. Therefore, based on unsupervised attribute-100
guided learning and attribute-specific feature decorrelation, the final learned hash codes are expected101
to be both attribute-aware and hash-bit independ nt.102
3.3.3 Hash Code Learning103
In the following, we conduct the hash code learning based on the obtained attribute-specific features.104 Assume that we have n query data points which are denoted as {qi}ni=1, as well as m database points105
3
which are denoted as {vj}mj=1. Note that, both qi and vi are belonging to the attribute space V . Then,106 the corresponding binary hash codes can be obtained via107 ui = sgn(tanh(qi)) , (9) zj = sgn(tanh(vj)) , (10) where ui, zj 2 { 1, +1}k. The goal of our hash code learning is to learn binary hash codes for both108 query points and database points from {qi}ni=1, {vj}mj=1, and the pairwise supervised information,109 i.e., S 2 { 1, +1}n⇥m. To preserve the pairwise similarity, we adopt the `2 loss between the110 supervised information (a.k.a. similarity) and the inner product of query-database point binary code111 pairs. It can be formulated as follows:112 minU ,Z Pn i=1 Pm j=1 u>i zj kSij 2 s.t. U 2 { 1, +1}n⇥k, Z 2 { 1, +1}m⇥k . (11) Overall, we get the final objective of the proposed A2-NET model by considering Eq. (7), Eq. (8) and113 Eq. (11) together as follows:114 min W ,⇥ L(I) = kX W>V 0k2F + kWX V 0k2F +↵kV 0V 0> nIk2F + nX i=1 mX j=1 u>i zj kSij 2 , (12) where , ↵ and are hyper-parameters as the trade-off.115 In practice, uri g training, it might be only available a set of database points {vj}mj=1 without query116 points. Thus, we randomly sample n data points from database to construct a query set, and denote117 the indices of all the database points as with the indices of the query set as ⌦. Additionally, because118 we cannot back-propagate the gradient to ⇥ due to the sgn(·) function, we omit the sgn(·) function119 and only apply tanh(·) for relaxation in Eq. (10) of the whole optimization process. Therefore, the120 optimization formulation of A2-NET can be rewritten with only database points {vj}mj=1 for training121 as:122 min W ,⇥ L(I) = kX W>V 0k2F + kWX V 0k2F + ↵kV 0V 0> nIk2F
+ X
i2⌦
X
j2
⇣ tanh(W · F (Ii;⇥))>zj kSij ⌘2 . (13)
For optimization, we employ the back-propagation algorithm and follow [8] to train the whole123 A2-NET m l in an end-to-end manner.124
3.4 Out-of-Sample Extension125
After training A2-NET, the learned model can be applied for generating binary codes for query points126 including unseen query points in th training phase. Specifically, we can use the following equation127 to generate the binary code for Iq:128
uq = sgn(tanh(W · F (Iq;⇥))) . (14)
4 Experiments129
In this section, we present the implementation details, empirical settings, main results, ablation studies130 and also some interesting qualitative a alyses of the learned attribute-aware binary hash codes.131
4.1 Datasets132
x0i By following ExchNet [4], our experiments are conducted on five fine-grained benchmark datasets,133 i.e., CUB-200-2011 [16], Aircraft [12], Food101 [1], NABirds [15] and VegFru [6]. Specifically,134 CUB-200-2011 is one of the most popular used fine-grained datasets. It contains 11,788 bird images135 from 200 bird species and is officially split into 5,994 images for training with 5,794 images for136 test. Aircraft contains 10,000 images spanning 100 aircraft models with 3,334 for training, 3,333137
4
Fine-grained representation learning
U ns
up er
vi se
d at
tr ib
ut e-
gu id
ed lo
ss (S
ec tio
n 3.
3. 1)
Feature ecorrelation loss (S ction 3.3.2)
3.3 Attribute-Aware Hash Codes Generating64
How to generate attribute-aware hash codes is the key of our A2-NET model. We laborate it in65 the foll wing four aspects, i.e., unsupervised attribute-guided learning, attribute-specifi feature66 decorrelation, hash code learning and the optimization learning algorithm.67
3.3.1 Unsupervised Attribute-Guided Learning68
In real-applications, especially for the large-scale and fine-grained tasks, attribute annotations are69 always infeasibl , which limits the learning process to be conducted in an unsupervised setti g.70 While, in the literature, the main goal of unsupervised learning is to capture regularities in data for the71 purp se of xtracting useful representations or for restoring corrupted data [13]. Many unsupervised72 methods explicitly produce internal latent units or codes, from which the data is to be reconstructed.73
Inspired by this, we develop an unsupervised attribute-guided learning component to project the74 holistic r presentation xi of Ii into a latent space, i.e., the attribute space V . In V , its high-level75 vectors are designed to have certain desirable properties, e.g., corresponding to semantic properties76 of fine-grained objects (a.k.a. “fine-grained attributes”).77
More specifically, in our A2-NET, the unsupervised attribute-guided learning is realized by a recon-78 struction paradigm with an encoder-decoder structure, as shown in Figure ??. In concretely, given79 a batch of n training data Ii, the holistic representations X = {x1; x2; . . . ; xn} 2 Rd⇥n can be80 obtained as aforementioned. By formulation, the encoder projects X into the attribute space V with81 a projection matrix W 2 Rk⇥d to get an internal latent representation V 2 Rk⇥n w.r.t. X . In82 particularl , we set tha the dimension of lat nt representation k quals to the number of hash bits in83 the final binary hash code ui. Furthermore, each column of V , i.e., vi 2 Rk, can derive ui by84
ui = sgn(tanh(vi)) . (5)
Meanwhile, regarding vi, we als consider to reconstruct its input xi by decoder as a counterpart85 of the encoder. Therefore, o one hand, such a reconstruction paradigm can reduce and further distill86 high-level semantic cues in the attribute space V . While, on the other hand, it can drive the training87 of A2-NET by preserving the similarity between queried hash codes and database points in terms of88 learning to hash (cf. Section 3.3.3).89
In concretely, the learning objective of unsupervised attribute-guided learning is written as follows:90
min W
kX W>WXk2F s. . WX = V 0 = tanh(V ) , (6)
where the decoder (i.e., a counterpart of the encoder) is realized by W> to simplify the network.91 Howev r, directly minimizing Eq. (6) with a hard constraint is hard to optimize. Therefore, we relax92 the constraint into a soft constraint, and the learning objective can be rewritten by93
min W
kX W>V 0k2F + kWX V 0k2F . (7)
3.3.2 Attribute-Specific Feature Decorrelation94
By conducting the aforementioned u supervised attribute-guided learning, we can obtain the internal95
latent vectors V 0 as the attribute-specific features. In order to both enhance the discriminative ability96
and remove the redundant correlation among these attribute-specific features, we introduce a feature97
d correlation constraint upon V 0, which is formulated by98
min V 0
kV 0V 0> nIk2F , (8)
where I is the i entity matrix. Such a feature decorrelation constraint is preferable to construct99
independent features and reduce redundant information. Therefore, b sed on unsupervise attribute-100
guided learning and attribute-specific feature decorrelation, the final learned hash codes are expected101 to be both attri ute-aware an hash-bit indepe dent.102
3.3.3 Hash Code Learning103
In the following, we conduct the hash code lear ing based on the obtained attribute-specific features.104 Assume that we have n query data points which are denoted as {qi}ni=1, s well as m database points105
3
Hash code learning loss (Section 3.3.3)
Attribute-aware hash codes generating“Red head” “On the tree” “Spotted body”
2 Related Work23
2.1 Fine-Grained Image Retrieval24
2.2 Learning to Hash25
2.3 Visual Attributes26
3 Methodology27
In this section, we introduce the overall framework and notations of the proposed A2-NET model, as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29
3.1 Overall Framework and Notations30
As illustrated in Figure ??, our A2-NET model consists of two crucial modules, i.e., a fine-grain d31 representation learning module and an attribute-aware hash codes generating module. Given an32 input image Ii, based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN, a set of attention guidance A = {Ac} is learned for capturing fine-grained tailored34 local patterns T ci from Ti. To distill semantical cues and further generate the final attribute-aware35 binary hash codes, we propose to transform these appearance-specific features T towards semantic-36 specific representations T̂ by performing a transform network (·). After aggregating T̂ , the obtained37 attentive local-level features xci are associated with the global-level feature x global i to f rm as a38 holistic feature representation xi. In order to generate attribute-aware binary hash codes, we conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond o40 an attribute vector w.r.t. one kind of namable properties of fine-grained objects. Finally, hash code41 learning is performed upon vi to obtain the final attribute-a are binary codes ui.42
3.2 Fine-Grained Representation Learning43
Attention plays an important role in human perception [3, 7], and humans exploit a sequence of44
partial glimpses and s lective y focus on salient parts of an object or a s ene in order to bet er capture45
visual structure [10]. Inspired by this, we incorporat the attention mechanism into representation46
learning to capture fine-grained local patterns for distinguishing subtle differ nces between these47
subordinate categories.48
In concretely, we extract the deep feature of its input image Ii via a backbone CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Then, based on Ti, C attention guidance Ac 2 RH⇥W is generated as a set of attention maps, i.e., A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obtain the final attribu e-aware binary co es, it is54 desirable to transform these appearance-specific (i.e., low-level) f atures T to semantic-specific (i.e.,55 mid-level) representations which are closer to the attribute space. Thus, a transforming network (·),56 which is equipped with a stack of convolution layers, is performed on T as follows:57
T̂ ci = (T c i ; ✓local) , (3)
T̂i = (Ti; ✓global) , (4)
where ✓ presents the para eters of the corresponding transfor ing networks w.r.t. T ci and Ti, respec-58 tively. Then, we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . The holistic60 feature representation w.r.t. the input image Ii is achieved by concatenating both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F (Ii;⇥) 2 R . Note that, we hereby abstract the aforementioned fine-grai ed62 feature learning process as a function F (Ii;⇥) with its parameters ⇥.63
2
Attention guidance
2 Related Work23
2.1 Fine-Grained Image Retrieval24
2.2 Learning to Hash25
2.3 Visual Attributes26
3 Methodology27
In this section, we introduce the overall framework and notations of the proposed A2-NET model, as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29
3.1 Overall Framework and Notations30
As illustrated in Figure ??, our A2-NET model consists of two crucial modules, i.e., a fine-graine31 representation learning module and an attribute-aware hash codes generating module. Given an32 input image Ii, based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN, a set of attention guidance A = {Ac} is le rned for capturing fi -grained tailored34 local patterns T ci from Ti. To distill semantical cues and further generate the final attribute-aware35 binary hash c des, we propose to transform these appearance-specific features T towa ds semantic-36 specific representations T̂ by performing a transform network (·). After aggregating T̂ , the obt ined37 attentive local-level features xci are associated with the global-level feature x global i to form as a38 holistic feature representation xi. In order to generate attribute-aware binary hash codes, we conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond to40 an attribute vector w.r.t. one kind of namable properties of fine-grained objects. Finally, hash code41 learning is performed upon vi o obtain the final attribute-aware binary codes ui.42
3.2 Fine-Grained Representation Learning43
Attention plays a imp rtant role in human perception [3, 7], and humans exploit a sequence of44 artial glimpses d selectively focus on salient parts of an object or a scene in or er to better capture45 visual structure [10]. Inspired by this, we incorporate the attention mechanism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48
In concretely, we extract the deep feature of its input image Ii via a backbone CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Then, based on Ti, C attention guidance Ac 2 RH⇥W is g nerated as a set of attention ma s, i.e., A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obtain the final attribute-aware binary codes, it is54 desirable to transform these appearance-specific (i.e., low-level) features T to semantic-specific (i.e.,55 mid-level) representations which are closer to the attribute space. Thus, a transforming network (·),56 which is equipped with a stack of convolution layers, is performed on T as follows:57
T̂ ci = (T c i ; ✓local) , (3)
T̂i = (Ti; ✓global) , (4)
where ✓ presents the parameters of the corresponding transforming networks w.r.t. T ci and Ti, respec-58 tively. Then, we aggregate T̂ ci and T̂i by conducting global average-pooling d correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . The holistic60 feature representation w.r.t. the input image Ii is achieved by concatenati g both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F (Ii;⇥) 2 Rd. Note that, we hereby abstract the aforementioned fine-grained62 feature learning process as a function F (Ii;⇥) with its parameters ⇥.63
2
2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27
In this section, we introduce the overall framework and notations of the proposed A2-NET model, as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29
3.1 Overall Framework and Notations30
As illustrated in Figure ??, our A2-NET model consists of two crucial modules, i.e., a fine-grained31 representation learning module and an attribute-aware hash codes generating module. Given an32 input image Ii, based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN, a set of attention guidance A = {Ac} is learned for capturing fine-grained tailored34 local patterns T ci from Ti. To distill semantical cues and further generate the final attribute-aware35 binary hash codes, we propose to transform these appearance-specific features T towards semantic-36 specific representations T̂ by performing a transform network (·). After aggregating T̂ , the obtained37 attentive local-level features xci are associated with the global-level feature x global i to form as a38 holistic feature representation xi. In order to generate attribute-aware binary hash codes, we conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond to40 an attribute vector w.r.t. one kind of namable properties of fine-grained objects. Finally, hash code41 learning is performed upon vi to obtain the final attribute-aware binary codes ui.42
3.2 Fine-Grained Representation Learning43
Attention plays an important role in human perception [3, 7], and humans exploit a sequence of44 partial glimpses and selectively focus on salient parts of an object or a scene in order to better capture45 visual structure [10]. Inspired by this, we incorporate the attention mechanism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48
In concretely, we extract the deep feature of its input image Ii via a backbone CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Then, based on Ti, C attention guidance Ac 2 RH⇥W is generated as a set of attention maps, i.e., A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obtain the final attribute-aware binary codes, it is54 desirable to transform these appearance-specific (i.e., low-level) f atures T to semantic-specific (i.e.,55 mid-level) representations which are closer to the attrib te space. Thus, a transforming n twork (·),56 which is equipped with a stack of convolution layers, is performed on T as follows:57
T̂ ci = (T c i ; ✓local) , (3)
T̂i = (Ti; ✓global) , (4)
where ✓ presents the parameters of the corresponding transfor ing networks w.r.t. T ci and Ti, respec-58 tively. Then, we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x glo al i . Th holistic60 feature representation w.r.t. the input image Ii is achieved by concatenating both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F (Ii;⇥) 2 Rd. Note that, we hereby abstract the aforementioned fine-grained62
feature learning process as a function F (Ii;⇥) with its parameters ⇥.63
2
2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section, we introduce the overall framework and notations of the proposed A2-NET model, as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29 3.1 Overall Framework and Notations30
As illustrated in Figure ??, our A2-NET model consists of two crucial modules, i.e., a fine-grai d31 representation learning module and an attribute-aware hash codes g nerating modul . Given an32 input image Ii, based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN, a set of attention guidance A = {Ac} is learned for capturing fine-grained tailored34 local patterns T ci from Ti. To distill semantical cues and further generate the final attribute-aware35 bi ary ash c des, we propose to transform these appearance-specific features T towards semantic-36 specific representations T̂ by perform ng a transform network (·). After aggregating T̂ , th obtained37 attentive local-level features xci are associated wi h the global-level f ature x global i to form as a38 holistic feature representation xi. In order to generate attribute-aware binary hash codes, we conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond to40 an attribute vector w.r.t. one kind of namable properties of fine-grained objects. Finally, hash code41 learning is performed upon vi to obtain the final attribute-aware binary codes ui.42
3.2 Fine-Grained Representation Learning43
Attention plays an important role in human perception [3, 7], and humans exploit a sequence of44 partial glimpses and selectively focus on salient parts of an object or a scene in order to b ter capture45 visu l structure [10]. I spired by this, we incorporate th attention mech nism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48
In concretely, we extra t the deep feature of its input image Ii via a backbon CNN model CNN(·)49 s50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Then, based on Ti, C attention guidance Ac 2 RH⇥W is generated as a set of attention maps, i.e., A.51 The attention guidance Ac is designed to ev luate which deep descript rs should be attended or ven52 ov rlooked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obtain the final attribute- ware binary codes, it is54 desirable to transform these appearance-specific (i.e., low-level) features T to semantic-specific (i.e.,55 mid-level) representations which are closer to the attribute space. Thus, a transforming network (·),56 which is equipped with a stack of convolution layers, is performed on T as follows:57
T̂ ci = (T c i ; ✓local) , (3)
T̂i = (Ti; ✓global) , (4)
where ✓ pr sents the parameters of the corresponding tr nsforming networks w.r.t. T ci nd Ti, respec-58 tively. Then, we aggregate T̂ ci and T̂i y conducting global average-pooling and cor espo dingly59 obtain the atte tive local-level f atures xci , as well as the global-level fe tur x global i . The holisti60 fe ture representation w.r.t. the input image Ii is achieved b concatenating both xci and xgl bali , i.e.,61 xi = h xci ; x global i i = F (Ii;⇥) 2 Rd. Note hat, we hereby abstract the aforementioned fin -grained62 featur learning process as a function F (Ii;⇥) with its parameters ⇥.63
2
…
2 Related Work23
2.1 Fine-Grained Imag Retrieval24
2.2 Learning to Hash25
2.3 Visual Attributes26
3 Methodology27
In this secti n, we int oduce the overall framew rk and otations of the p oposed A2-NET model, as28 well as elaborating the detailed modules of A2-NET a its corresp nding ptimiza ion algorithm.29
3.1 Overall Framework and No ations30
As illustrated in Figure ??, our A2-NET model consists of two crucial modules, i.e., a fine-grained31 representation learning module and an attribute-aware hash codes generating module. Given an32 input image Ii, based n its corr po ding deep activa io ten o Ti 2 RC⇥H⇥W extracted by a33 backbone CNN, a set of attention guida ce A = {Ac} i learned for capt ring fine-grained t ilored34 local patterns T ci from Ti. To distill semantical cues and further generate the final attribute-aware35 bi ary has c des, we pro ose to transfo m these ppearan e-specific f atures T owards semantic-36 sp cific representations T̂ by performing a transform etwork (·). Aft r aggre ating T̂ , h obtained37 attentive local-level features xi ar associated with t e gl bal-level feature x global i to form as a38 holistic feature representation xi. In order to ge erate attribute-aware binary hash codes, we conduct39 a reconstructing paradigm to projec xi a vi in an attribute space wh re ts data point correspond to40 an attribute vector w.r.t. one kind of namable properties of fine-grained obj cts. Finally, hash code41 learning is performed upon vi to obtain the final attribute-aware binary codes ui.42
3.2 Fine-Grained Representation Learning43
Att ntion plays an important role in hu an perception [3, 7], nd humans exploit a sequenc of44 partial glimpses and selectively focus on salient parts of an object or a scene in order to better capture45 visu l structure [10]. Inspired by this, we incorp rate the attentio mechanism into represent tion46 lear ing to capture fine-grain d local patt rns for distinguishing subtle differences between these47 subordinate categories.48
In concretely, we extract the deep feature of its input image Ii via a backbone CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Then, based on Ti, C attention guida c Ac 2 RH⇥W is gen rat d as set of attention maps, i.e., A.51 The ttent on guidance Ac s designed to ev luate which deep descriptors should be attended or even52 overl ked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obt in the final attribute-aware binary codes, it is54 desirable to transform these appea a ce-specific (i.e., low-level) features T to semantic-specific (i.e.,55 mid-l vel) repres ntations which are closer to the attribute space. Thus, a transforming network (·),56 which is equipped with a stack of convolution layers, is performed on T as follows:57
T̂ ci = (T c i ; ✓local) , (3)
T̂i = (Ti; ✓global) , (4)
where ✓ prese ts the parameters of the corresponding transforming networks w.r.t. T ci and Ti, respec-58 tively. Then, we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentiv local-level f atures xci , as well as the global-level feature x global i . Th holistic60 feature representation w.r.t. the input image Ii is achieved by concatenating both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F (Ii;⇥) 2 Rd. Note that, we hereby abstract the aforementioned fine-grained62 feature learning process as a function F (Ii;⇥) with its parameters ⇥.63
2
2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section, we introduce the overall framework and notations of the proposed A2-NET model, as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29 3.1 Overall Framework and Notations30 As illustrated in Figure ??, our A2-NET model consists of two crucial modules, i.e., a fine-grained31 representation learning module and an attribute-aware hash codes generating module. Given an32 input image Ii, based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN, a set of attention guidance A = {Ac} is learned for capturing fine-grained tailored34 local patterns T ci from Ti. To distill semanti al cu s d further generate the final attribute-aware35 binary hash codes, we propose to transform these appearance-specific features T towards semantic-36 specific representations T̂ by performing a transform network (·). After aggregating T̂ , the obtained37 attentive local-level features xci are associated with the global-level feature x global i to form as a38
holistic feature representation xi. In order to generate attribute-aware binary hash codes, we conduct39 a r constructing paradigm to project xi as vi in an attribute sp ce where it data point correspond to40 an attribute vector w.r.t. one kind of namable properties of fine-grained objects. Finally, hash code41 lear ing is perfor ed upon vi to obtain the final attribute-aware binary codes ui.42
3.2 Fine-Grained Representation Learning43
Attention plays an important role in human percept on [3, 7], and humans exploit a sequence of44 partial glimpses and selectively focus on salient parts of an object or a scene in order to better capture45 visual structure [10]. Inspired by this, we inc rpor te t e attention mechanism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48
In concretely, we extract the deep feature f its i put im ge Ii via a ba kbone CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Then, based on Ti, C attention guidance Ac 2 RH⇥W is generated as a set of ttentio maps, i.e., A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obtain the final attribut -aware binary codes, it is54 desirable to tran for these appearance-specific (i.e., low-level) features T to semantic-spec fic (i.e.,55 mid-level) representations which are closer to the attribute space. Thus, a transforming network (·),56 which is equipped with a stack of convolution layers, is performed on T as follows:57
T̂ ci = (T c i ; ✓local) , (3)
T̂i = (Ti; ✓global) , (4)
where ✓ pr sents the parameters of he c rrespo ding transforming networks w.r.t. T ci a d Ti, respec-58 tively. Then, we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . Th holistic60 feature rep esentation w.r.t. the input image Ii i achieved by concat at g both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F (Ii;⇥) 2 Rd. Note that, we hereby abstract the aforem ntioned fine-grained62 f ature le ning pro ess as a function F (Ii;⇥) with its parameters ⇥.63
2
Aggregation …
2 Related Work23
2.1 Fine-Grained Image Retri val24
2.2 Le rning to Hash25
2.3 Visual Attributes26
3 Method logy27
In this section, we introduce the ove ll framework nd notati ns of the propos A2-NET model, as28 well as elaborating the detailed modules of A2-NET and its correspon ing optimization algorithm.29
3.1 Over l Fra ework and Notations30
As illustrated in Figur ??, our A2-NET model consist of two crucial modules, i.e., a fin -gr ined31 representation learning module and a attribute-aware hash c des ge erating mo ul . Given a32 input image Ii, based on its corres onding de p activatio tens r Ti 2 RC⇥H⇥W extracted by a33 back o e CNN, a set of ttention guidance A = {Ac} is learn d f capturing fine-grained tailored34 local patterns T ci from Ti. To distill semantical cues and further gen rate the fi al attribute-aware35 binary hash cod s, we propose to tr nsform these ppearance-specific features T towards semantic-36 specific repres ntations T̂ by performing a transform network (·). After aggregating T̂ , the obtained37 attentive local-level features xci are associated with the global-level featur x global i to form as a38 holistic e ture r pr s ntation xi. In order to g nerate attribute-aware binary hash cod s, we conduct39 a econstructing p adigm to project xi as vi i an attribute space wh re it dat point correspond to40 an ttribute tor w.r.t. o e kind f n m ble properties of fin -grained objects. Finally, hash code41 le rning is p rformed pon vi to obtain th final attribute- ware bi ary codes ui.42
3.2 Fi e-Grained Representation Le rning43
Attention plays an important role in human perception [3, 7], and humans exploit a sequence of44
partial glimpses and selectively focus on sali nt parts of an object or a scene in or e to b tt r capture45
visual structur [10]. Inspired by this, we incorporate the attention mechanism into representati n46
learning to capture fin -grained local patterns for distinguishing subtle differences between these47
s bordi te tegories.48
In concretely, we extract the deep feature of its input image Ii via a backbone CNN model CNN(·)49
as50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Th n, based o i, C attention uidan e Ac 2 RH⇥W is g nerated as a set f att ntion m ps, i.e., A.51 The atte ion i ance Ac is designed to evaluate which deep descriptors should b attended or even52 overlooked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obtain the final attribute-aware binary codes, it is54 desirable to tran form these appearance-specific (i.e., low-level) features T to semantic-specific (i.e.,55 mid-level) representations whi h are closer to the attribute space. Thu , a transforming network (·),56 which is equipped with a stack of convolution layers, is performed on T as follows:57
T̂ ci = (T c i ; ✓local) , (3)
T̂i = (Ti; ✓global) , (4)
wher ✓ presents the parameters of the corresponding transforming networks w.r.t. T ci and Ti, respec-58 tively. Then, we ggrega e T̂ ci and T̂i by c n ucting global average-pooling and correspondingly59 o tain the attentive local-level featur s xci , as well a the global-l vel feature x global i . The holistic60 feature representation w.r.t. the i put mage Ii is chi ved by concatenating both xci and xglobali , i.e.,61 xi = h xci ;
l al = F (Ii;⇥) 2 Rd. Note that, we hereby abstract the aforementioned fine-grained62 feature learnin proc ss as a function F (Ii;⇥) with its parameters ⇥.63
2
2 Rel t d Work23
2.1 Fi -Grained Image Retrieval24
2.2 Lear i g to Ha h25
2.3 Visu l Attributes26
3 Methodology27
In this section, we introduce the overall framework and notations of the proposed A2-NET model, as28 w ll as elaborating the detailed modules of A2-NET and its correspon ing optimization algorithm.29
3.1 Overall Fram work and Notation30
As illustrated in Figur ??, our A2-NET model consists of two cruc al modules, i.e., a fi e-grained31 r p tatio l arning modul nd an attribut - ware has codes generating module. Given an32 inp t image Ii, b sed o its respond ng deep a tivation tensor Ti 2 RC⇥H⇥W extracted by a33 ba kbone CNN, a set of attention g idance A = {Ac} is learned for capturing fine-grained tailored34 l l patterns T ci from Ti. To distill semantical cues and further en rate the final attribute-aware35 binary hash odes, we prop se to transform these ppearance-specific features T towards semantic-36 specific representations T̂ by performing a transform netwo k (·). After aggregating T̂ , he obt ined37 attentive local-level features xci re associated with the global-level feature x global i to form as a38 holistic feature representation xi. In order to generate attribute-aware binary hash codes, w conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond to40 an attribute vector w.r.t. one kind of namable properties of fine-grained objects. Finally, has c de41
learning is performed upon vi to obtain the fi al attribute-aware binary codes ui.42
3.2 Fine-Grained Representation Learning43
Attention plays a important role in human perception [3, 7], and humans exploit a sequence of44
partial glimpses and select vely focus on salient parts of an bject or a scene in order to b ter capture45 visual structure [10]. Inspired by this, we incorporate the attention mechanism into representation46 learning to c pture fine-grained local patterns for distinguishing subtle differences b tween th se47 subordinate c tegories.48
In concretely, we extract the deep feature of its input image Ii via a backbone CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Then, based on Ti, C attention guidance Ac 2 RH⇥W is generat d as a set of attenti n maps, i.e., A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be tte d d or even52 overlooked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obtain the final attribute-aware binary codes, it is54 desirable to transform these appearance-specific (i.e., low-level) features T to semantic-specific (i.e.,55 mid-level) representations which are closer to the attribute space. T us, transf rming netw rk (·),56 which is equipped with a stack of convolution layers, is performed on T as follows:57
T̂ ci = (T c i ; ✓local) , (3)
T̂i = (Ti; ✓global) , (4)
where ✓ presents the parameters of the corresponding transformi g networks w.r.t. T ci and Ti, r spec-58 tively. Then, we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . The holistic60 feature representation w.r.t. the input image Ii is achieved by concatenating both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F (Ii;⇥) 2 Rd. Note that, we her by abstr c the af rementioned fine-gr ined62 feature learning process as a function F (Ii;⇥) with its parameters ⇥.63
2
2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section, we introduce the overall framework and notations of the proposed A2-NET model, as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorith .29 3.1 Overall Framework and Notations30 As illustrated in Figure ??, our A2-NET model cons sts of two crucial modules, i.e., a fine-grain d31 representation learning module and an attribute-aware hash codes gene ating module. Given an32 input image Ii, based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN, a set of attention guidance A = {Ac} is learned for capturing fin -grained tail red34 local patterns T ci from Ti. To distill semantical cues and further generat the final at ribute-aware35 binary hash codes, we propose to transfor th se appearance-specific features T towards semantic-36 specific representations T̂ by performing a transform network (·). Aft r aggregating T̂ , the obtained37 attentive local-level features xci are asso iated with the global-level feature x global i o form as a38 holistic featur representation xi. In ord r to generate attribute-awar binary hash codes, we conduct39 a reconstructing paradigm to project xi as vi in an attribute space wh re its data point corr spond to40 an attribute vector w.r.t. one kind of namable properties of fine-grained objects. Finally, hash code41 learning is performed upon vi to obtain the final attribute-aware binary codes ui.42
3.2 Fine-Grain d Representation Learning43
Attention plays an important role in human perception [3, 7], and humans exploit a sequence of44 partial glimps s and s lectively focus on s lient parts of an object or a scene in order to better capture45 visual structure [10]. Inspired by this, we incorporate the attention mechanism into repr sentation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48
In concretely, we extract the deep feature of its nput image Ii via a backbone CNN model CNN(·)49 as50 Ti = CNN(Ii) 2 RC⇥H⇥W . (1) Then, based on Ti, C attention guidance Ac 2 RH⇥W is generated as a set of attention maps, i.e., A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53
T ci = A c Ti , (2)
where is the element-wise Hadama d product. T obtain he final a tribute-aware binary cod s, it is54 desirable to transform these appearance-specific (i.e., low-level) features T to semantic-specific (i.e.,55 mid-level) representations which are closer to the attribute space. Thus, a transforming netwo k (·),56 which is equipped with a stack of convolution layers, is performed on T as follows:57
T̂ ci = ( i ; local) , (3)
T̂i = (Ti; ✓global) , (4)
where ✓ presents the parameters of the corresponding transforming networks w.r.t. T ci and Ti, respec-58 tively. Then, we aggregate T̂ ci and T̂i by onducting glob l average-poolin and correspondingly59 obtain the attentive local-level features xci , as w ll as the gl bal-level feature x global i . Th holistic60 feature representation w.r.t. the input image Ii is achiev d by concat nating both xci a d xglobali , i.e.,61
xi = h xci ; x global i i
= F (Ii;⇥) 2 Rd. Note that, we hereby abstract the aforementioned fine-grained62
feature learning process as a function F (Ii;⇥) with its parameters ⇥.63
2
Aggregation
where t e d coder (i.e., a counterpart of the encoder) is realized by W> to simplify the netwo k.91 However, directly minimizing Eq. (6) with a hard constraint is hard to optimize. Therefore, we relax92 the constraint into a soft constraint, and the learning objective can be rewritten by93
min W
kX W>V 0k2F + kWX V 0k2F . (7)
3.3.2 Attribute-Specifi F ur Decorrelation94
By conducting the aforementioned unsupervised attribut -guided learning, we can obtain the internal95 l tent vectors V 0 as the attribute-specific features. I order to both enhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features, we introduce a feature97 decorrelation c n r int upon V 0, which is formulat d by98
mi V 0 kV 0V 0> nIk2F , (8) where I is the identity matrix. Such a feature ecorrelation co straint is pref rable to construct99 independent features and reduce redundant information. Therefore, based on unsupervised attribute-100 guided learning and attribute-specific feature decorrelation, the final learned hash codes are expected101 to be bo h attribute-aware and hash-bit ind pend nt.102
3.3.3 H sh Co e L arning103
In the f llowing, we conduct the hash code learning based on the obtained attribute-specific features.104 Assume that w have n qu ry data p ints which are denoted as {qi}i=1, as w ll a m database points105 which a noted as {vj}j=1. Note hat, both qi and vi are belo gin to the attribute space V . Then,106 the corresponding binary has codes can be obt ined via107
ui = sgn(tanh(qi)) , (9) zj = sgn(tanh(vj)) , (10)
where ui, zj 2 { 1, +1}k. The goal of our hash code learning is to learn binary hash codes for both108 qu ry poin s and database points from {qi}ni=1, {vj}mj=1, and the pairwise supervised information,109 i.e., S 2 { 1, +1}n m. To pr se ve the pairwise similarity, adopt the `2 loss between the110 supervised inf rmati ( .k.a. similarity) d t inner product f qu ry-database point binary code111 pairs. It can be formulated as follows:112
minU ,Z Pn
i=1 Pm j=1 u>i zj kSij 2
s.t. U 2 { 1, +1}n⇥k, Z 2 { 1, +1}m⇥k . (11)
Overall, e g t the fina obj iv of the propo ed A2-NET model by considering Eq. (7), Eq. (8) and113 Eq. (11) toge a follows:114
min
W ,⇥
L(I) = kX W>V 0k2F + kWX V 0k2F +↵kV 0V 0> nIk2F + nX
i=1
mX
j=1
>
i zj kSij
2 ,
(12) where , ↵ and are hyper-parameters as the trade-off.115
In practice, during training, it might be only available a set of database poin s {vj}mj=1 without query116 points. Thus, we randomly sample n data points from database to construct a query set, and denote117 the indices of all the database points as with the indices of the query set as ⌦. Additionally, because118 we cannot back-propagate the gradient to ⇥ due to the sgn(·) function, we omit the sgn(·) function119 and o ly apply t nh(·) for elaxation in Eq. (10) of the whole optimization process. Therefore, the120 optimization formulation of A2-NET can be rewritten with only database points {vj}mj=1 for training121 as:122
min W ,⇥
L(I) = kX W>V 0k2F + kWX V 0k2F + ↵kV 0V 0> nIk2F
+ X
i2⌦
X
j2
⇣ tanh(W · F (Ii;⇥))>zj kSij ⌘2 . (13)
For optimization, our A2-NET does not require co plicated two-stage learning algorithms, e.g., the123 alternative optimization strategy. We employ the back-propagation algorithm and follow [8] to train124 the whol A2-NET model in a u ified end-to-end manner.125
4
“Blue tail”
Figure 2: Overall framewo k of the proposed A2-NET m el, which consists of two crucial modules, i.e., fine-grained representation learning and attribute-aware hash codes generating. The whole netw rk c n be end-to-e d trainabl , d is generally riven by the nsupervi ed attribute-guided reconstruction loss, the featur decorrelation loss an the sh o e l arning loss, cf. Section 3.3.
objects of fine-grained images have only subtle differences, and ften larg ly vary in pose, c l , and orientation or can exhibit cr ss-mo al differences (e.g., sketch-based retrieval [25]).
Depe ding on he type of query image, the ost studied areas of fin -grai ed image retrieval can be separat into two groups: fine-grai e content-b s d imag retrieval (FG-CBIR) and finegrained sketch-based image retrieval (FG-SBIR). More specifically, in G-CBIR, unsupervised learning based [40] a d supervise learning based methods [44, 47, 48] were developed from different perspe tives for handling fine-grained retrieval tasks e.g., localizing fine-grained parts [40], enhan ing intra-class separability with inter-class compactn ss [48], an re ci g the co fid nce of the fine-grained predictions [44], tc. While, FG-SBIR eeds to n t only captur fine- rained characteristic pr sent in th sket h s, but also poss ss he ability t traverse the sketch and image domain gap. In the literature of FG-SBIR, the earlier works, e.g., [23, 43, 46], were mostly based on Siamese-triplet networ s [3] to tackle the aforemention d challenges. Recently, s me works tried to incorporate the advances of rec nt progress in self-supervised learning [6] a d attention mechanisms [7] for further improving the retrieval accuracy of FG-SBIR, e.g., [29, 2].
However, although these fine-grained retrieval methods achieved good results, they still have the limitations in the face of large-scale data, i.e., the searching time for exact nearest neighbor is typically expensive or even impossible for the given queries. To alleviate this issue, fine-grained hashing, which aims to generat comp ct binary codes o represent fine-grained images, as a promising direction has attracted the attention in the fine-grained community very recently [8, 19]. More specifically, ExchNet [8] was the first to define the fine-grained hashing task and develop a fine-grained tailored method to firstly locate discriminative object parts and further learn binary hash codes for representing fine-grained images. In the same period, DSaH [19] was proposed to automatically mine salient regions and learn semantic-preserving hash codes simultaneously. Unfortunately, the learned hash bits of these methods lack any semantics which are more meaningful to fine-grained objects, and thus lack the model interpretability. Compared with them, our proposed A2-NET can not only outperform the previous fine-grained hashing methods, but more importantly, the learned hash codes of A2-NET are attribute-aware, i.e., the hash bits of A2-NET have strong correspondence to semantic visual properties that are useful for fine-grained image retrieval.
2.2 Learning to Hash
Hashing [38] is a widely-studied solution to approximate nearest neighbor search, which transforms the data item to a short code consisting of a sequence of bits (i.e., hash codes). The research efforts of hashing can be categorized into two groups, including data-independent hashing (aka locality sensitive hashing [9, 27, 34]) and data-dependent hashing (aka learning to hash [4, 12, 17, 33]).
Specifically, locality sensitive hashing methods attempted to adjust hash learning from different perspectives, e.g., the theory or machine learning views, to name a few: proposing random hash functions satisfying local sensitive property [9], developing better search schemes [27], providing faster computation of hash functions [34], etc. While, compared with locality sensitive hashing methods, since data-dependent hashing methods learn hash functions from a specific dataset to achieve similarity preserving, they can generally obtain superior retrieval accuracy. Especially for capitalizing on advances in deep learning, many well-performing methods were proposed to integrate feature learning and hash code learning into an end-to-end framework based on deep networks, e.g., [4, 12, 17]. In particular, very recently researchers in the vision community have begun to pay attention to the more challenging and practical hashing task, i.e., fine-grained hashing [8, 19]. To the best of our knowledge, this is the first work to equip these learned hash codes with strong correspondence to visual attributes for dealing with large-scale fine-grained image retrieval.
2.3 Visual Attributes
Attributes are typically mid-level semantic properties of objects [10], such as colors (e.g., “red”, “blue”), texture (e.g., “striped”, “spotted”), or even life habits of animals (e.g., “living on the tree”, “living in the water”). Visual attributes have exhibited their impact for strengthening various vision tasks, including facial verification [21], fine-grained categorization [45], zero-shot transfer [42], scene understanding [30], and so on. Most of the previous attribute learning methods are supervised by costly human-generated annotations and also are dependent on pre-defined attribute labels, e.g., [15, 21, 42, 45]. In consequence, for large-scale problems, these supervised methods might be not feasible due to the restriction caused by the cumbersomely obtained attribute annotations. Moreover, even for some tasks, their visual attributes are quite hard to define. In this paper, to alleviate the aforementioned issues, we propose an A2-NET model to formulate an unsupervised learning structure to project the learned visual features into an attribute space where it finally generates attribute-aware binary hash codes. Compared with previous attribute learning methods, our A2-NET is independent with pre-defined attribute labels, and could automatically learn discriminative attribute-aware hash codes in a unified end-to-end trainable fashion. Furthermore, our method can not only correspond hash bits to visual attributes tailored for fine-grained objects, which shows significant improvements of retrieval accuracy, but also offer an intuitive way of deep hashing interpretation.
3 Methodology
In this section, we introduce the overall framework and notations of the proposed A2-NET model, as well as elaborating the key modules of A2-NET and its corresponding optimization algorithm.
3.1 Overall Framework and Notations
As illustrated in Figure 2, our A2-NET model consists of two crucial modules, i.e., a fine-grained representation learning module and an attribute-aware hash codes generating module. Given an input image Ii, based on its corresponding deep activation tensor Ti ∈ RC×H×W extracted by a backbone CNN, a set of attention guidance A = {Ac} is learned for capturing fine-grained tailored local patterns T ci from Ti. To distill semantical cues and further generate the final attribute-aware binary hash codes, we propose to transform these appearance-specific features T towards semanticspecific representations T̂ by performing a transform network φ(·). After aggregating T̂ , the obtained attentive local-level features xci are associated with the global-level feature x global i to form as a holistic feature representation xi. In order to generate attribute-aware binary hash codes, we conduct a reconstructing paradigm to project xi as vi in an attribute space where its data point corresponds to an attribute vector w.r.t. a certain kind of nameable properties of fine-grained objects (e.g., “red head” or “spotted body”). Furthermore, with the aid of feature decorrelation, vi is expected to be more discriminative by removing redundant correlation information. Finally, hash code learning is performed upon vi to obtain the final attribute-aware binary codes ui.
3.2 Fine-Grained Representation Learning
Attention plays an important role in human perception [7, 16], and humans exploit a sequence of partial glimpses and selectively focus on salient parts of an object or a scene in order to better capture
visual structure [22]. Inspired by this, we incorporate the attention mechanism into representation learning to capture fine-grained local patterns for distinguishing subtle differences between these subordinate categories.
In concretely, we extract the deep feature of its input image Ii via a backbone CNN model ΦCNN(·) by
Ti = ΦCNN(Ii) ∈ RC×H×W . (1) Then, based on Ti, C attention guidance Ac ∈ RH×W is generated as a set of attention maps, i.e., A. The attention guidance Ac is designed to evaluate which deep descriptors [40] in these H ×W cells should be attended or even overlooked by conducting
T ci = A c Ti , (2)
where is the element-wise Hadamard product. To obtain the final attribute-aware binary codes, it is desirable to transform these appearance-specific (i.e., low-level) features T to semantic-specific (i.e., mid-level) representations which are closer to the attribute space. Thus, a transforming network φ(·), which is equipped with a stack of convolution layers, is performed on T as follows:
T̂ ci = φ(T c i ; θlocal) , (3)
T̂i = φ(Ti; θglobal) , (4)
where θ presents the parameters of the corresponding transforming networks w.r.t. T ci and Ti, respectively. Then, we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly obtain the attentive local-level features xci , as well as the global-level feature x global i . The holistic feature representation w.r.t. the input image Ii is achieved by concatenating both xci and xglobali , i.e., xi = [ xci ;x global i ] = F (Ii; Θ) ∈ Rd. Note that, we hereby abstract the aforementioned fine-grained
feature learning process as a function F (Ii; Θ) associated with its parameters Θ.
3.3 Attribute-Aware Hash Codes Generating
How to generate attribute-aware hash codes is the key of our A2-NET model. We elaborate it in the following three aspects, i.e., unsupervised attribute-guided learning, attribute-specific feature decorrelation, and hash code learning.
3.3.1 Unsupervised Attribute-Guided Learning
In real-applications, especially for the large-scale and fine-grained tasks, attribute annotations are always infeasible, which limits the learning process to be conducted in an unsupervised setting. While, in the literature, the main goal of unsupervised learning is to capture regularities in data for the purpose of extracting useful representations or for restoring corrupted data [31]. Many unsupervised methods explicitly produce internal latent units or codes, from which the data is to be reconstructed.
Inspired by this, we develop an unsupervised attribute-guided reconstruction component to project the holistic representation xi of Ii into a latent space, i.e., the attribute space V . In V , its high-level vectors are designed to have certain desirable properties, e.g., corresponding to semantic properties of fine-grained objects (aka “fine-grained attributes”).
More specifically, in our A2-NET, the unsupervised attribute-guided learning is realized by a reconstruction paradigm with an encoder-decoder structure, as shown in Figure 2. In concretely, given a batch of n training data Ii, their holistic representations X = {x1;x2; . . . ;xn} ∈ Rd×n can be obtained as aforementioned. By formulation, the encoder projects X into the attribute space V with a projection matrix W ∈ Rk×d to get an internal latent representation V ∈ Rk×n w.r.t. X . In particularly, we set that the dimension of latent representation k equals the number of hash bits in the final binary hash code ui. Furthermore, each column of V , i.e., vi ∈ Rk, can derive ui by
ui = sgn(tanh(vi)) . (5)
Meanwhile, regarding vi, we also consider to reconstruct its input xi by a decoder as a counterpart of the encoder. Therefore, on one hand, such a reconstruction paradigm can reduce and further distill high-level semantic cues in the attribute space V . While, on the other hand, it can drive the training
of A2-NET by preserving the similarity between queried hash codes and database points in terms of hash code learning (cf. Section 3.3.3).
In concretely, the learning objective of unsupervised attribute-guided reconstruction is written as follows:
min W ‖X −W>WX‖2F s.t.WX = V ′ = tanh(V ) , (6)
where the decoder (i.e., a counterpart of the encoder) is realized by W> to simplify the network. However, directly minimizing Eq. (6) with a hard constraint is difficult to optimize. Therefore, we relax the constraint into a soft constraint, and then the learning objective can be rewritten as
min W ‖X −W>V ′‖2F + λ‖WX − V ′‖2F . (7)
3.3.2 Attribute-Specific Feature Decorrelation
By conducting the aforementioned unsupervised attribute-guided learning, we can obtain the internal latent vectors V ′ as the attribute-specific features. In order to both enhance the discriminative ability and remove the redundant correlation among these attribute-specific features, we introduce a feature decorrelation constraint upon V ′, which is formulated by
min V ′ ‖V ′V ′> − nI‖2F , (8)
where I is the identity matrix and n is the batch size. Such a feature decorrelation constraint is preferable to construct independent features and reduce redundant information. Therefore, based on both unsupervised attribute-guided reconstruction and attribute-specific feature decorrelation, the final learned hash codes are expected to be both attribute-aware and hash-bit independent.
3.3.3 Hash Code Learning
In the following, we conduct the hash code learning based on the obtained attribute-specific features. Assume that we have n query data points which are denoted as {qi}ni=1, as well as m database points which are denoted as {vj}mj=1. Note that, both qi and vi are belonging to the attribute space V . By following Eq. (5), the corresponding binary hash codes can be obtained via
ui = sgn(tanh(qi)) , (9) zj = sgn(tanh(vj)) , (10)
where ui, zj ∈ {−1,+1}k. The goal of our hash code learning is to learn binary hash codes for both query points and database points from {qi}ni=1, {vj}mj=1, and the pairwise supervised information, i.e., S ∈ {−1,+1}n×m. To preserve the pairwise similarity, we adopt the `2 loss between the supervised information (aka similarity) and the inner product of query-database point binary code pairs. It can be formulated as follows:
minU ,Z ∑n
i=1 ∑m j=1 ( u>i zj − kSij )2
s.t. U ∈ {−1,+1}n×k,Z ∈ {−1,+1}m×k . (11)
Overall, we get the final objective of the proposed A2-NET model by considering Eq. (7), Eq. (8) and Eq. (11) together as follows:
min W ,Θ
L(I) = ‖X−W>V ′‖2F +λ‖WX−V ′‖2F +α‖V ′V ′>−nI‖2F +β n∑
i=1
m∑
j=1
( u>i zj − kSij )2 ,
(12) where λ, α and β are hyper-parameters as the trade-off.
In practice, during training, it might be only available a set of database points {vj}mj=1 without query points. Thus, we randomly sample n data points from database to construct a query set, and denote the indices of all the database points as Γ with the indices of the query set as Ω. Additionally, because we cannot back-propagate the gradient to Θ due to the sgn(·) function, we omit the sgn(·) function and only apply tanh(·) for relaxation in Eq. (10) of the whole optimization process. Therefore, the
optimization formulation of A2-NET can be rewritten with only database points {vj}mj=1 for training as:
min W ,Θ
L(I) = ‖X −W>V ′‖2F + λ‖WX − V ′‖2F + α‖V ′V ′> − nI‖2F
+ β ∑
i∈Ω
∑
j∈Γ
( tanh(W · F (Ii; Θ))>zj − kSij )2 . (13)
For optimization, our A2-NET does not require complicated two-stage learning algorithms, e.g., the alternative optimization strategy. In experiments, we employ the back-propagation algorithm and follow [17] to train the whole A2-NET model in a unified end-to-end manner.
3.4 Out-of-Sample Extension
After training A2-NET, the learned model can be applied for generating binary codes for query points including unseen query points in the training phase. Specifically, we can use the following equation to generate the binary code for Iq:
uq = sgn(tanh(W · F (Iq; Θ))) . (14)
4 Experiments
4.1 Datasets
By following ExchNet [8], our experiments are conducted on five fine-grained benchmark datasets, i.e., CUB200-2011 [37], Aircraft [28], Food101 [2], NABirds [35] and VegFru [14]. Specifically, CUB200-2011 is one of the most popular used fine-grained datasets. It contains 11,788 bird images from 200 bird species and is officially split into 5,994 images for training and 5,794 images for test. Aircraft contains 10,000 images spanning 100 aircraft models with 3,334 for training, 3,333 for validation and 3,333 for test. For large-scale datasets, Food101 contains 101 kinds of foods with 101,000 images, where for each class, 250 test images are checked manually for correctness while 750 training images still contain a certain amount of noises. NABirds is a high quality dataset which has 48,562 images of North American birds with 555 sub-categories, where 23,929 for training with 24,633 for test. VegFru is another large-scale fine-grained dataset covering 200 kinds of vegetables and 92 kinds of fruits with 29,200 for training, 14,600 for validation and 116,931 for test.
4.2 Baselines and Implementation Details
Baselines In experiments, we compare our proposed model to the following competitive baselines, i.e., ITQ [11], SDH [33], DPSH [24], HashNet [5], and ADSH [17]. Among them, DPSH, HashNet and ADSH are deep learning based methods, while ITQ and SDH are not. Furthermore, we also compare the results of our A2-NET with state-of-the-arts of fine-grained hashing methods, including ExchNet [8]. Additionally, another fine-grained hashing method, i.e., DSaH [19], also achieved good retrieval accuracy. However, due to its empirical settings quite distant from other existing fine-grained hashing methods, for fair comparisons, we strictly control empirical settings as the same as those of [19] and compare the results of our A2-NET with its results in the supplementary materials.
Implementation Details For fair comparisons, we follow the efficient training setting in ExchNet [8]. In concretely, for CUB200-2011, Aircraft and Food101, we sample 2,000 images per epoch, while 4,000 samples are randomly selected for NABirds and VegFru. For the training details, regarding the backbone model, we can choose any network structure as the base network for the fine-grained representation learning module. While, by following [8], ResNet-50 [13] is employed in experiments. The total number of training epochs is 20, and the number of batch size is set as 16. While, different from ExchNet, our model only requires a smaller iteration time until convergence. Specifically, for these datasets containing less than 20,000 training images, the iteration time Tmax is 60, and the learning rate is divided by 10 at the 50th iteration. For other datasets, Tmax is set as 70, and the learning rate is divided by 10 at the 60th iteration. The hyper-parameters, i.e., λ, α and β in Eq. (13), are set as 1, 1n×k and 12 k , respectively. The number of attention guidance equals the number of hash bits. The optimizer is standard mini-batch stochastic gradient descent with the weight decay as 10−4. All experiments are conducted with a GeForce RTX 2080 Ti GPU.
4.3 Main Results
Table 1 presents the mean average precision (mAP) results of fine-grained retrieval on these five aforementioned fine-grained benchmark datasets. For each dataset, we report the results of four lengths of hash bits, i.e., 12, 24, 32, and 48, for evaluations. As shown in that table, our proposed A2NET model significantly and consistently outperforms the other baseline methods on these datasets. In particular, compared with the state-of-the-art method ExchNet [8], our A2-NET achieves 17.83% and 17.88% improvements over ExchNet of 24-bit and 32-bit experiments on Aircraft and Food-101, respectively. Moreover, A2-NET also obtains superior results with an absolute value of about 80% mAP on CUB200-2011, Aircraft and Food101 with 48-bit hash codes. These observations validate the effectiveness of the proposed A2-NET model, as well as its promising practicality in real-applications of fine-grained retrieval. Additionally, in Figure 3, we illustrate several retrieval results on CUB2002011, which shows that A2-NET can retrieve well among multiple subordinate categories when the same species of birds with diverse variations appear in different kinds of background. Also, there also exist several failure cases, where quite tiny differences (e.g., caused by different views) between the query image and the returned images are demanded by carefully observations.
4.4 Ablation Studies
In this section, we demonstrate the effectiveness of these crucial components of the proposed A2-NET model, i.e., the attention-based fine-grained representation learning component 3.2, the unsupervised
attribute-guided reconstruction component (cf. Section 3.3.1) and the attribute-specific feature decorrelation component (cf. Section 3.3.2). In the ablation studies, we apply these components incrementally on a vanilla backbone (i.e., ResNet-50) as the baseline. As evaluated in Table 2, by stacking these two components one by one, the retrieval results are steadily improved, which justifies the effectiveness of our proposed components in A2-NET.
4.5 Qualitative Analyses of Attribute-Aware Hash Codes
We hereby discuss the quality of the learned attribute-aware hash codes ui of A2-NET. After obtaining ui, we visualize fine-grained images retrieved by a random single hash bit of ui to demonstrate the strong correspondence between visual attributes and the obtained hash bits. All the five datasets in experiments are used as examples to illustrate the quality. As observed in Figure 4, images of each column have some similar fine-grained object properties, i.e., visual attributes. Indeed, the learned hash codes are apparently attribute-aware, which could provide an explanation of the A2-NET’s success in fine-grained retrieval. Meanwhile, it also offers human-understandable interpretation for such a deep learning based fine-grained hashing method.
5 Conclusion
In this paper, we proposed an Attribute-Aware hashing Network, i.e., A2-NET, for dealing with the large-scale fine-grained image retrieval task. Particularly, A2-NET was designed as expected to be efficient, effective and more importantly interpretable. In concretely, by developing an unsupervised attribute-guided reconstruction method based on the obtained appearance-specific visual representation with attention, it can distill attribute-specific vectors in a high-level attribute space. After further performing feature decorrelation upon attribute-specific vectors, their discriminative ability is strengthened for representing a fine-grained object. Then, hash codes can be generated from these attribute-specific vectors and thus became attribute-aware. Both qualitative and quantitative experiments demonstrate the effectiveness of our A2-NET. Additionally, visual attributes have been shown to be useful in describing both known and unknown entities, which motivates us to study identifying unobserved sub-categories, i.e., zero-shot fine-grained recognition, based on our attribute-aware hash codes as the future work.
References [1] Thomas Berg, Jiongxin Liu, Seung Woo Lee, Michelle L. Alexander, David W. Jacobs, and
Peter N. Belhumeur. Birdsnap: Large-scale fine-grained visual categorization of birds. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 2019–2026, 2014.
[2] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 – mining discriminative components with random forests. In Proc. Eur. Conf. Comp. Vis., pages 446–461, 2014.
[3] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using a “siamese” time delay neural network. In Proc. Advances in Neural Inf. Process. Syst., pages 737–744, 1993.
[4] Fatih Cakir, Kun He, Sarah Adel Bargal, and Stan Sclaroff. Hashing with mutual information. IEEE Trans. Pattern Anal. Mach. Intell., 41(10):2424–2437, 2019.
[5] Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Philip S. Yu. HashNet: Deep learning to hash by continuation. In Proc. IEEE Int. Conf. Comp. Vis., pages 5608–5617, 2017.
[6] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proc. Int. Conf. Mach. Learn., pages 1597–1607, 2020.
[7] Maurizio Corbetta and Gordon L. Shulman. Control of goal-directed and stimulus-driven attention in the brain. Nature Reviews Neuroscience, 3:201–215, 2002.
[8] Quan Cui, Qing-Yuan Jiang, Xiu-Shen Wei, Wu-Jun Li, and Osamu Yoshie. ExchNet: A unified hashing network for large-scale fine-grained image retrieval. In Proc. Eur. Conf. Comp. Vis., pages 189–205, 2020.
[9] Anirban Dasgupta, Ravi Kumar, and Tamas Sarlos. Fast locality-sensitive hashing. In Proc. ACM SIGKDD Int. Conf. Knowledge discovery & data mining, pages 1073–1081, 2011.
[10] Vittorio Ferrari and Andrew Zisserman. Learning visual attributes. In Proc. Advances in Neural Inf. Process. Syst., pages 433–440, 2007.
[11] Yunchao Gong, Svetlana Lazebnik, Albert Gordo, and Florent Perronnin. Iterative quantization: A procrustean approach to learning binary codes for large-scale image retrieval. IEEE Trans. Pattern Anal. Mach. Intell., 35(12):2916–2929, 2012.
[12] Kun He, Fatih Cakir, Sarah Adel Bargal, and Stan Sclaroff. Hashing as tie-aware learning to rank. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 4023–4032, 2018.
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 770–778, 2016.
[14] Saihui Hou, Yushan Feng, and Zilei Wang. VegFru: A domain-specific dataset for fine-grained visual categorization. In Proc. IEEE Int. Conf. Comp. Vis., pages 541–549, 2017.
[15] Junshi Huang, Rogerio Feris, Qiang Chen, and Shuicheng Yan. Cross-domain image retrieval with a dual attribute-aware ranking network. In Proc. IEEE Int. Conf. Comp. Vis., pages 1062–1070, 2015.
[16] Laurent Itti, Christof Koch, and Ernst Niebur. A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell., 20(11):1254–1259, 1998.
[17] Qing-Yuan Jiang and Wu-Jun Li. Asymmetric deep supervised hashing. In Proc. Conf. AAAI, pages 3342–3349, 2018.
[18] Sheng Jin, Hongxun Yao, Xiaoshuai Sun, Shangchen Zhou, Lei Zhang, and Xiansheng Hua. Deep saliency hashing for fine-grained retrieval. IEEE Trans. Image Process., 29:5336–5351, 2020.
[19] Sheng Jin, Hongxun Yao, Xiaoshuai Sun, Shangchen Zhou, Lei Zhang, and Xiansheng Hua. Deep saliency hashing for fine-grained retrieval. IEEE Trans. Image Process., 29:5336–5351, 2020.
[20] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D object representations for fine-grained categorization. In ICCV Workshop on 3D Representation and Recognition, 2013.
[21] Neeraj Kumar, Alexander Berg, Peter N. Belhumeur, and Shree Nayar. Describable visual attributes for face verification and image search. IEEE Trans. Pattern Anal. Mach. Intell., 33(10):1962–1977, 2011.
[22] Hugo Larochelle and Geoffrey Hinton. Learning to combine foveal glimpses with a third-order boltzmann machine. In Proc. Advances in Neural Inf. Process. Syst., pages 1243–1251, 2010.
[23] Ke Li, Kaiyue Pang, Yi-Zhe Song, Timothy M. Hospedales, Tao Xiang, and Honggang Zhang. Synergistic instance-level subspace alignment for fine-grained sketch-based image retrieval. IEEE Trans. Image Process., 26(12):5908–5921, 2017.
[24] Wu-Jun Li, Sheng Wang, and Wang-Cheng Kang. Feature learning based deep supervised hashing with pairwise labels. In Proc. Int. Joint Conf. Artificial Intell., pages 1711–1717, 2015.
[25] Yi Li, Timothy M. Hospedales, Yi-Zhe Song, and Shaogang Gong. Fine-grained sketch-based image retrieval by matching deformable part models. In Proc. British Machine Vis. Conf., pages
1–12, 2014. [26] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. DeepFashion: Powering
robust clothes recognition and retrieval with rich annotations. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 1096–1104, 2016.
[27] Qin Lv, William Josephson, Zhe Wang, Moses Charikar, and Kai Li. Multi-probe LSH: Efficient indexing for high-dimensional similarity search. In Proc. Int. Conf. Very Large Data Bases, pages 950–961, 2007.
[28] Subhransu Maji, Esa Rahtu, Juho Kannala, Mathhew Blaschko, and Andrea Vedaldi. Finegrained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013.
[29] Kaiyue Pang, Yongxin Yang, Timothy M. Hospedales, Tao Xiang, and Yi-Zhe Song. Solving mixed-modal jigsaw puzzle for fine-grained sketch-based image retrieval. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 10347–10355, 2020.
[30] Genevieve Patterson, Chen Xu, Hang Su, and James Hays. The SUN attribute database: Beyond categories for deeper scene understanding. Int. J. Comput. Vision, 108(1-2):59–81, 2014.
[31] Marc’Aurelio Ranzato, Y-Lan Boureau, Sumit Chopra, and Yann LeCun. A unified energy-based framework for unsupervised learning. In ICAIS, pages 371–379, 2007.
[32] Aneeshan Sain, Ayan Kumar Bhunia, Yongxin Yang, Tao Xiang, and Yi-Zhe Song. Cross-modal hierarchical modelling for fine-grained sketch based image retrieval. In Proc. British Machine Vis. Conf., pages 1–14, 2020.
[33] Fumin Shen, Chunhua Shen, Wei Liu, and Heng Tao Shen. Supervised discrete hashing. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 37–45, 2015.
[34] Anshumali Shrivastava and Ping Li. Densifying one permutation hashing via rotation for fast near neighbor search. In Proc. Int. Conf. Mach. Learn., pages 557–565, 2014.
[35] Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie. Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 595–604, 2015.
[36] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The iNaturalist species classification and detection dataset. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 8769–8778, 2017.
[37] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech-UCSD birds-200-2011 dataset. Tech. Report CNS-TR-2011-001, 2011.
[38] Jingdong Wang, Heng Tao Shen, and Ting Zhang. A survey on learning to hash. IEEE Trans. Pattern Anal. Mach. Intell., 40(4):769–790, 2018.
[39] Xiu-Shen Wei, Quan Cui, Lei Yang, Peng Wang, and Lingqiao Liu. RPC: A large-scale retail product checkout dataset. arXiv preprint arXiv:1901.07249, 2019.
[40] Xiu-Shen Wei, Jian-Hao Luo, Jianxin Wu, and Zhi-Hua Zhou. Selective convolutional descriptor aggregation for fine-grained image retrieval. IEEE Trans. Image Process., 26(6):2868–2881, 2017.
[41] Xiu-Shen Wei, Jianxin Wu, and Quan Cui. Deep learning for fine-grained image analysis: A survey. arXiv preprint arXiv:1907.03069, 2019.
[42] Yongqin Xian, Christoph H. Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning–a comprehensive evaluation of the good, the bad and the ugly. IEEE Trans. Pattern Anal. Mach. Intell., 41(9):2251–2265, 2019.
[43] Qian Yu, Feng Liu, Yi-Zhe Song, Tao Xiang, Timothy M. Hospedales, and Chen-Change Loy. Sketch me that shoe. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 799–807, 2016.
[44] Xianxian Zeng, Yun Zhang, Xiaodong Wang, Kairui Chen, Dong Li, and Weijun Yang. Finegrained image retrieval via piecewise cross entropy loss. Image and Vision Computing, 93:2868– 2881, 2020.
[45] Hua Zhang, Xiaochun Cao, and Rui Wang. Audio visual attribute discovery for fine-grained object recognition. In Proc. Conf. AAAI, pages 7542–7549, 2018.
[46] Jingyi Zhang, Fumin Shen, Li Liu, Fan Zhu, Mengyang Yu, Ling Shao, Heng Tao Shen, and Luc Van Gool. Generative domain-migration hashing for sketch-to-image retrieval. In Proc. Eur. Conf. Comp. Vis., pages 297–314, 2018.
[47] Xiawu Zheng, Rongrong Ji, Xiaoshuai Sun, Yongjian Wu, Feiyue Huang, and Yanhua Yang. Centralized ranking loss with weakly supervised localization for fine-grained object retrieval. In Proc. Int. Joint Conf. Artificial Intell., pages 1226–1233, 2018.
[48] Xiawu Zheng, Rongrong Ji, Xiaoshuai Sun, Baochang Zhang, Yongjian Wu, and Feiyue Huang. Towards optimal fine grained retrieval via decorrelated centralized loss with normalize-scale layer. In Proc. Conf. AAAI, pages 9291–9298, 2019.

Review summaries:
This paper presents an attribute-aware hashing network for generating attribute-aware hash codes. The proposed method includes two keys components: (1) visual feature extraction with attention, (2) learning attribute-specific features with autoencoder.   The authors conducted experiments on many datasets, and it outperforms recent SOTA approaches.

In this paper, authors propose to learn attribute-aware hash codes for fine-grained image retrieval, which establish the explicit correspondences between hash codes and visual attributes.   The attention mechanism is employed to extract the fine-grained tailored visual patterns from both global-level and local-level, and unsupervised reconstruction task is defined to distill the attribute related vectors through an encoder-decoder structure. Finally the hash codes are generated by the attribute vectors.

This paper proposes a unified framework to generate attribute-aware hash codes for dealing with the large-scale fine-grained image retrieval task. Specifically, it develops an attribute-aware hashing network, termed as A^2-Net, consisting of two main modules including fine-grained representation learning and attribute-aware hash codes generating. In A^2-Net, the proposed reconstruction task realizing by an encoder-decoder structure is proven to be able to unsupervisedly distill high-level attribute-specific vectors from the appearance visual representations without attribute annotations. Thus, the hash codes can be generated by the attribute vectors based on a feature decorrelation constraint and the similarity preserving constraint. Experiments are conducted on several fine-grained benchmarks, which can show outperforming on both qualitative and quantitative aspects.

(1)	This paper aims to address the large-scale fine-grained image retrieval, and it introduces an attention mechanism into a CNN backbone to capture fine-grained local patterns in an image. And it integrates the attentive local-level feature and the extracted global-level feature by the same CNN as the holistic feature of an image, (2)	It proposes an Attribute-Aware Hash Codes Generating module with an encoder-decoder network by encoding the holistic feature to a latent space, and decoding the latent feature to the holistic feature, and projecting the latent feature to a binary hash code.  (3)	It designs the empirical experiments on 5 fine-grained benchmark datasets to verify the performance improvement of the proposed method compared to existing methods.