A$^2$-Net: Learning Attribute-Aware Hash Codes for Large-Scale Fine-Grained Image Retrieval

1 Introduction Fine-grained image retrieval in computer vision aims to retrieve images belonging to multiple subordinate categories of a super-category ( aka a meta-category ) , e.g. , different species of animals/plants [ 36 ] , different models of cars [ 20 ] , different kinds of retail products [ 39 ] , etc . Its key challenge therefore lies with understanding fine-grained visual differences that sufficiently discriminate between objects that are highly similar in overall appearance , but differ in fine-grained features . Also , fine-grained retrieval still demands ranking all the instances so that images depicting the concept of interest ( e.g. , the same sub-category label ) are ranked highest based on the fine-grained details in the query . In particular , with the explosive growth of fine-grained data in real applications [ 1 , 14 , 26 , 36 , 39 ] , fine-grained hashing , as a promising solution for dealing with large-scale fine-grained retrieval tasks , has proven to be able to greatly reduce the storage cost and increase the query speed [ 8 , 18 ] benefiting from the learned compact binary hash code representations . However , although previous works , ∗Corresponding author . X.-S. Wei , Y. Shen , X . Sun and J. Yang are with PCA Lab , Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education , and Jiangsu Key Lab of Image and Video Understanding for Social Security , School of Computer Science and Engineering , Nanjing University of Science and Technology . This work was supported by Natural Science Foundation of Jiangsu Province of China under Grant ( BK20210340 ) , the Fundamental Research Funds for the Central Universities ( No . 30920041111 ) , and CAAI-Huawei MindSpore Open Fund ( CAAIXSJLJJ-2020-022A ) . The first two authors contributed equally to this work . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Query image 101 1 Encoder 3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET model . We elaborate it in65 the following four aspects , i.e. , unsupervised attribute-guided learning , attribute-specific feature66 decorrelation , hash code learning and the optimization learning algorithm.67 3.3.1 Unsupervised Attribute-Guided Learning68 In real-applications , especially for the large-scale and fine-grained tasks , attribute annotations are69 always infeasible , which limits the learning process to be conducted in an unsupervised setting.70 While , in the literature , the main goal of unsupervised learning is to capture regularities in data for the71 purpose of extracting useful representations or for restoring corrupted data [ 4 ] . Many unsupervised72 methods explicitly produce internal latent units or codes , from which the data is to be reconstructed.73 Inspired by this , we develop an unsupervised attribute-guided learning component to project the74 holistic representation xi of Ii into a latent space , i.e. , the attribute space V . In V , its high-level75 vectors are designed to have certain desirable properties , e.g. , corresponding to semantic properties76 of fine-grained objects ( a.k.a . “ fine-grained attributes ” ) .77 More specifically , in our A2-NET , the unsupervised attribute-guided learning is realized by a recon-78 struction paradigm with an encoder-decoder structure , as shown in Figure ? ? . In concretely , given79 a batch of n training data Ii , the holistic representations X = { x1 ; x2 ; . . . ; xn } 2 Rd⇥n can be80 obtained as aforementioned . By formulation , the encoder projects X into the attribute space V with81 a projection matrix W 2 Rk⇥d to get an internal latent representation V 2 Rk⇥n w.r.t . X . In82 particularly , we set that the dimension of latent representation k equals to the number of hash bits in83 the final binary hash code ui . Furthermore , each column of V , i.e. , vi 2 Rk , can derive ui by84 ui = sgn ( tanh ( vi ) ) . ( 5 ) Meanwhile , regarding vi , we also consider to reconstruct its input xi by a decoder as a counterpart85 of the encoder . Therefore , on one hand , such a reconstruction paradigm can reduce and further distill86 high-level semantic cues in the attribute space V . While , on the other hand , it can drive the training87 of A2-NET by preserving the similarity between queried hash codes and database points in terms of88 learning to hash ( cf . Section 3.3.3 ) .89 In concretely , the learning objective of unsupervised attribute-guided learning is written as follows:90 min W kX W > WXk2F s.t . WX = V 0 = tanh ( V ) , ( 6 ) where the decoder ( i.e. , a counterpart of the encoder ) is realized by W > to simplify the network.91 However , directly minimizing Eq . ( 6 ) with a hard constraint is hard to optimize . Therefore , we relax92 the constraint into a soft constraint , and the learning objective can be rewritten by93 min W kX W > V 0k2F + kWX V 0k2F . ( 7 ) 3.3.2 Attribute-Specific Feature Decorrelation94 By conducting the aforementioned unsupervised attribute-guided learning , we can obtain the internal95 latent vectors V 0 as the attribute-specific features . In order to both enhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features , we introduce a feature97 decorrelation constraint upon V 0 , which is formulated by98 min V 0 kV 0V 0 > kIk2F , ( 8 ) where I is the identity matrix . Such a feature decorrelation constraint is preferable to construct99 independent features and reduce redundant information . Therefore , based on unsupervised attribute-100 guided learning and attribute-specific feature decorrelation , the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102 3.3.3 Hash Code Learning103 In the following , we conduct the hash code learning based on the obtained attribute-specific features.104 Assume that we have n query data points which are denoted as { qi } ni=1 , as well as m database points105 3 3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET model . We elaborate it in65 the following four aspects , i.e. , unsupervised attribute-guided learning , attribute-specific feature66 decorrelation , hash code learning and the optimization learning algorithm.67 3.3.1 Unsupervised Attribute-Guided Learning68 In real-applications , especially for the large-scale and fine-grained tasks , attribute annotations are69 always infeasible , which limits the learning process to be conducted in an unsupervised setting.70 While , in the literature , the main goal of unsupervised learning is to capture regularities in data for the71 purpose of extracting useful representations or for restoring corrupted data [ 4 ] . Many unsupervised72 methods explicitly produce internal latent units or codes , from which the data is to be reconstructed.73 Inspired by this , we develop an unsupervised attribute-guided learning component to project the74 holistic representation xi of Ii into a latent space , i.e. , the attribute space V . In V , its high-level75 vectors are designed to have certain desirable properties , e.g. , corresponding to semantic properties76 of fine-grained objects ( a.k.a . “ fine-grained attributes ” ) .77 More specifically , in our A2-NET , the unsupervised attribute-guided learning is realized by a recon-78 struction paradigm with an encoder-decoder structure , as shown in Figure ? ? . In concretely , given79 a batch of n training data Ii , the holistic representations X = { x1 ; x2 ; . . . ; xn } 2 Rd⇥n can be80 obtained as aforementioned . By formulati n , th en oder pr jects X into the attribute sp ce V with81 a projection matrix W 2 Rk⇥d to get an internal latent representation V 2 Rk⇥n w. .t . X . In82 particularly , we set that the dimension of latent representation k equals to the number of hash bits in83 the final binary hash code ui . Furthermore , each column of V , i.e. , vi 2 Rk , ca derive ui by84 ui = sgn ( tanh ( vi ) ) . ( 5 ) Meanwhile , regarding vi , we also consider to reconstruct its input xi by a decoder as a counterpart85 of the encoder . Therefore , on one hand , such a reconstruction paradigm can reduce and further distill86 high-level semantic cues in the attribute space V . While , on the other hand , it can drive the training87 of A2-NET by preserving the similarity between queried hash cod s and database points in terms of88 learning to hash ( cf . Section 3.3.3 ) .89 In concretely , the learning objective of unsupervised attribute-guided learning is written as follows:90 min W kX W > WXk2F s.t . WX = V 0 = tanh ( V ) , ( 6 ) where the decoder ( i.e. , a counterpart of the encoder ) is realized by W > to simplify the network.91 However , directly minimizing Eq . ( 6 ) with a hard constraint is hard to optimize . Therefore , we relax92 the constraint into a soft constraint , and the learning objective can be rewritten by93 min W kX W > V 0k2F + kWX V 0k2F . ( 7 ) 3.3.2 Attribute-Specific Feature Decorrelation94 By conducting the aforementioned unsupervised attribute-guided learning , we can obtain the internal95 latent vectors V 0 as the attribute-specific features . In order to both nhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features , we introduce a feature97 decorrelation constraint upon V 0 , which is formulated by98 mi V 0 kV 0V 0 > kIk2F , ( 8 ) where I is the identity matrix . Such a feature decorrelation constraint is preferable to construct99 independent features and reduce redundant information . Therefore , based on unsupervised attribute-100 guided learning and attribute-specific feature decorrelation , the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102 3.3.3 Hash Code Learning103 In the following , we conduct the hash code learning based on the obtained attribute-specific features.104 Assume that we have n query data points which are denoted as { qi } ni=1 , as well as m database points105 3 3.3 Attribute-Aware Hash C es Gen ating64 How to generate attribute-aware hash codes is the key of our A2-NET model . We elaborate it in65 the following four aspec s , i.e. , unsup rvised attribute-guided learning , attribute-specific feature66 decorrelati n , hash code learning and the optimization learning algorithm.67 3.3.1 Unsupervised Attribute-Guided Learning68 In real-applications , especially for the large-scale and fine-grained tasks , attribute annotations are69 always infeasible , which limits the learning process to be conducted in an unsupervised setting.70 While , in the literature , the main goal of unsupervised learning is to capture regularities in data for the71 purpose of extracting useful representations or for restoring corrupted data [ 4 ] . Many unsupervised72 methods explicitly produce internal latent units or codes , from which the data is to be reconstructed.73 Inspired by this , we develop an unsupervised attribute-guided learning component to project the74 holistic representation xi of Ii into a latent space , i.e. , the attribute space V . In V , its high-level75 vectors are designed to have certain desirable properties , e.g. , corresponding to semantic properties76 of fine-grained objects ( a.k.a . “ fine-grained attributes ” ) .77 More specifically , in our A2-NET , the unsupervised attribute-guided learning is realized by a recon-78 structi n paradigm with an encoder-decod r structure , as shown in Figure ? ? . In concretely , given79 a batch of n training data Ii , the holistic representations X = { x1 ; x2 ; . . . ; xn } 2 Rd⇥n can be80 obtained as aforementioned . By formulation , the encoder projects X into the attribute space V with81 a projectio matrix W 2 Rk⇥d to get an nternal latent representation V 2 Rk⇥n w.r.t . X . I82 particularly , w set that the dimension of latent repre entation k equals t the number of hash bits in83 the final binary ash c de ui . Furthermore , each column of V , i.e. , vi 2 Rk , can derive ui by84 ui = sgn ( tanh ( vi ) ) . ( 5 ) Meanwhile , regarding vi , we also consider to reconstruct its input xi by a decoder as a counterpart85 of the encod r. Therefore , on one hand , uch a reconstructio paradigm can reduce and further distill86 high-level semantic cues in the attribute space V . While , on the other hand , it can drive the training87 of A2-NET by preserving the similarity between queried hash codes and database points in terms of88 learning to hash ( cf . Section 3.3.3 ) .89 In con retely , the learn ng objective of u supervised attribute-guided learning is written as follows:90 min W kX W > WXk2F .t . WX = V 0 = tanh ( V ) , ( 6 ) where the decoder ( i.e. , a counterpart f the encoder ) is r alized y W > to simplify the network.91 However , irectly minimizi g Eq . ( 6 ) with a hard con traint is hard to optimize . Therefore , we relax92 the constraint into a soft constraint , and the learning objective can be rewritten by93 min W kX W > V 0k2F + kWX V 0k2F . ( 7 ) 3.3.2 Attribute-Specific Feature Decorrelation94 By conducting the aforementioned unsupervised attribute-guided learning , we can obtain the internal95 latent vectors V 0 as the attribute-specific features . In order to both enhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features , we introduce a feature97 decorrelation constraint upon V 0 , which is formulated by98 min V 0 kV 0V 0 > kIk2F , ( 8 ) where I is the identity matrix . Such a feature decorrelation constraint is preferable to construct99 independent features and reduce redundant information . Therefore , based on unsupervised attribute-100 guided learning and attribute-specific feature decorrelation , the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102 3.3.3 Hash Code Learning103 In the following , we conduct the hash code learning based on the obtained attribute-specific features.104 Assume that we have n query data points which are denoted as { qi } ni=1 , as well as m database points105 3 3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET model . We elaborat it in65 the following four aspects , i.e. , unsupervised attribute-guided learning , attribute-specific feature66 decorrelation , hash code learning and the optimization learning algorithm.67 3.3.1 Unsupervised Attribute-Guided Learning68 In real-applications , especially for the large-sca e and fine-grained tasks , attribute an otation re69 always infeasible , which limits the learning process to be nducted in an unsupervised setting.70 While , in the literature , the main goal of unsupervised learning is to capture regularities in data for the71 purpose of extracting useful representations or for restoring corrupted data [ 4 ] . Many unsupervised72 methods explicitly produce internal latent units or codes , from which the data is to be reconstructed.73 Inspired by this , we develop an unsupervised attribute-guided learning component to project the74 holistic representation xi of Ii into a latent space , i.e. , the attribute sp ce V . In V , its high-lev l75 vectors are designed to have certain desirable properties , e.g. , corresponding to semantic properties76 of fine-grained objects ( a.k.a . “ fine-grained attributes ” ) .77 More specifically , in our A2-NET , the unsupervised attribute-guided learning is realized by a recon-78 struction paradigm with an encoder-decoder structure , as shown in Figure ? ? . In concretely , given79 a batch of n training data Ii , the holistic representations X = { x1 ; x2 ; . . . ; xn } 2 Rd⇥n can be80 obtained as aforementioned . By formulation , the encoder projects X into the attribute space V with81 a projection matrix W 2 Rk⇥d to get an internal latent representation V 2 Rk⇥n w.r.t . X . In82 particularly , we set that the dimension of latent representation k equals to the number of hash bits in83 the final binary hash code ui . Furthermore , each column of V , i.e. , vi 2 Rk , can derive ui by84 ui = sgn ( tanh ( vi ) ) . ( 5 ) Meanwhile , regarding vi , we also consider to reconstruct its input xi by a decoder as a counterpart85 of the encoder . Therefore , on one hand , such a reconstruction paradigm can reduce and further distill86 high-level semantic cues in the attribute space V . While , on the other hand , it can drive the training87 of A2-NET by preserving the similarity between queried hash codes and database points in terms of88 learning to hash ( cf . Section 3.3.3 ) .89 In concretely , the learning objective of unsupervised attribute-guided learning is written as follows:90 min W kX W > WXk2F s.t . WX = V 0 = tanh ( V ) , ( 6 ) where the decoder ( i.e. , a counterpart of the encoder ) is realized by W > to simplify the network.91 However , directly minimizing Eq . ( 6 ) with a hard constraint is hard to optimize . Therefore , we relax92 the constraint into a soft constraint , and the learning objective can be rewritten by93 min W kX W > V 0k2F + kWX V 0k2F . ( 7 ) 3.3.2 Attribute-Specific Feature Decorrelation94 By conducting the aforementioned unsupervised attribute-guided learning , we can obtain the internal95 latent vectors V 0 as the attribute-specific features . In order to both enhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features , we introduce a feature97 decorrelation constraint upon V 0 , which is formulated by98 min V 0 kV 0V 0 > kIk2F , ( 8 ) where I is the identity matrix . Such a feature decorrelation constraint is pref rable to construct99 independent features and reduce redundant information . Therefore , based on unsupervise ttri t -100 guided learning and attribute-specific feature decorrelation , the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102 3.3.3 Hash Code Learning103 In the following , we conduct the hash code learning based on the obtained attribute-sp cific feat res.104 Assume that we have n query data points which are denoted as { qi } ni=1 , as well as m database points105 3 Decoder 3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET model . We elaborate it in65 the following four aspects , i.e. , unsupervised attribute-guided learning , attribute-specific feature66 decorrelation , hash code learning and the optimization learning algorithm.67 3.3.1 Unsupervised Attribute-Guided Learning68 In real-applications , especially for the large-scale and fine-grained tasks , attribute annotations are69 always infeasible , which limits the learning process to be conducted in an unsupervised setting.70 While , in the literature , the main goal of unsupervised learning is to capture regularities in data for the71 purpose of extracting useful representations or for restoring corrupted data [ 4 ] . Many unsupervised72 methods explicitly produce internal latent units or codes , from which the data is to be reconstructed.73 Inspired by this , we develop an unsupervised attribute-guided learning component to project the74 holistic representation xi of Ii into a latent space , i.e. , the attribute space V . In V , its high-level75 vectors are designed to have certain desirable properties , e.g. , corresponding to semantic properties76 of fine-grained objects ( a.k.a . “ fine-grained attributes ” ) .77 More specifically , in our A2-NET , the unsupervised attribute-guided learning is realized by a recon-78 struction paradigm with an encoder-decoder structure , as shown in Figure ? ? . In concretely , given79 a batch of n training data Ii , the holistic representations X = { x1 ; x2 ; . . . ; xn } 2 Rd⇥n can be80 obtained as aforementioned . By formulation , the encoder projects X into the attribute space V with81 a projection matrix W 2 Rk⇥d to get an internal latent representation V 2 Rk⇥n w.r.t . X . In82 particularly , we set that the dimension of latent representation k equals to the number of hash bits in83 the final binary hash code ui . Furthermore , each column of V , i.e. , vi 2 Rk , can derive ui by84 ui = sgn ( tanh ( vi ) ) . ( 5 ) Meanwhile , regarding vi , we lso consider to reco struct its input xi by a decod r as a counterpart85 of the ncoder . Therefore , on one ha d , such a rec nstruction paradigm can reduce and further distill86 high-level semantic cues in the attribute space V . While , on the other hand , it can drive the training87 of A2-NET by preserving the similarity between queried hash codes and database points in terms of88 learning to hash ( cf . Section 3.3.3 ) .89 In concretely , the learni g objective of unsupervised attribute-guided learni g is written as follows:90 min W kX W > WXk2F s.t . WX = V 0 = tanh ( V ) , ( 6 ) where the decoder ( i.e. , a counterpart of the encod r ) is realized by W > to simplify the network.91 How ve , directly minimizing Eq . ( 6 ) with a hard constraint is hard to optimize . Therefore , we relax92 the constraint into a soft constraint , nd the learning objective can be rewritten by93 min W kX W > V 0k2F + kWX V 0k2F . ( 7 ) 3.3.2 Attribute-Specific Feature Decorrelation94 By conducting the aforementioned unsupervised attribute-guided learning , we can obtain the internal95 latent vectors V 0 as the attribute-specific features . In order to both enhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features , we introduce a feature97 decorrelation constraint upon V 0 , which is formulated by98 min V 0 kV 0V 0 > kIk2F , ( 8 ) where I is the identity matrix . Such a feature decorrelation constraint is preferable to construct99 independent features and reduce redundant information . Therefore , based on unsupervised attribute-100 guided learning and attribute-specific feature decorrelation , the final learned hash codes are expected101 to be both attribute-aware and hash-bit independ nt.102 3.3.3 Hash Code Learning103 In the following , we conduct the hash code learning based on the obtained attribute-specific features.104 Assume that we have n query data points which are denoted as { qi } ni=1 , as well as m database points105 3 which are denoted as { vj } mj=1 . Note that , both qi and vi are belonging to the attribute space V . Then,106 the corresponding binary hash codes can be obtained via107 ui = sgn ( tanh ( qi ) ) , ( 9 ) zj = sgn ( tanh ( vj ) ) , ( 10 ) where ui , zj 2 { 1 , +1 } k. The goal of our hash code learning is to learn binary hash codes for both108 query points and database points from { qi } ni=1 , { vj } mj=1 , and the pairwise supervised information,109 i.e. , S 2 { 1 , +1 } n⇥m . To preserve the pairwise similarity , we adopt the ` 2 loss between the110 supervised information ( a.k.a . similarity ) and the inner product of query-database point binary code111 pairs . It can be formulated as follows:112 minU , Z Pn i=1 Pm j=1 u > i zj kSij 2 s.t . U 2 { 1 , +1 } n⇥k , Z 2 { 1 , +1 } m⇥k . ( 11 ) Overall , we get the final objective of the proposed A2-NET model by considering Eq . ( 7 ) , Eq . ( 8 ) and113 Eq . ( 11 ) together as follows:114 min W , ⇥ L ( I ) = kX W > V 0k2F + kWX V 0k2F +↵kV 0V 0 > nIk2F + nX i=1 mX j=1 u > i zj kSij 2 , ( 12 ) where , ↵ and are hyper-parameters as the trade-off.115 In practice , uri g training , it might be only available a set of database points { vj } mj=1 without query116 points . Thus , we randomly sample n data points from database to construct a query set , and denote117 the indices of all the database points as with the indices of the query set as ⌦ . Additionally , because118 we can not back-propagate the gradient to ⇥ due to the sgn ( · ) function , we omit the sgn ( · ) function119 and only apply tanh ( · ) for relaxation in Eq . ( 10 ) of the whole optimization process . Therefore , the120 optimization formulation of A2-NET can be rewritten with only database points { vj } mj=1 for training121 as:122 min W , ⇥ L ( I ) = kX W > V 0k2F + kWX V 0k2F + ↵kV 0V 0 > nIk2F + X i2⌦ X j2 ⇣ tanh ( W · F ( Ii ; ⇥ ) ) > zj kSij ⌘2 . ( 13 ) For optimization , we employ the back-propagation algorithm and follow [ 8 ] to train the whole123 A2-NET m l in an end-to-end manner.124 3.4 Out-of-Sample Extension125 After training A2-NET , the learned model can be applied for generating binary codes for query points126 including unseen query points in th training phase . Specifically , we can use the following equation127 to generate the binary code for Iq:128 uq = sgn ( tanh ( W · F ( Iq ; ⇥ ) ) ) . ( 14 ) 4 Experiments129 In this section , we present the implementation details , empirical settings , main results , ablation studies130 and also some interesting qualitative a alyses of the learned attribute-aware binary hash codes.131 4.1 Datasets132 x0i By following ExchNet [ 4 ] , our experiments are conducted on five fine-grained benchmark datasets,133 i.e. , CUB-200-2011 [ 16 ] , Aircraft [ 12 ] , Food101 [ 1 ] , NABirds [ 15 ] and VegFru [ 6 ] . Specifically,134 CUB-200-2011 is one of the most popular used fine-grained datasets . It contains 11,788 bird images135 from 200 bird species and is officially split into 5,994 images for training with 5,794 images for136 test . Aircraft contains 10,000 images spanning 100 aircraft models with 3,334 for training , 3,333137 4 Fine-grained representation learning U ns up er vi se d at tr ib ut e- gu id ed lo ss ( S ec tio n 3 . 3 . 1 ) Feature decorrelation loss ( S ction 3.3.2 ) 3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET model . We laborate it in65 the foll wing four aspects , i.e. , unsupervised attribute-guided learning , attribute-specifi feature66 decorrelation , hash code learning and the optimization learning algorithm.67 3.3.1 Unsupervised Attribute-Guided Learning68 In real-applications , especially for the large-scale and fine-grained tasks , attribute annotations are69 always infeasibl , which limits the learning process to be conducted in an unsupervised setting.70 While , in the literature , the main goal of unsupervised learning is to capture regularities in data for the71 purp se of xtracting useful representations or for restoring corrupted data [ 13 ] . Many unsupervised72 methods explicitly produce internal latent units or codes , from which the data is to be reconstructed.73 Inspired by this , we develop an unsupervised attribute-guided learning component to project the74 holistic r presentation xi of Ii into a latent space , i.e. , the attribute space V . In V , its high-level75 vectors are designed to have certain desirable properties , e.g. , corresponding to semantic properties76 of fine-grained objects ( a.k.a . “ fine-grained attributes ” ) .77 More specifically , in our A2-NET , the unsupervised attribute-guided learning is realized by a recon-78 struction paradigm with an encoder-decoder structure , as shown in Figure ? ? . In concretely , given79 a batch of n training data Ii , the holistic representations X = { x1 ; x2 ; . . . ; xn } 2 Rd⇥n can be80 obtained as aforementioned . By formulation , the encoder projects X into the attribute space V with81 a projection matrix W 2 Rk⇥d to get an internal latent representation V 2 Rk⇥n w.r.t . X . In82 particularl , we set that the dimension of lat nt representation k quals to the number of hash bits in83 the final binary hash code ui . Furthermore , each column of V , i.e. , vi 2 Rk , can derive ui by84 ui = sgn ( tanh ( vi ) ) . ( 5 ) Meanwhile , regarding vi , we als consider to reconstruct its input xi by decoder as a counterpart85 of the encoder . Therefore , o one hand , such a reconstruction paradigm can reduce and further distill86 high-level semantic cues in the attribute space V . While , on the other hand , it can drive the training87 of A2-NET by preserving the similarity between queried hash codes and database points in terms of88 learning to hash ( cf . Section 3.3.3 ) .89 In concretely , the learning objective of unsupervised attribute-guided learning is written as follows:90 min W kX W > WXk2F s. . WX = V 0 = tanh ( V ) , ( 6 ) where the decoder ( i.e. , a counterpart of the encoder ) is realized by W > to simplify the network.91 Howev r , directly minimizing Eq . ( 6 ) with a hard constraint is hard to optimize . Therefore , we relax92 the constraint into a soft constraint , and the learning objective can be rewritten by93 min W kX W > V 0k2F + kWX V 0k2F . ( 7 ) 3.3.2 Attribute-Specific Feature Decorrelation94 By conducting the aforementioned u supervised attribute-guided learning , we can obtain the internal95 latent vectors V 0 as the attribute-specific features . In order to both enhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features , we introduce a feature97 d correlation constraint upon V 0 , which is formulated by98 min V 0 kV 0V 0 > nIk2F , ( 8 ) where I is the i entity matrix . Such a feature decorrelation constraint is preferable to construct99 independent features and reduce redundant information . Therefore , b sed on unsupervise attribute-100 guided learning and attribute-specific feature decorrelation , the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102 3.3.3 Hash Code Learning103 In the following , we conduct the hash code lear ing based on the obtained attribute-specific features.104 Assume that we have n query data points which are denoted as { qi } ni=1 , s well as m database points105 3 Hash code learning loss ( Section 3.3.3 ) Attribute-aware hash codes generating “ Red head ” “ On the tree ” “ Spotted body ” 2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section , we introduce the overall framework and notations of the proposed A2-NET model , as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29 3.1 Overall Framework and Notations30 As illustrated in Figure ? ? , our A2-NET model consists of two crucial modules , i.e. , a fine-grain d31 representation learning module and an attribute-aware hash codes generating module . Given an32 input image Ii , based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN , a set of attention guidance A = { Ac } is learned for capturing fine-grained tailored34 local patterns T ci from Ti . To distill semantical cues and further generate the final attribute-aware35 binary hash codes , we propose to transform these appearance-specific features T towards semantic-36 specific representations T̂ by performing a transform network ( · ) . After aggregating T̂ , the obtained37 attentive local-level features xci are associated with the global-level feature x global i to form as a38 holistic feature representation xi . In order to generate attribute-aware binary hash codes , we conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond o40 an attribute vector w.r.t . one kind of namable properties of fine-grained objects . Finally , hash code41 learning is performed upon vi to obtain the final attribute-a are binary codes ui.42 3.2 Fine-Grained Representation Learning43 Attention plays an important role in human perception [ 3 , 7 ] , and humans exploit a sequence of44 partial glimpses and s lective y focus on salient parts of an object or a s ene in order to bet er capture45 visual structure [ 10 ] . Inspired by this , we incorporat the attention mechanism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differ nces between these47 subordinate categories.48 In concretely , we extract the deep feature of its input image Ii via a backbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Then , based on Ti , C attention guidance Ac 2 RH⇥W is generated as a set of attention maps , i.e. , A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obtain the final attribu e-aware binary co es , it is54 desirable to transform these appearance-specific ( i.e. , low-level ) features T to semantic-specific ( i.e.,55 mid-level ) representations which are closer to the attribute space . Thus , a transforming network ( · ) ,56 which is equipped with a stack of convolution layers , is performed on T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ presents the para eters of the corresponding transfor ing networks w.r.t . T ci and Ti , respec-58 tively . Then , we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . The holistic60 feature representation w.r.t . the input image Ii is achieved by concatenating both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 R . Note that , we hereby abstract the aforementioned fine-grai ed62 feature learning process as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 Attention guidance 2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section , we introduce the overall framework and notations of the proposed A2-NET model , as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29 3.1 Overall Framework and Notations30 As illustrated in Figure ? ? , our A2-NET model consists of two crucial modules , i.e. , a fine-graine31 representation learning module and an attribute-aware hash codes generating module . Given an32 input image Ii , based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN , a set of attention guidance A = { Ac } is learned for capturing fi e-grained tailored34 local patterns T ci from Ti . To distill semantical cues and further generate the final attribute-aware35 binary hash c des , we propose to transform these appearance-specific features T towa ds semantic-36 specific representations T̂ by performing a transform network ( · ) . After aggregating T̂ , the obt ined37 attentive local-level features xci are associated with the global-level feature x global i to form as a38 holistic feature representation xi . In order to generate attribute-aware binary hash codes , we conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond to40 an attribute vector w.r.t . one kind of namable properties of fine-grained objects . Finally , hash code41 learning is performed upon vi o obtain the final attribute-aware binary codes ui.42 3.2 Fine-Grained Representation Learning43 Attention plays a imp rtant role in human perception [ 3 , 7 ] , and humans exploit a sequence of44 artial glimpses d selectively focus on salient parts of an object or a scene in or er to better capture45 visual structure [ 10 ] . Inspired by this , we incorporate the attention mechanism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48 In concretely , we extract the deep feature of its input image Ii via a backbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Then , based on Ti , C attention guidance Ac 2 RH⇥W is g nerated as a set of attention ma s , i.e. , A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obtain the final attribute-aware binary codes , it is54 desirable to transform these appearance-specific ( i.e. , low-level ) features T to semantic-specific ( i.e.,55 mid-level ) representations which are closer to the attribute space . Thus , a transforming network ( · ) ,56 which is equipped with a stack of convolution layers , is performed on T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ presents the parameters of the corresponding transforming networks w.r.t . T ci and Ti , respec-58 tively . Then , we aggregate T̂ ci and T̂i by conducting global average-pooling d correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . The holistic60 feature representation w.r.t . the input image Ii is achieved by concatenati g both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 Rd . Note that , we hereby abstract the aforementioned fine-grained62 feature learning process as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section , we introduce the overall framework and notations of the proposed A2-NET model , as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29 3.1 Overall Framework and Notations30 As illustrated in Figure ? ? , our A2-NET model consists of two crucial modules , i.e. , a fine-grained31 representation learning module and an attribute-aware hash codes generating module . Given an32 input image Ii , based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN , a set of attention guidance A = { Ac } is learned for capturing fine-grained tailored34 local patterns T ci from Ti . To distill semantical cues and further generate the final attribute-aware35 binary hash codes , we propose to transform these appearance-specific features T towards semantic-36 specific representations T̂ by performing a transform network ( · ) . After aggregating T̂ , the obtained37 attentive local-level features xci are associated with the global-level feature x global i to form as a38 holistic feature representation xi . In order to generate attribute-aware binary hash codes , we conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond to40 an attribute vector w.r.t . one kind of namable properties of fine-grained objects . Finally , hash code41 learning is performed upon vi to obtain the final attribute-aware binary codes ui.42 3.2 Fine-Grained Representation Learning43 Attention plays an important role in human perception [ 3 , 7 ] , and humans exploit a sequence of44 partial glimpses and selectively focus on salient parts of an object or a scene in order to better capture45 visual structure [ 10 ] . Inspired by this , we incorporate the attention mechanism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48 In concretely , we extract the deep feature of its input image Ii via a backbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Then , based on Ti , C attention guidance Ac 2 RH⇥W is generated as a set of attention maps , i.e. , A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obtain the final attribute-aware binary codes , it is54 desirable to transform these appearance-specific ( i.e. , low-level ) f atures T to semantic-specific ( i.e.,55 mid-level ) representations which are closer to the attrib te space . Thus , a transforming n twork ( · ) ,56 which is equipped with a stack of convolution layers , is performed on T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ presents the parameters of the corresponding transfor ing networks w.r.t . T ci and Ti , respec-58 tively . Then , we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x glo al i . Th holistic60 feature representation w.r.t . the input image Ii is achieved by concatenating both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 Rd . Note that , we hereby abstract the aforementioned fine-grained62 feature learning process as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section , we introduce the overall framework and notations of the proposed A2-NET model , as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29 3.1 Overall Framework and Notations30 As illustrated in Figure ? ? , our A2-NET model consists of two crucial modules , i.e. , a fine-grai d31 representation learning module and an attribute-aware hash codes g nerating modul . Given an32 input image Ii , based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN , a set of attention guidance A = { Ac } is learned for capturing fine-grained tailored34 local patterns T ci from Ti . To distill semantical cues and further generate the final attribute-aware35 bi ary ash c des , we propose to transform these appearance-specific features T towards semantic-36 specific representations T̂ by perform ng a transform network ( · ) . After aggregating T̂ , th obtained37 attentive local-level features xci are associated with the global-level f ature x global i to form as a38 holistic feature representation xi . In order to generate attribute-aware binary hash codes , we conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond to40 an attribute vector w.r.t . one kind of namable properties of fine-grained objects . Finally , hash code41 learning is performed upon vi to obtain the final attribute-aware binary codes ui.42 3.2 Fine-Grained Representation Learning43 Attention plays an important role in human perception [ 3 , 7 ] , and humans exploit a sequence of44 partial glimpses and selectively focus on salient parts of an object or a scene in order to b ter capture45 visu l structure [ 10 ] . I spired by this , we incorporate th attention mech nism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48 In concretely , we extra t the deep feature of its input image Ii via a backbon CNN model CNN ( · ) 49 s50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Then , based on Ti , C attention guidance Ac 2 RH⇥W is generated as a set of attention maps , i.e. , A.51 The attention guidance Ac is designed to ev luate which deep descript rs should be attended or ven52 ov rlooked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obtain the final attribute- ware binary codes , it is54 desirable to transform these appearance-specific ( i.e. , low-level ) features T to semantic-specific ( i.e.,55 mid-level ) representations which are closer to the attribute space . Thus , a transforming network ( · ) ,56 which is equipped with a stack of convolution layers , is performed on T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ pr sents the parameters of the corresponding tr nsforming networks w.r.t . T ci nd Ti , respec-58 tively . Then , we aggregate T̂ ci and T̂i y conducting global average-pooling and cor espo dingly59 obtain the atte tive local-level f atures xci , as well as the global-level fe tur x global i . The holisti60 fe ture representation w.r.t . the input image Ii is achieved b concatenating both xci and xgl bali , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 Rd . Note hat , we hereby abstract the aforementioned fin -grained62 featur learning process as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 … 2 Related Work23 2.1 Fine-Grained Imag Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this secti n , we int oduce the overall framew rk and otations of the p oposed A2-NET model , as28 well as elaborating the detailed modules of A2-NET a its corresp nding ptimiza ion algorithm.29 3.1 Overall Framework and No ations30 As illustrated in Figure ? ? , our A2-NET model consists of two crucial modules , i.e. , a fine-grained31 representation learning module and an attribute-aware hash codes generating module . Given an32 input image Ii , based n its corr po ding deep activa io ten o Ti 2 RC⇥H⇥W extracted by a33 backbone CNN , a set of attention guida ce A = { Ac } i learned for capt ring fine-grained t ilored34 local patterns T ci from Ti . To distill semantical cues and further generate the final attribute-aware35 bi ary has c des , we pro ose to transfo m these ppearan e-specific f atures T owards semantic-36 sp cific representations T̂ by performing a transform etwork ( · ) . Aft r aggre ating T̂ , h obtained37 attentive local-level features xi ar associated with t e gl bal-level feature x global i to form as a38 holistic feature representation xi . In order to ge erate attribute-aware binary hash codes , we conduct39 a reconstructing paradigm to projec xi a vi in an attribute space wh re ts data point correspond to40 an attribute vector w.r.t . one kind of namable properties of fine-grained obj cts . Finally , hash code41 learning is performed upon vi to obtain the final attribute-aware binary codes ui.42 3.2 Fine-Grained Representation Learning43 Att ntion plays an important role in hu an perception [ 3 , 7 ] , nd humans exploit a sequenc of44 partial glimpses and selectively focus on salient parts of an object or a scene in order to better capture45 visual structure [ 10 ] . Inspired by this , we incorp rate the attentio mechanism into represent tion46 lear ing to capture fine-grain d local patt rns for distinguishing subtle differences between these47 subordinate categories.48 In concretely , we extract the deep feature of its input image Ii via a backbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Then , based on Ti , C attention guida c Ac 2 RH⇥W is gen rat d as set of attention maps , i.e. , A.51 The ttent on guidance Ac s designed to ev luate which deep descriptors should be attended or even52 overl ked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obt in the final attribute-aware binary codes , it is54 desirable to transform these appeara ce-specific ( i.e. , low-level ) features T to semantic-specific ( i.e.,55 mid-l vel ) repres ntations which are closer to the attribute space . Thus , a transforming network ( · ) ,56 which is equipped with a stack of convolution layers , is performed on T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ prese ts the parameters of the corresponding transforming networks w.r.t . T ci and Ti , respec-58 tively . Then , we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentiv local-level f atures xci , as well as the global-level feature x global i . Th holistic60 feature representation w.r.t . the input image Ii is achieved by concatenating both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 Rd . Note that , we hereby abstract the aforementioned fine-grained62 feature learning process as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section , we introduce the overall framework and notations of the proposed A2-NET model , as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29 3.1 Overall Framework and Notations30 As illustrated in Figure ? ? , our A2-NET model consists of two crucial modules , i.e. , a fine-grained31 representation learning module and an attribute-aware hash codes generating module . Given an32 input image Ii , based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN , a set of attention guidance A = { Ac } is learned for capturing fine-grained tailored34 local patterns T ci from Ti . To distill semanti al cu s d further generate the final attribute-aware35 binary hash codes , we propose to transform these appearance-specific features T towards semantic-36 specific representations T̂ by performing a transform network ( · ) . After aggregating T̂ , the obtained37 attentive local-level features xci are associated with the global-level feature x global i to form as a38 holistic feature representation xi . In order to generate attribute-aware binary hash codes , we conduct39 a r constructing paradigm to project xi as vi in an attribute sp ce where its data point correspond to40 an attribute vector w.r.t . one kind of namable properties of fine-grained objects . Finally , hash code41 lear ing is perfor ed upon vi to obtain the final attribute-aware binary codes ui.42 3.2 Fine-Grained Representation Learning43 Attention plays an important role in human percept on [ 3 , 7 ] , and humans exploit a sequence of44 partial glimpses and selectively focus on salient parts of an object or a scene in order to better capture45 visual structure [ 10 ] . Inspired by this , we inc rporate t e attention mechanism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48 In concretely , we extract the deep feature f its i put im ge Ii via a ba kbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Then , based on Ti , C attention guidance Ac 2 RH⇥W is generated as a set of ttention maps , i.e. , A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obtain the final attribut -aware binary codes , it is54 desirable to tran for these appearance-specific ( i.e. , low-level ) features T to semantic-spec fic ( i.e.,55 mid-level ) representations which are closer to the attribute space . Thus , a transforming network ( · ) ,56 which is equipped with a stack of convolution layers , is performed on T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ pr sents the parameters of he c rrespo ding transforming networks w.r.t . T ci a d Ti , respec-58 tively . Then , we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . Th holistic60 feature rep esentation w.r.t . the input image Ii i achieved by concat ati g both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 Rd . Note that , we hereby abstract the aforem ntioned fine-grained62 f ature le ning pro ess as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 Aggregation … 2 Related Work23 2.1 Fine-Grained Image Retri val24 2.2 Le rning to Hash25 2.3 Visual Attributes26 3 Method logy27 In this section , we introduce the ove ll framework nd notati ns of the propos A2-NET model , as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29 3.1 Over l Fra ework and Notations30 As illustrated in Figur ? ? , our A2-NET model consist of two crucial modules , i.e. , a fin -gr ined31 representation learning module and a attribute-aware hash c des ge erating mo ul . Given a32 input image Ii , based on its corres onding de p activatio tens r Ti 2 RC⇥H⇥W extracted by a33 back o e CNN , a set of ttention guidance A = { Ac } is learn d f capturing fine-grained tailored34 local patterns T ci from Ti . To distill semantical cues and further gen rate the fi al attribute-aware35 binary hash cod s , we propose to tr nsform these ppearance-specific features T towards semantic-36 specific repres ntations T̂ by performing a transform network ( · ) . After aggregating T̂ , the obtained37 attentive local-level features xci are associated with the global-level featur x global i to form as a38 holistic e ture r pr s ntation xi . In order to g nerate attribute-aware binary hash cod s , we conduct39 a econstructing p adigm to project xi as vi i an attribute space wh re it dat point correspond to40 an ttribute tor w.r.t . o e kind f n m ble properties of fin -grained objects . Finally , hash code41 le rning is p rformed pon vi to obtain th final attribute- ware bi ary codes ui.42 3.2 Fi e-Grained Representation Le rning43 Attention plays an important role in human perception [ 3 , 7 ] , and humans exploit a sequence of44 partial glimpses and selectively focus on sali nt parts of an object or a scene in or e to b tt r capture45 visual structur [ 10 ] . Inspired by this , we incorporate the attention mechanism into representation46 learning to capture fin -grained local patterns for distinguishing subtle differences between these47 s bordi te tegories.48 In concretely , we extract the deep feature of its input image Ii via a backbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Th n , based o i , C attention uidan e Ac 2 RH⇥W is g nerated as a set f att ntion m ps , i.e. , A.51 The atte ion i ance Ac is designed to evaluate which deep descriptors should b attended or even52 overlooked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obtain the final attribute-aware binary codes , it is54 desirable to tran form these appearance-specific ( i.e. , low-level ) features T to semantic-specific ( i.e.,55 mid-level ) representations whi h are closer to the attribute space . Thu , a transforming network ( · ) ,56 which is equipped with a stack of convolution layers , is performed on T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) wher ✓ presents the parameters of the corresponding transforming networks w.r.t . T ci and Ti , respec-58 tively . Then , we ggrega e T̂ ci and T̂i by c n ucting global average-pooling and correspondingly59 o tain the attentive local-level featur s xci , as well a the global-l vel feature x global i . The holistic60 feature representation w.r.t . the i put mage Ii is chi ved by concatenating both xci and xglobali , i.e.,61 xi = h xci ; l al = F ( Ii ; ⇥ ) 2 Rd . Note that , we hereby abstract the aforementioned fine-grained62 feature learnin proc ss as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 2 Rel t d Work23 2.1 Fi -Grained Image Retrieval24 2.2 Learni g to Ha h25 2.3 Visu l Attributes26 3 Methodology27 In this section , we introduce the overall framework and notations of the proposed A2-NET model , as28 w ll as elaborating the detailed modules of A2-NET and its correspon ing optimization algorithm.29 3.1 Overall Fram work and Notation30 As illustrated in Figur ? ? , our A2-NET model consists of two cruc al modules , i.e. , a fi e-grained31 r p tatio l arning modul nd an attribut - ware has codes generating module . Given an32 inp t image Ii , b sed o its respond ng deep a tivation tensor Ti 2 RC⇥H⇥W extracted by a33 ba kbone CNN , a set of attention g idance A = { Ac } is learned for capturing fine-grained tailored34 l l patterns T ci from Ti . To distill semantical cues and further en rate the final attribute-aware35 binary hash odes , we prop se to transform these ppearance-specific features T towards semantic-36 specific representations T̂ by performing a transform netwo k ( · ) . After aggregating T̂ , he obt ined37 attentive local-level features xci re associated with the global-level feature x global i to form as a38 holistic feature representation xi . In order to generate attribute-aware binary hash codes , w conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond to40 an attribute vector w.r.t . one kind of namable properties of fine-grained objects . Finally , has c de41 learning is performed upon vi to obtain the fi al attribute-aware binary codes ui.42 3.2 Fine-Grained Representation Learning43 Attention plays a important role in human perception [ 3 , 7 ] , and humans exploit a sequence of44 partial glimpses and select vely focus on salient parts of an bject or a scene in order to be ter capture45 visual structure [ 10 ] . Inspired by this , we incorporate the attention mechanism into representation46 learning to c pture fine-grained local patterns for distinguishing subtle differences b tween th se47 subordinate c tegories.48 In concretely , we extract the deep feature of its input image Ii via a backbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Then , based on Ti , C attention guidance Ac 2 RH⇥W is generat d as a set of attention maps , i.e. , A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be tte d d or even52 overlooked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obtain the final attribute-aware binary codes , it is54 desirable to transform these appearance-specific ( i.e. , low-level ) features T to semantic-specific ( i.e.,55 mid-level ) representations which are closer to the attribute space . T us , transf rming netw rk ( · ) ,56 which is equipped with a stack of convolution layers , is performed on T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ presents the parameters of the corresponding transformi g networks w.r.t . T ci and Ti , r spec-58 tively . Then , we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . The holistic60 feature representation w.r.t . the input image Ii is achieved by concatenating both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 Rd . Note that , we her by abstr c the af rementioned fine-gr ined62 feature learning process as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section , we introduce the overall framework and notations of the proposed A2-NET model , as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorith .29 3.1 Overall Framework and Notations30 As illustrated in Figure ? ? , our A2-NET model cons sts of two crucial modules , i.e. , a fine-grain d31 representation learning module and an attribute-aware hash codes gene ating module . Given an32 input image Ii , based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN , a set of attention guidance A = { Ac } is learned for capturing fin -grained tail red34 local patterns T ci from Ti . To distill semantical cues and further generat the final at ribute-aware35 binary hash codes , we propose to transfor th se appearance-specific features T towards semantic-36 specific representations T̂ by performing a transform network ( · ) . Aft r aggregating T̂ , the obtained37 attentive local-level features xci are asso iated with the global-level feature x global i o form as a38 holistic featur representation xi . In ord r to generate attribute-awar binary hash codes , we conduct39 a reconstructing paradigm to project xi as vi in an attribute space wh re its data point corr spond to40 an attribute vector w.r.t . one kind of namable properties of fine-grained objects . Finally , hash code41 learning is performed upon vi to obtain the final attribute-aware binary codes ui.42 3.2 Fine-Grain d Representation Learning43 Attention plays an important role in human perception [ 3 , 7 ] , and humans exploit a sequence of44 partial glimps s and s lectively focus on s lient parts of an object or a scene in order to better capture45 visual structure [ 10 ] . Inspired by this , we incorporate the attention mechanism into repr sentation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48 In concretely , we extract the deep feature of its nput image Ii via a backbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Then , based on Ti , C attention guidance Ac 2 RH⇥W is generated as a set of attention maps , i.e. , A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadama d product . T obtain he final a tribute-aware binary cod s , it is54 desirable to transform these appearance-specific ( i.e. , low-level ) features T to semantic-specific ( i.e.,55 mid-level ) representations which are closer to the attribute space . Thus , a transforming network ( · ) ,56 which is equipped with a stack of convolution layers , is performed on T as follows:57 T̂ ci = ( i ; local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ presents the parameters of the corresponding transforming networks w.r.t . T ci and Ti , respec-58 tively . Then , we aggregate T̂ ci and T̂i by onducting glob l average-poolin and correspondingly59 obtain the attentive local-level features xci , as w ll as the gl bal-level feature x global i . Th holistic60 feature representation w.r.t . the input image Ii is achiev d by concat nating both xci a d xglobali , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 Rd . Note that , we hereby abstract the aforementioned fine-grained62 feature learning process as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 Aggregation where t e d coder ( i.e. , a counterpart of the encoder ) is realized by W > to simplify the netwo k.91 However , directly minimizing Eq . ( 6 ) with a hard constraint is hard to optimize . Therefore , we relax92 the constraint into a soft constraint , and the learning objective can be rewritten by93 min W kX W > V 0k2F + kWX V 0k2F . ( 7 ) 3.3.2 Attribute-Specifi F ur Decorrelation94 By conducting the aforementioned unsupervised attribut -guided learning , we can obtain the internal95 l tent vectors V 0 as the attribute-specific features . I order to both enhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features , we introduce a feature97 decorrelation c n r int upon V 0 , which is formulat d by98 mi V 0 kV 0V 0 > nIk2F , ( 8 ) where I is the identity matrix . Such a feature ecorrelation co straint is pref rable to construct99 independent features and reduce redundant information . Therefore , based on unsupervised attribute-100 guided learning and attribute-specific feature decorrelation , the final learned hash codes are expected101 to be bo h attribute-aware and hash-bit ind pend nt.102 3.3.3 H sh Co e L arning103 In the f llowing , we conduct the hash code learning based on the obtained attribute-specific features.104 Assume that w have n qu ry data p ints which are denoted as { qi } i=1 , as w ll a m database points105 which a noted as { vj } j=1 . Note hat , both qi and vi are belo gin to the attribute space V . Then,106 the corresponding binary has codes can be obt ined via107 ui = sgn ( tanh ( qi ) ) , ( 9 ) zj = sgn ( tanh ( vj ) ) , ( 10 ) where ui , zj 2 { 1 , +1 } k. The goal of our hash code learning is to learn binary hash codes for both108 qu ry poin s and database points from { qi } ni=1 , { vj } mj=1 , and the pairwise supervised information,109 i.e. , S 2 { 1 , +1 } n m. To pr se ve the pairwise similarity , adopt the ` 2 loss between the110 supervised inf rmati ( .k.a . similarity ) d t inner product f qu ry-database point binary code111 pairs . It can be formulated as follows:112 minU , Z Pn i=1 Pm j=1 u > i zj kSij 2 s.t . U 2 { 1 , +1 } n⇥k , Z 2 { 1 , +1 } m⇥k . ( 11 ) Overall , e g t the fina obj tiv of the propo ed A2-NET model by considering Eq . ( 7 ) , Eq . ( 8 ) and113 Eq . ( 11 ) toge a follows:114 min W , ⇥ L ( I ) = kX W > V 0k2F + kWX V 0k2F +↵kV 0V 0 > nIk2F + nX i=1 mX j=1 > i zj kSij 2 , ( 12 ) where , ↵ and are hyper-parameters as the trade-off.115 In practice , during training , it might be only available a set of database poin s { vj } mj=1 without query116 points . Thus , we randomly sample n data points from database to construct a query set , and denote117 the indices of all the database points as with the indices of the query set as ⌦ . Additionally , because118 we can not back-propagate the gradient to ⇥ due to the sgn ( · ) function , we omit the sgn ( · ) function119 and o ly apply t nh ( · ) for elaxation in Eq . ( 10 ) of the whole optimization process . Therefore , the120 optimization formulation of A2-NET can be rewritten with only database points { vj } mj=1 for training121 as:122 min W , ⇥ L ( I ) = kX W > V 0k2F + kWX V 0k2F + ↵kV 0V 0 > nIk2F + X i2⌦ X j2 ⇣ tanh ( W · F ( Ii ; ⇥ ) ) > zj kSij ⌘2 . ( 13 ) For optimization , our A2-NET does not require co plicated two-stage learning algorithms , e.g. , the123 alternative optimization strategy . We employ the back-propagation algorithm and follow [ 8 ] to train124 the whol A2-NET model in a u ified end-to-end manner.125 4 “ Blue he d ” Figure 1 : Overall framewo k of the proposed A2-NET model , which consists of two crucial modules , i.e. , fine-grained representation learning and attribute-aware hash codes generating . The whole netw rk c n be end-to-e d trainabl , d is generally riven by the nsupervi ed attribute-guided loss , the feature decorrelation loss and he hash cod learn g loss , cf . S ction 3.3. has th potenti l to help with large-sc le fine- ra ed d ta t o . Therefore , targeting the ig data35 challenge , fine-grained hashing [ 7 , 13 ] is a promisi g direction orthy of further explorations.36 low-level to mid-level appearance-specific to se antic-specific37 obtain corresponding fine-grai ed obj ct attributes ( e.g. , xx ) , without significant annotation cost38 2 Relat d Work39 In this section , we briefly review t e r lated work i the following t ree aspects , i.e. , n -g ained40 image retrieval , learning to hash and the studies about visual attributes in computer vision problems.41 2.1 Fine-Grained Image Retrieval42 Fine-grained image retrieval as an integral part of fine-grained image analysis [ 34 ] has gained more43 and more tractio i recent years [ 7 , 20 , 23 , 33 , 35 , 37 , 38 ] . What makes it challenging is that44 objects of fine-grained images have only subtle differences , and often largely vary in pose , scale , and45 orientation or can exhibit cross-modal differences ( e.g. , sketch-based retrieval [ 20 ] ) .46 Depending on the type of query image , the most studied areas of fine-grained image retrieval47 can be separated into two groups : fine-grained content-based image retrieval ( FG-CBIR ) and fine-48 grained sketch-based image retrieval ( FG-SBIR ) . More specifically , in FG-CBIR , unsupervised49 learning based [ 33 ] and supervised learning based methods [ 36–38 ] were developed from different50 perspectives for dealing with fine-grained retrieval tasks , e.g. , localizing fine-grained parts [ 33 ] ,51 enhancing intra-class separability with inter-class compactness [ 38 ] , and reducing the confidence52 of the fine-grained predictions [ 36 ] , etc . While , FG-SBIR needs to not only capture fine-grained53 characteristics present in the sketches , but also possess the ability to traverse the sketch and image54 domain gap . In the literature of FG-SBIR , the earlier works , e.g. , [ 17 , 27 , 35 ] , were mostly based55 on Siamese-triplet networks [ 3 ] to tackle the aforementioned challenges . Recently , some works56 tried to incorporate the advances of recent progress in self-supervised learning [ 5 ] and attention57 mechanisms [ 6 ] for further improving the retrieval accuracy of FG-SBIR , e.g. , [ 23 , 25 ] .58 However , although these fine-grained retrieval methods achieved good results , they still show the59 limitations in the face of large-scale data , i.e. , the searching time for exact nearest neighbor is typically60 expensive or even impossible for the given queries . To alleviate this issue , fine-grained hashing , which61 aims to generate compact binary codes to represent fine-grained images , as a promising direction62 has attracted the attention in the fine-grained community very recently [ 7 , 14 ] . More specifically,63 2 R d_head ? Blue_tail ? On_ he_ r e ? Spotted_body ? YES NO YES YES 1 0 1 1 Att ibute-aware hash code of Retrieval Top-k retrieved images 1 0 1 1 1 0 1 1 1 0 1 0 Query image 101 1 Encoder 3.3 ttribute-Aware Hash Codes G nerating64 How to generate attribute-aware hash codes is the key of our A2-NET model . We elaborate it in65 the following four aspects , i.e. , u supervised attribute-guided learning , attribute-specifi feature66 decorrelation , hash code learning and the optimization learning algorithm.67 3.3.1 Unsupervised Attribute-Guided Le rning68 In r al-applications , especially for the larg -scale and fine-grained tasks , attribut annotations are69 always infeasible , which limits the learning process to be conducted in an unsupervised setting.70 While , in the literature , th main goal of nsupervised learning is to capture regularities in data for the71 purpose of xtracting useful rep es n tio s or for restoring c rupted data [ 4 ] . Many unsupervi ed72 methods explicitly produce int rnal lat t units or codes , from which the data is to be reconstructed.73 Inspired by this , we develop an unsu rvised attribute-guided learni component to proj ct the74 holistic representation xi of Ii into a latent space , i.e. , the attribute space V . In V , its high-level75 v ctors are designed to have certain desirable properties , e.g. , corresp nding to semantic properti s76 of fine-grained objects ( a.k.a . “ fine-grained attributes ” ) .77 More specifically , in our A2-NET , the unsupervised attribute-guided learning is realized by a recon-78 struction paradigm with n encoder-de oder structure , as shown in Figure ? ? . In concretely , given79 a batch of n tr ining data Ii , the holis ic representations X = { x1 ; x2 ; . . . ; xn } 2 Rd⇥n can b80 btained as aforemention d. By formulation , the encoder projects X into the attribute space V with81 a projection matrix W 2 Rk⇥d to get an internal latent representation V 2 Rk⇥n w.r.t . X . In82 particul rly , we set tha the dimension of latent representation k equals to th number of hash bits in83 the final binary hash code ui . Furt rmore , each lumn of V , i . . , vi 2 Rk , can derive ui by84 ui = sgn ( tan ( vi ) ) . ( 5 ) Meanwhile , regarding vi , we also consider to reconstruct its input xi by a decoder as a counterpart85 of the encoder . Therefore , o one hand , such a reconstruction paradigm can reduce and further distil86 high-lev l semantic cues in the attribute space V . While , on the other ha d , it can drive the training87 of A2-NET by preserving the similarity between queried hash codes and database points in terms of88 learning to hash ( cf . Section 3.3.3 ) .89 In concretel , the learning objective of unsuperv sed attribut -guided l arning is written as follows:90 min W kX W > WXk2F s.t . WX = V 0 = tanh ( V ) , ( 6 ) where the decoder ( i.e. , a counterpart of the encoder ) is realized by W > to simplify the network.91 However , directly minimizing Eq . ( 6 ) with hard constraint is hard to ptimize . Therefore , we r lax92 the constraint into a soft constraint , and the lear ing objective can be rewritten by93 min W kX W > V 0k2F + kWX V 0k2F . ( 7 ) 3 . .2 Att ibute-Specific Fe tu e Decorrel ti n94 By conducting the aforementioned unsupervised attribut -guided le rning , we can obtain the internal95 latent vectors V 0 as the attribute-specific features . In order to both enhance the discriminative ability96 and re ove the redundant correl tion amo g these attribute-specific features , we introduce a feat r97 correlation co straint upo V 0 , which is formulated by98 min V 0 kV 0V 0 > kIk2F , ( 8 ) whe e I is the identity matrix . Such feature decorr lation constraint is pref rable to construct99 independent features and reduce r dundant information . Therefore , based on un upervised attribu -100 guided learning and attribute-specific feature decorrelation , the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102 3.3.3 Hash Cod Le rning103 In the following , we conduct the hash code learning based on the obtained attribute-specific features.104 Assume that we have n query data points which are denoted as { qi } n=1 , as well as m database points105 3 3.3 Attribute-Aware Hash Codes Ge rating64 How to generate attribute-aware hash codes is he key of our A2-NET model . We elaborate it in65 the f llowing four aspects , i.e. , unsupervised attribute-guided learning , a tribute-spec fic feature66 decorrelation , hash code learning and the optimizatio le rning algorithm.67 3 . .1 Unsupervised Attr bute-Guided Learning68 In real-applica ions , especially for the larg -scal and fine-grained tasks , attribute annotations are69 always infeasible , which limits th learnin p ocess to be conducted in an unsupervised setting.70 While , in the litera ure , the main goal of unsuperv sed learning is to capture regu arities in data for the71 purpose of extracting us ful representations or for restoring corrupted data [ 4 ] . Many unsupervised72 meth ds explicitly produce internal latent units or codes , from which the data is to be reconstr ct d.73 Inspired by this , we develop an unsupervised attribute-guided learning component to project the74 holistic rep sentation xi of Ii int a latent space , i.e. , the attribute sp ce V . In V , i s high-level75 vectors are designed to have certain desirable properties , e.g. , corresponding to semantic properties76 of fine-grained objects ( a.k.a . “ fi e-gr ined attribute ” ) .77 More specifically , in our A2-NET , the unsupervis d attribute-gu d learning is realized by a recon-78 struction paradigm with an encoder-decoder structure , as shown in Figure ? ? . In concretely , given79 batch of n training data Ii , the holistic representations X = { x1 ; x2 ; . . . ; xn } 2 Rd⇥n can be80 obtained as aforementioned . By formulati n , th en oder pr jects X into the attribute sp ce V with81 a projection matrix W 2 Rk⇥d to e an internal l tent repre entation V 2 Rk⇥ w. .t . X . I82 parti ularly , we set that the dimension of latent representation k equals to the nu ber of hash bit in83 the final binary hash co e ui . Furthermore , ach column of V , i.e. , vi 2 Rk , ca d rive ui by84 ui = sgn ( tanh ( vi ) ) . ( 5 ) Meanw ile , regarding vi , we also consider to reconstruct i s input xi by a decoder s a counterpart85 of the encoder . Therefore , on one h nd , such a reconstructi n para igm can redu e and furth r distill86 high-level semantic cues in the attribute space V . While , on the other h nd , it can driv t e training87 of A2-NET by preserving the similarity between queried hash cod s and database points in terms of88 learning to hash ( cf . S ction 3.3 3 ) 89 In co cretely , the learning objective of unsupervised attribute-guided learning is written as foll ws:90 min W kX W > WXk2F s.t . WX = V 0 = tanh ( V ) , ( 6 ) where the decoder ( i.e. , a counterpart of the encoder ) is realized by W > to simplify the network.91 How ver , directly minimizing Eq . ( 6 ) with a ha d constraint i hard t optimize . Therefore , we relax92 the constraint into soft constraint , and the lear ing objective can be rewritten by93 min W kX W > V 0k2F + kWX V 0k2F . ( 7 ) 3 . .2 Attribute-Specific Feature Decorrelation94 By conducting the aforementioned unsupervised attribute-guided learning , we can obtain the internal95 latent vectors V 0 as the attribute-specific features . In order to bo nhance th discri inative ab lity96 and remov the r dundant cor elation amo thes ttribute-specific features , we introduce a feature97 decorrelation constraint up V 0 , which is formulated by98 mi V 0 kV 0V 0 > kIk2F , ( 8 ) where I is the i entity matrix . Such a f ature d correlation constraint is prefer ble to construct99 independent features and reduce redundant information . Therefore , based on unsupervised attribute-100 guided learning and attribute-specific feature decorrelation , the final learned hash codes are expected101 to be both attribute-aware and hash- it independent.102 3.3.3 Has Co e Learning103 In the following , we conduct the hash code learning based on the obtained attribute-specific features.104 Assume that we h ve n query dat poi ts which are denoted as { qi } ni=1 , as well as m database points105 3 3.3 Attribute-Aware Hash C s Gene ating64 How to generate attribute-aware hash codes is the key of our A2-NET model . We elaborate it in65 the following fou asp c s , .e. , unsup rvis attribute-guided le rni g , at ribute-specific feature66 decorrelati n , h sh code learning and the o timization earni g algorithm.67 3.3.1 Unsupervised Attr bute-Gui ed Learn ng68 In real- pplications , specially for the large-scale and fine-grai ed tasks , attribute annotations are69 always infeasible , which limits t l arning process to be conducted in an nsupervised setti g.70 While , in the literature , the main goal of unsupervised learn ng is to capture regularities in d ta for th71 purpose of extracting useful representations or for restoring corrupted data [ 4 ] . Many unsupervised72 methods explicitly rodu e internal latent units or codes , from which the data is to e reconstructed.73 Inspired by this , we develop an unsupervised attribute-guided learning component to project the74 holistic prese tation xi of Ii into a latent space , i.e. , the attribute space V . I V , its high-level75 vectors are designed to have certain desirable properties , e.g. , corres onding to semantic properties76 of fine-grained jects ( a.k.a . “ fine-grained attribut s ” ) .77 More specifically , in our A2-NET , the unsupervised attribute-guided learning is realized by a recon-78 structi n paradigm with an encoder-decod r structure , as shown in Figure ? ? . In concretely , given79 a batch of n trai ing data Ii , the h listic representations X = { x1 ; x2 ; . . . ; xn } 2 Rd⇥n ca be80 obtained as aforementio ed . By formulation , the encoder pr jects X into the attribute sp ce V with81 a projectio mat ix W 2 Rk⇥d to get an nternal latent repres ntation V 2 Rk⇥n w.r.t . X . I82 particularly , w set that the dimension of lat nt repre entation k equals t the number of hash bits in83 the final binary ash c e i . Furthermore , each c lumn of V , i.e. , vi 2 Rk , c n deriv ui b84 ui = sgn ( tanh ( vi ) ) . ( 5 ) Meanwhile , regarding vi , we also consider to r co st uct its input xi by a decoder as a counterpart85 of the encod r. Therefore , on one hand , uch a reconstruction paradigm can duce and further distill86 high-level semantic cues in the attribute space V . While , on the other hand , it can drive the training87 of A2-NET by preserving the similarity between queried hash codes and database points in terms of88 learning to hash ( cf . Section 3.3.3 ) .89 In con retely , the learn ng obj ctive of u sup rvised at ribute-guided learning is written as foll ws:90 min W kX W > WXk2F .t . WX = V 0 = tanh ( V ) , ( 6 ) where the decoder ( i.e. , a counterpart f he encoder ) is r alized y W > to simplify the network.91 However , directly minimizing Eq . ( 6 ) with a hard con traint is hard to optimize . Therefore , we re ax92 the c nstraint into a soft co straint , and the learning objectiv can be re ritt n by93 min W kX W > V 0k2F + kWX V 0k2F . ( 7 ) 3.3.2 Attribute-Specific Feature Decorrela ion94 By conducting the aforementioned unsupervised attr bute-gu ded learning , we can obtain the inte nal95 l tent v ctors V 0 as th ttribute-specific features . In order to both enhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features , we introduce a feature97 decorrelation constraint upon V 0 , which is formulated by98 min V 0 kV 0V 0 > kIk2F , ( 8 ) where I is the identity matrix . Such a feature decorrelation constraint is preferable to construct99 independent features and reduce redundant information . Therefore , based on unsupervised attribute-100 guided learning and attribute-specific feature decorrelation , the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102 3.3.3 Hash Code Learni g103 In the following , we c nduct the hash code learning base n the btained attribute-spe ific features.104 Assume that we ve query ata points which are denoted as { qi } ni=1 , as well as m database points105 3 3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET model . We elaborat it in65 the following four aspects , i.e. , unsupervised attribute-guided learning , attribute-specific feature66 decorrel tion , h sh code learning and the optimization learning algorithm.67 3.3.1 Unsupe vised At ribute-Guided Learnin68 In real-applications , specially for the large-sca e and fine-gr ined tasks , attribute an tation re69 always infeasible , which limits the learning process to be nducted in an unsuperv sed setting.70 While , i the literature , the ma goal of unsupervis d learning is to capture regularities in data for the71 purp se of extracting useful representations or f r restoring corrupted data [ 4 ] . Many unsupervise72 methods explicitly oduce int r al latent units or codes , from which the d ta is to be reconstructed.73 Inspired by this , we develop an unsupervised attribute-guided learning component to project the74 holistic representation xi of Ii into a latent space , i.e. , the attribute sp ce V . In V , its high-lev l75 vectors are designed to have certain desirable properti s , .g. , corresponding to semantic properties76 of fine-grained objects ( a.k.a . “ fine-grained attributes ” ) .77 More specifically , in our A2-NET , the unsupervis attribute-guided l arn ng is realized by a recon-78 structio paradigm with an enco er-decoder structure , as shown in Figure ? ? . I concretely , given79 batch f n training data Ii , the holistic representatio s X = { x1 ; x2 ; . . . ; xn } 2 Rd⇥n ca be80 obtained as forem ntioned . By formulation , the encoder projects X into the attribute space V with81 a pr jection matrix W 2 Rk⇥d to get an intern l l tent represent on V 2 Rk⇥ w.r.t . X . In82 particularly , we set that the dimension of latent representation k equals t the number of hash bits in83 the final binary hash code ui . Furthermore , each column of V , i.e. , vi 2 Rk , can derive ui by84 ui = sgn ( tanh ( vi ) ) . ( 5 ) Meanwhile , regarding vi , w also consider to reconstruct its input xi by a decoder s a counterpart85 of the encoder . Therefore , on one hand , such a reconstruction paradigm can reduce and further distill86 high-level semantic cues in the attribute space V . While , on th other hand , it can drive the training87 of A2-NET by preserving the similarity between qu ried hash codes and database oints in terms of88 learning to hash ( cf . Section 3.3.3 ) .89 In concretely , the learning objective f unsupervis d attribut -guided learning is written s follows:90 min W kX W > WXk2F s.t . WX = V 0 = tanh ( V ) , ( 6 ) where the decoder ( i.e. , a counterpart of the encoder ) is realized by W > to simplify the network.91 However , directly minimizing Eq . ( 6 ) with a hard constraint is hard to optimize . Therefore , we relax92 the constraint into a soft constraint , and the learning objective ca be rewritten by93 min W kX W > V 0k2F + kWX V 0k2F . ( 7 ) 3.3.2 Attribute-Specific Feature Decorrelation94 By con ucting the aforementioned unsupervised attribute-guided learning , we ca obtain th internal95 latent vectors V 0 as the attribute-specific features . In order to both enhance the discriminative bility96 and remove the redundant correlation among these attribute-specific features , we introduce a feature97 decorrelation constraint upon V 0 , which is formulated by98 m V 0 kV 0V 0 > kIk2F , ( 8 ) where I is the identity matrix . Such a feature decorrelation constraint is pref rable to construct99 independent features and reduce redundant inf rmation . T erefo e , based on unsupervise ttri t -100 guided learning and attribute-specific feature decorr lation , the final learned hash codes are expected101 to be both attribute-aware and hash-bit indep ndent.102 3.3.3 Hash Cod Learning103 In the f llowing , we conduct the hash code learning based on the btained attribute-sp cific feat res.104 Assume that we have n query data points which are denoted as { qi } ni=1 , as well as m database points105 3 Dec der 3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET model . We elaborate it in65 the following four a pects , i.e. , unsupervised attribute-guided learning , attribute-spec c feature66 decorrelation , hash code learning and the optimization learning algorithm.67 3.3.1 Unsupervised Attribute-Guided Lear ing68 In real-applications , especially for the large-scale and fine-grained tasks , attribute annotations are69 always infeasible , which limits the lear ing proc ss to be conducted in an u supervised setting.70 While , in the literatur , the main goal of u supervised l arning is to captu e regularities in data f r the71 purpose of extra ting useful repre entat ons or for rest ring orrupted da a [ 4 ] . Many unsupervised72 methods explicitly produce internal latent units or codes , from which the data is to be rec nstructed.73 Inspired by this , we develop an unsupervised attribute-guided learning component to project the74 holistic representation xi of Ii into a latent spac , i.e. , the att ibut space V . In V , its high-level75 vectors are designed to have certain d irable properties , e.g. , corresponding to semanti properties76 of fine-grained objects ( a.k.a . “ fin -grained attributes ” ) .77 More specifically , in our A2-NET , the unsupervised attribute-guided learning is re lized by a recon-78 struction paradigm wi h an encoder-decoder structure , as shown in Figure ? ? . In concretely , given79 a batch of n ra ning data Ii , th holistic repres ntatio s X = { x1 ; x2 ; . . . ; xn } 2 Rd⇥n can be80 obtained as aforementioned . By formulation , the encoder projects X into the attribute sp ce V with81 a projection matrix W 2 Rk⇥d to get an internal atent representation V 2 Rk⇥n w.r.t . X . In82 particularly , we set that the dimension of latent representation k equals to the number of hash bits in83 the final binary hash code ui . Furthermore , each col mn of V , i.e. , vi 2 Rk , can derive ui by84 ui = sgn ( tanh ( vi ) ) . ( 5 ) Meanwhile , regard ng vi , we lso consi r to reco struct its input xi by a decod r as a counterpart85 of the nco er . Therefore , on one ha d , such a rec nstruction paradigm can reduce and furthe distill86 high-level semantic cues in the attribute sp ce V . While , on the other hand , it can drive the training87 of A2-NET by preserving the similarity between queried hash codes and database points in terms of88 learning to hash ( cf . Sectio 3.3.3 ) .89 In concretely , th lear i g objective of unsupervised attribute-guided learni g is written as follows:90 min W kX W > WXk2F s.t . WX = V 0 = tanh ( V ) , ( 6 ) where the decoder ( i.e. , count rpart of t e encod r ) s realize by W > to simplify the net k.91 How ve , directly minimizing Eq . ( 6 ) with hard constraint is hard to optimize . Therefore , we relax92 the con traint into a oft constraint , nd the learning objective can be rewritten by93 mi W kX W > V 0k2F + kWX V 0k2F . ( 7 ) 3.3.2 Attribute-Specific Feature Decorrelation94 By con ucting he aforementioned unsupervis attribut -guided lear ing , we can obtain the internal95 lat nt vectors V 0 as the ttribute-specific features . In order to both enhance the discriminative bility96 and remove the redundant correlation among these attribute-specific features , we introduce a feature97 decorrelation constraint upon V 0 , which s formulated by98 min V 0 kV 0V 0 > kIk2F , ( 8 ) where I is the dentity matrix . S ch a featur decorrelation constraint is pref able to construct99 independent f tures a d r duce redundant informa ion . Therefore , based on unsupervised attr bute-100 guided learning and attribute-specific feature decorrelation , the final learned hash codes are expected101 to be both attribute-aware and hash-bit independ n .102 3.3 3 Hash C e Lear ing103 In the following , we conduct the ash code l a ning based on the obtain attribute-specific fe tures.104 Assume that we have n query data poi ts which are denoted as { qi } ni=1 , as well as m database poi ts105 3 which are denoted as { vj } mj=1 . Note that , both qi and vi are belonging to the attribute space V . Then,106 the corresponding binary hash codes can be obtained via107 ui = sgn tanh ( qi ) ) , ( 9 ) zj = sgn ( tan ( vj ) ) , ( 10 ) where ui , zj 2 { 1 , +1 } k. The goal of our hash code learning is to learn binary hash codes for both108 query points and database points from { qi } ni=1 , { vj } mj=1 , and the pairwise supervised information,109 i.e. , S 2 { 1 , +1 } n⇥m . To preserve the pairwise similarity , we adopt the ` 2 loss between the110 supervised information ( a.k.a . similarity ) and the inner product of query-database point binary code111 pairs . It can be formulated as follows:112 minU , Z Pn i=1 Pm j=1 u > i zj kSij 2 s.t . U 2 { 1 , +1 } n⇥k , Z 2 { 1 , +1 } m⇥k . ( 11 ) Overall , we get the final objective of the proposed A2-NET model by considering Eq . ( 7 ) , Eq . ( 8 ) and113 Eq . ( 11 ) together as follows:114 min W , ⇥ L ( I ) = kX W > V 0k2F + kWX V 0k2F +↵kV 0V 0 > nIk2F + nX i=1 mX j=1 u > i zj kSij 2 , ( 12 ) where , ↵ and are yper-param ters as th trade-off.115 In practi e , uri g training , it might be only available a set of atabase points { vj } mj=1 without query116 points . Thus , we randomly sample n data points from database to construct a query set , and denote117 th indices of all the database points as with the indices of the query set as ⌦ . Additionally , because118 we can not back-propagate the gradient to ⇥ due to the sgn ( · ) function , we omit the sgn ( · ) function119 and only apply tanh ( · ) for relaxation in Eq . ( 10 ) of the whole optimization process . Therefore , the120 optimization formulation of A2-NET can be rewritten with only database points { vj } mj=1 for training121 as:122 min W , ⇥ L ( I ) = kX W > V 0k2F + kWX V 0k2F + ↵kV 0V 0 > nIk2F + X i2⌦ X j2 ⇣ tanh ( W · F ( Ii ; ⇥ ) ) > zj kSij ⌘2 . ( 13 ) For optimization , we employ the back-propagation algorithm and follow [ 8 ] to train the whole123 2-NET m l in an e d-to-end manner.124 3.4 Out-of-Sample Extension125 After traini g A2-NET , the learned model c n be applied for generating binary codes for query points126 in luding uns en query p ints in th training phase . Sp ifically , we can use the following equation127 to generate the binary code for Iq:128 uq = sgn ( tanh ( W · F ( Iq ; ⇥ ) ) ) . ( 14 ) 4 Experime ts129 In this section , we present the implementation details , empirical settings , main resu ts , ablation studies130 and also s m interesting qualitative a alyses of the learned ttribute-aware binary hash cod s.131 4.1 Datasets132 x0i By following ExchNet [ 4 ] , our experiments a e conducted on five fine-grained b nchmark datas ts,133 i . . , CUB-200-2011 [ 16 ] , Aircraft [ 12 ] , Food101 [ 1 ] , NABird [ 15 ] and VegFru [ 6 ] . Specifically,134 CUB-200-2011 is on of t e most p pular us d fine-graine datasets . It contains 11,788 bird images135 from 200 bird pe i s and is officially split into 5,994 image for training with 5,794 images for136 test . Aircraf contains 10,000 im ges spanning 100 aircraft m dels with 3,334 for training , 3,333137 4 Fine-grained repr sentatio learning Un su pe rv is ed a ttr ib ut egu id ed lo ss ( S ec tio n 3 . 3 . 1 ) Feature decorrelation loss ( S ction 3.3.2 ) 3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET model . We laborate it in65 the foll wing four aspects , i.e. , unsupervised attribute-guided learning , attribute-specifi feature66 decorrelation , hash code learning and the optimization learning algorithm.67 3.3.1 Unsupervised Attribute-Guided L arning68 In real-applications , especially for the large-scale and fine-grained tasks , attribu e annotations are69 always infeasibl , which limits the learning process to be conducted in an unsupervised setti g.70 While , in the literature , the main goal of unsupervised learning is to capture regularities in data for the71 purp se of xtracting useful representations or for restoring corrupted data [ 13 ] . Many unsupervised72 methods explicitly produce internal latent units or codes , from which the data is to be recon tructed.73 Inspired by this , w develop an unsupervised attr bute-guided learning component to project the74 holistic r presentati n xi of Ii i to a latent space , i.e. , the attribute space V . In V , its high-level75 vec ors are designed to h ve certain desirable properties , e.g. , cor espo ding to semantic properties76 of fine-grained objects ( a.k.a . “ fine-grained attributes ” ) .77 More specifically , in our A2-NET , the unsupervised attribute-guided learning is realized by a recon-78 struction paradigm with an encoder-decoder structure , as shown in Figure ? ? . In concretely , given79 a batch of n training data Ii , the holistic r presentations X = { x1 ; 2 ; . . . ; xn } Rd⇥n ca be80 obtained as aforementioned . By formulation , th encoder projects X into the attribute space V with81 a projection matrix W 2 Rk⇥d to get an internal latent representation V 2 Rk⇥n w.r.t . X . In82 particularl , we set that the dimension of lat nt representation k quals to the number of hash bits i83 the final binary hash code ui . Furthermore , each column of V , i.e. , vi 2 Rk , can derive ui by84 ui = sgn ( tanh ( vi ) ) . ( 5 ) Meanwhile , regarding vi , we als consider to reconstruct its input xi by decoder as a c unterpart85 of the encoder . Therefore , o one hand , such a reconstruction paradigm can reduce and further distill86 high-level s mantic cues in the attribute space V . While , on the other hand , it an drive the training8 of A2-NET by preserving the similarity between queried hash codes and database points in terms of88 learning to hash ( cf . Section 3.3.3 ) .89 In concretely , the le rning objectiv of unsupervised attribute-guided learning is written as follows:90 min W kX W > WXk2F s. . WX = V 0 = tanh ( V ) , ( 6 ) where the decoder ( i.e. , a counterpart of the encoder ) is realized by W > to simplify the network.91 Ho ev r , dir ctly minimizing Eq . ( 6 ) with a hard constraint is hard to optimize . Therefore , we relax92 the constraint into a soft constraint , and the learning objective can be rewritten by93 min W kX W > V 0k2F + kWX V 0k2F . ( 7 ) 3.3.2 Attribute-Specific Feature Decorrelation94 By conduc ing the af rementioned u supervised attribute-guided learning , we can obtain the internal95 latent vectors V 0 as the attribute-specific features . In order to both nhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features , we introduce a feature97 d correlation constraint upon V 0 , which is formulated by98 min V 0 kV 0V 0 > nIk2F , ( 8 ) where I is the i entity matrix . Such a feature decorrelation co straint is pref rable t construct99 independent features and reduce redundant information . Therefore , b sed on unsupervise attribute-100 guided learning and attribute-specific feature decorrelation , the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102 3.3.3 Hash Code Learning103 In the foll wing , we conduct the hash code lear ing based on the obtained attribute-specific features.0 Assume that we have n query data points which are denoted as { qi } ni=1 , s well as m database points105 3 Hash code learning loss ( Section 3.3.3 ) Attrib te-aware hash codes generating “ Red head ” “ On the tree ” “ Spotted body ” 2 Related Work23 2.1 Fine-Grained Im ge Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section tr duce the overall framework and notations of the proposed A2-NET model , as28 well as elaborating the detailed modul f A2-NET and its corre ponding optimization algorit m.29 3.1 Overall Framework and Notations30 As illustrated in Figure ? ? , our A2-NET model consists of two crucial modules , i.e. , a fine-grain d31 represent tion learni g module and attribute-aware h sh codes generating module . Given a32 input image Ii , based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN , a set of attention guidanc A = { Ac } is learned for capturing fine-grained t ilored34 local patterns T ci from Ti . To distill semantical cues and further generate the fin l attribute-aware35 bi ary hash c des , we prop se to transform these appear nce-specifi features T towards sema tic-36 specific representations T̂ by performing a transform network ( · ) . After aggregating T̂ , the obtained37 attentive lo al-l vel features xci are ssociated with the global-lev l feature x global i to form as a38 holistic feature representation xi . In rder to generate attribute-aware binary hash codes , we conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond o40 an attribute vector w.r.t . one kind of namable properties of fine-grained objects . Fi ally , hash code41 learning is performed upon vi to obtain the final attribute-a are binary codes ui.42 3.2 Fine-Grained Representation L arning43 Attention plays an important role in human perception [ 3 , 7 ] , and humans exploi sequence of44 partial glim ses and s lective y focus on salient parts of an object or s ene in order to bet r cap ure45 visual structure [ 10 ] . Inspired by this , we incor orat the attention m chanism into represent ion46 learning to capture fine-grained local patter s for distinguishing subtle differ nces be ween these47 ubordi a e categories.48 In concretely , we extract the de p feature of its input image Ii via a backbo e CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 C⇥H⇥W . ( 1 ) Then , based on Ti , C attention guidance Ac 2 RH⇥W is generated as a set of attention maps , i.e. , A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be ttend d or even52 overlooked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obtain the final attribu e-aware binary co es , it is54 desirable to transform these appearance-specific ( i.e. , low-level ) f atures T to semantic-specific ( i.e.,55 mid-level ) representations which are closer to the attribute space . Thus , a transforming network ( · ) ,56 which is equipped with a stack of convolution layers , is performed on T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ presents the para eters of the corresponding transfor ing networks w.r.t . T ci and Ti , respec-58 tively . Then , we aggregate T̂ ci and T̂i by conducting global average-pooling and correspon ingly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . The holistic60 feature representation w.r.t . the input image Ii is achieved by concatenating both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 R . Note that , we hereby abstract the aforementioned fine-grai ed62 feature learning process as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 Att ntion guidance 2 Rela ed Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section , we introduce the overall framework and notations of the proposed A2-NET model , as28 well as el borating the detailed modules of A2-NET and its corresp nding optimization algorithm.29 3.1 Overall Framework and Notations30 As illustrated in Figure ? ? , our A2-NET model consists of two crucial modules , i.e. , a fine-graine31 repres ntation learning module and an ttribute-aw re hash codes generating m dule . Given an32 input image Ii , based on its corresponding de p activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN , a set of attention guidance A = { Ac } is le rned for capturing fi -grained tailored34 local patterns T ci from Ti . To distill semantical cues and further generate the final attribute-aware35 binary hash c des , we propos t transform th se app arance-specific feature T towa ds semantic-36 specific representations T̂ by performing a transform network ( · ) . After aggregating T̂ , the obt ined37 attentive local-level f atures xci are associated with the global-level feature x global i to form as38 holistic feature representation xi . In order to gen ra e attribute-aware bin ry hash c des , we conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond to40 an attribute vector w.r.t . one kind of nam bl properties of fine-grained objects . Finally , hash code41 learn ng is perform upon v o obtain the fi al attribute-aware inary cod s ui.42 3.2 Fine-Grained Representation Learning43 Attention plays a imp rtant role in human perception [ 3 , 7 ] , and humans exploit a sequence of44 artial gl mpses d selectively focus on salient part of an bj ct or a scene in or er to b tter capture45 visual structure [ 10 ] . Inspired by this , we incorporate the attention mechanism into representation46 learn ng to capture fine-grained local patterns for distingu shing subtle diff rences betw en these47 subordinate categories.48 In concretely , we extract the deep feature of its input image Ii via a backbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Then , based on Ti , C attention guidance Ac 2 RH⇥W is g nerated as a set of attention ma s , i.e. , A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obtain the final attribute-aware binary codes , it is54 desirable to transform these appearance-specific ( i.e. , low-level ) features T to semantic-specific ( i.e.,55 mid-level ) representations which are clos r to the attrib te space . Thus , a transforming network ( · ) ,56 which is equipped with a stack of convolution layers , is perf rmed on T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ presents the paramet rs of the corresponding transforming networks w.r.t . T ci and Ti , respec-58 tive y . Then , we aggregate T̂ ci and T̂i by condu ting global average-pooling d correspondin ly59 obtain the attentive local-level features xci , as well as the global-level feature x lobal i . The holistic60 feature representation w.r.t . the input image Ii is achieved by concatenati g both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 Rd . Note that , we hereby abstract the aforementioned fine-grai ed62 feature learning proc ss as a function F ( I ; ⇥ ) with its parameters ⇥.63 2 2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section , we introduce the overall framework and notati ns of the proposed A2-NET model , as28 well as elaborating the detailed odules of A2-NET and its corresponding optimization algorithm.29 3.1 Overall Framework and Notat ons30 As illustrated in Figure ? ? , our A2-NET model consists of two crucial modules , i.e. , a fine-grained31 representation learning module and an attribute-aware hash codes g nerating module . Given an32 input image Ii , based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN , a set of attention guidance A = { Ac } is learned for capturing fine-gr i ed tailored34 local patterns T ci fr m Ti . To distill semantic l cues and further generat the fin l ttribute-aware35 binary hash codes , we propose to transform these appearance-specific features T towards semantic-36 specific representations T̂ by performing a transform network ( · ) . After aggregating T̂ , the obtained37 attentive local-level features xci are associated with the global-level feature x global i to form s a38 holistic feature representation xi . In order to generate attribute-aware binary hash codes , we conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond to40 an attribute vector w.r.t . one kind of namable properties of fine-grained obj cts . Finally , hash code41 lear ing is performed upon vi to obtain the final attribut -awar binary codes ui.42 3.2 Fine-Grained Represe tatio Learning43 Attention plays an important role in human p rception [ 3 , 7 ] , and humans exploit a sequence of44 partial glimpses and se ectively focus on salient parts of an object or a scene i order to better capture45 visual structure [ 10 ] . Inspired by this , we incorporate the att ntion mechanism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48 In concretely , we extract the dee feature of its input image Ii via a backbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Then , based on Ti , C attention guidance Ac 2 RH⇥W is generated as a set of attention maps , i.e. , A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obtain the final attribute-aware binary codes , it is54 desirabl to transform these a pearance-specific ( i.e. , low-level ) f atures T to semantic-sp cific ( i.e.,55 mid-level ) representations which are closer to the attrib te space . Thus , a transforming n twork ( · ) ,56 which i equipped with a stack of convolution layers , is p rformed on T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ presents the parameters of the corresponding transfor ing networks w.r.t . T ci and Ti , respec-58 tively . Then , we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentive local-level features xci , as w l as the global-level feature x glo al i . Th holistic60 feature representation w.r.t . the input image Ii is achi ved by conc tenating both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 Rd . Note that , we hereby abstract the af rementioned fine-grained62 feature learning process as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section , we introduce the overall framework and notations of the proposed A2-NET mod l , as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29 3.1 Overall Framework and Notations30 As illustrated in Figure ? ? , our A2-NET model consists of two crucial modules , i.e. , a fine-grai d31 representation learning module and an attribute-aware hash odes g ne ating modul . Given an32 input image Ii , based n its corresponding de activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN , a set of attention guidance A = { Ac } is learned for capturing fine-grained tailored34 local patterns T ci from Ti . To distill semantical cues and further generate the final attribute-aware35 bi ary ash c d s , we propose to trans r the e ppearance-specific features T towards semantic-36 s ecific repr s nt t ons T̂ by perform ng transform network ( · ) . After aggregating T̂ , th o tained37 attentive local-level features xci are asso iated with the global-level f ature x global i to form as a38 holistic f ature representation xi . In order to ge erate attribute-aware binary hash codes , we conduct39 a reconstructing paradigm to project xi as vi in an attribute space wher its data point correspond to40 an a tribute vect r w.r.t . one kind of amabl properties of fine-grained bjects . Finally , has code41 le rning is performed up n vi to obtain the final attribute-awar binary c des ui.42 3.2 Fine-Grained Representation Learning43 Atte tion plays an important role in hum n perception [ 3 , 7 ] , and hum ns exploit a sequence of44 partial glimpses and selectively focus salient parts of an object or a scene in order to b ter capture45 visu l st ucture [ 10 ] . I spir d by this , we in orporate th attention mech nism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differences between thes47 ubordinate categories.48 In concretely , we extra t the deep feature of its input image Ii via a backbon CNN model CNN ( · ) 49 s50 Ti = CNN ( Ii ) 2 RC⇥H⇥W ( 1 ) Then , based on Ti , C attenti n guidance Ac 2 RH⇥W is generated as a set of attention maps , i.e. , A.51 The attention guidance Ac is desi ned to ev luate which deep descript rs should be attended or ven52 ov rlooked by condu ting53 T ci = A c Ti , ( 2 ) w re is the elem nt-wise Hadamard product . T obtain the final attri ute- ware binary codes , it is54 desirable to transform these appearanc -specific ( i.e. , low-level ) features T to semantic-specific ( i.e.,55 mid-level ) repr sentations which are closer to the attribute space . Thus , a transforming net ork ( · ) ,56 which is equipped with a stack of convolution layers , is performed on T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ pr sents he parameters of the correspo ding r nsforming networks w.r.t . T ci nd Ti , respec-58 tively . Then , we aggregate T̂ ci and T̂i y conducting global average-pooling and co espo dingly59 obtain the atte tive local-level f atures xci , as well as the global-level fe tur x global i . The holisti60 fe ture representation w.r.t . the input image Ii is achieved b concatenating both xci and xgl bali , i.e.,61 xi = h xci ; x lobal i i = F ( Ii ; ⇥ ) 2 Rd . Note hat , we hereby abstract the aforementioned fin -grained62 featur learning process as a function F ( Ii ; ⇥ ) with its paramet rs ⇥.63 2 … 2 Related Work23 2.1 Fi e-Grained Imag Retrieval24 2.2 Learning to Hash25 2.3 Visual Attribu es26 3 Methodology27 In thi secti n , we int oduce the verall framew rk and otation f the p oposed A2-NET model , as28 well as elaborating the detailed modules of A2-NET a its corresp nding pti iza ion algorithm.29 3.1 Overall Framework and No ations30 As illustrated in Figure ? ? , our A2-NET model consists of two crucial modules , i.e. , a fine-grained31 representation learning modul nd an attribute-aware hash codes generati g module . Given an32 input image Ii , based n it corr po ding de p activa i t n o Ti 2 RC⇥H⇥W extracted by a33 backbone CNN , a set of attention guida ce A = { Ac } i learned for capt ring fine-grained t ilored34 local patterns T ci from Ti . To distill semantical cues and further generate th final attribute-aware35 bi ary has c des , we pro ose to transfo m these ppearan e-specific f atures T owards semantic-36 sp cific representations T̂ by perf rming a transform etwork ( · ) . Aft r aggre ating T̂ , h obtained37 a tentive local-level features xi ar as ociat d with t e gl bal- evel feature x global i to form as a38 holistic feature representation xi . In order to generate attribute-aware binary h sh codes , we conduct39 a reconstr cting paradigm to projec xi a vi in an attribute space wh re ts data point correspond to40 an attrib te vector w.r.t . one kind of namable properties of fine-grained obj cts . Finally , hash code41 learning s performed upon vi to obtain the final attribute-aw re binary codes ui.42 3.2 Fine-Grained Representation Learning43 Att ntion plays an important role in hu an perception [ 3 , 7 ] , nd humans exploit a sequenc of44 partial glimpses and selectively focus on salient parts of an object or a scene in order to bet er c pture45 visual structure [ 10 ] . Inspired by this , we incorp r te the attentio mec anism into represent tion46 lear ing to capture fine-grain d local patt rns for distinguishing subtl differences between these47 subordi ate categories.48 In concretely , we extract the deep feature of its input image Ii via a backbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥ ⇥W . ( 1 ) Then , based on Ti , C attention guida c Ac 2 RH⇥W is gen rat d as s t of at nti n maps , i.e. , A.51 Th ttent on guidance Ac s designed to ev luate which deep descriptors should be attended or even52 overl ked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obt in the final attribute-aware bin ry cod s , it is54 desirable to transform these appeara ce-specific ( i.e. , low-level ) features T to semantic-specific ( i.e.,55 mid-l vel ) repres ntations which are closer to the attribute space . Thus , a transforming network ( · ) ,56 which is equipped with a stack of co volution layers , is p rformed on T s follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ prese ts the paramet rs of the corresponding transforming networks w.r.t . T ci and Ti , respec-58 tively . Then , we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentiv local-level f atures xci , as well as the global-level feature x global i . Th holistic60 f ature representation w.r.t . the input image Ii is a hieved by concatenating both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 Rd . Note that , we hereby abstract the forementioned fine-grained62 feature learning process as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section , we introduce the overall fra ework and notations of the proposed A2-NET model , as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization lgorithm.29 3.1 Overall Framework nd Notations30 As illustrated in Figure ? ? , our A2-NET model consists of two crucial modules , i.e. , a fine-grained31 representation learning module and an attribute-aware hash codes generati g module . Given an32 input image Ii , based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN , a set of attention guidance A = { Ac } is learned for capturing fine-grained tailored34 local patterns T ci from Ti . To distill semanti al cu s d further generate the final attribut -aware35 binary hash cod s , we propose to transform these a pearance-specific eatures T towards semantic-36 specific representations T̂ by performing a transform etwork ( · ) . After aggregating T̂ , the obtained37 attentive local-lev l features xci are associated with th global-level feature x global i to form as a38 holistic feature representation xi . In order to generate attribute-aware binary hash codes , we conduct39 a r constructing paradigm to project xi as vi in an ttribu e sp c where its data point correspond to40 an attribute vector w.r.t . one kind of namable properties f fine-grained objects . Finally , hash code41 lear ing is perfor ed upon vi to obtain the final attribute-aware binary codes ui.42 3.2 Fine-Grained Repres ntatio Learn g43 Attention plays an important role in human percept on [ 3 , 7 ] , and humans expl it a s quence of44 partial glimpses and selectively focu on salient parts of an object or a scene in order to better capture45 visua structure [ 10 ] . Inspir d by this , we i c rporate t e attention mechanism into representation46 learning to capture fine-grained local pa terns for distinguishing subtle differences between these47 subordinate categories.48 In concretely , we extract the deep feature f its i put im ge Ii v a a ba kbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Then , based on Ti , C attention guidance Ac 2 RH⇥W is generated as a set of ttention maps , i.e. , A.51 The attention guidance Ac is des gned to valuate which de p descriptors should be attend d or eve52 overlooked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obtain the final attribut -aware binary codes , it is54 desirable to tran for these appearance-specific ( i.e. , lo -level ) features T to semantic-spec fic ( i.e.,55 mid-level ) represe tations which are closer to the attribute space . Thus , a transforming network ( · ) ,56 which is equipped with a stack of convolution layers , is performed on T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( i ; ✓global ) , ( 4 ) where ✓ pr sents the parameters of he c rrespo ding transforming networks w.r.t . T ci d Ti , resp c-58 tively . Then , we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . Th holistic60 feature rep es ntation w.r.t . the inp t image I is achieve by concat ati g bot xci and xglobali , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 Rd . Note that , we hereby abstract the aforem ntioned fine-grained62 f ature l ning pro ess a a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 Aggre ation … 2 Related Work23 2.1 Fi e-Grained Image Retri val24 2.2 Le r ing o Hash25 2.3 Visual Attributes26 3 Method logy27 In this section , we introduce the ve ll framework nd notati ns of the propos A2-NET model , as28 well s elaborating the d t iled modules f A2-NET and its corresponding optimization algorithm.29 3.1 Over l Fra ework and Notations30 As ill str ted in Figur ? ? , our A2-NET model consist of two crucial modules , i.e. , a fin -gr ined31 repr sentation learning module and a attribute-aware hash c des ge erating mo ul . Given a32 input i age Ii , based on its co re nding de activatio tens r Ti 2 RC⇥H⇥W xtr cted by a33 back o e CNN , a set of ttention guidance A = { Ac } is learn d f capturing fine-grained tail red34 lo al patterns T ci fr m Ti . To dis ill semantical cues and further gen rate the fi al attribute-aware35 binary hash cod s , we propose to tr nsfo m these ppearance-specific features T towards semantic-36 specifi r pres tations T̂ by performing a transform network ( · ) . Afte aggrega ing T̂ , th obtaine37 atten ive l cal-level f t r s xci are ssociated with th global-level featur x global i o f m as a38 holistic e ture r pr s ntation xi . In order to g nerate attribute-aware binary hash cod s , we conduct39 a econstructing p adigm to project xi as vi i an attribute space wh re it dat point correspo d to40 an ttribute tor w.r.t . o e kind f n m ble properties of fi -grained objects . Finally , hash code41 le rning is p rformed pon vi to obtai th final attribute- ware bi ary codes ui.42 3.2 Fi e-Grained R pres ntation Le rning43 A tentio plays an important role in human pe ception [ 3 , 7 ] , and humans exploit a sequence of44 partial glimpses d selectiv ly focus on sali t parts of an object or a s ene in or e to b tt r capture45 visual stru tur [ 10 ] . I sp red by this , w incorporat the at entio mechanism into representation46 learning to ca u e fi -grained local patter s for distinguishing subtle differences between these47 s bordi te tegories.48 In concretely , we extract the deep feature of its input image Ii via a backbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 Th n , based o i , C attention uidan e Ac 2 RH⇥W is g nerated as a set f att ntion m ps , i.e. , A.51 The atte ion i a ce Ac is designed to evaluate which deep descriptors should b attended or even52 overlooked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obtain the final attribute-aware b nary codes , it is54 desirable t tran form th se appeara ce-specific ( i.e. , low-level ) features T to semantic-specific ( i.e.,55 mid-level ) representa ions whi h are closer to the attribute space . Thu , a transforming network ( · ) ,56 which is equipped with a stack of convolution layers , is performed n T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) wher ✓ presents the parameters of the corresponding transforming n tworks .r.t . T ci and Ti , respec-58 tively . Th n , we ggrega e T̂ ci and T̂i by c n ucting global average-pooling and correspondingly59 o tain the attentive local-level featur s xci , as well a the global-l vel feature x global i . The holistic60 feature representation w.r.t . the i put mage Ii is chi ved by concatenating both xci and xglobali , i.e.,61 xi = h xci ; l al = F ( Ii ; ⇥ ) 2 Rd . Note that , we hereby abstract the aforementioned fine-grained62 feature learnin proc ss as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 2 Rel t d Work23 2.1 Fi -Grain d Ima e Retrieval24 2.2 Lear i g to Ha h25 2.3 Visu l Attributes26 3 Methodology27 In this section , we introduce the overall framework and notations of the proposed A2-NET model , as28 w ll as elaborating th detailed m dules of A2-NET and its correspon ing optimization algorithm.29 3.1 Overall Framework and Notation30 As illustrated in Fig r ? ? , our A2-NET m del consists f two cruc al modules , i . . , a fi e-grained31 r p tatio l arning modul d an attribut - ware hash codes generating module . Given an32 inp t image Ii , b sed o its respond ng deep a tivation tensor Ti 2 RC⇥H⇥W extracted by a33 ba kbone CNN , a set of attention g id nce A = { Ac } is learned f r capturing fine grained tailored34 l l atterns T c from Ti . To distill semantical cues and further en r te the final att ibute-aware35 binary hash des , w pr p se to transform these ppeara ce-specific features T towards semantic-36 specific representations T̂ by perfor ing a transform netwo k ( · ) . After aggregating T̂ , he obt ined37 attentive local-lev l features xci re associ te with the glob l-lev l feature x global i to form as38 holistic feature representation xi . In order to g nerate attribute-aware binary h sh cod s , w conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond to40 an attribute v ctor w.r.t . one kind of namable properties of fine-gra ned o jects . Finally , has c de41 learning is performed upon vi to btain the fi al attribute-aware binary codes ui.42 3.2 Fine-Grained Representation Lear ing43 At ention plays a important role in human perception [ 3 , 7 ] , and humans exploit a sequence of44 pa tial glimpses and select vely f cus on salient parts of an bject or scene in ord r to be ter capture45 visual structure [ 10 ] . Inspired by this , we incorporate the attention mechanism into representatio46 lear ing to c pture fin -grain d local patterns for distinguishing ubtle differences b twe n th s47 subordinate c tegories.48 In concretely , we xtract the deep feature of its input image Ii via a backbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Then , based on Ti , C attention guidance Ac 2 RH⇥W is generat d as a set of attention maps , i.e. , A.51 The attention guidance Ac is designed to evaluate hich deep descriptors should be tte d d or eve52 overlooked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obtain the final attribute-aware binary codes , it is54 desirable to transform these appearance-specific ( i.e. , low-level ) features T to sema tic-specific ( i.e.,55 mid-level ) representatio s which are clo er to the attribute sp c . T us , transf rming etw rk ( · ) ,56 which is equipped with a stack of convolution layers , is performed on T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ presents the parameters of the corresponding transformi g networks w.r.t . T ci and Ti , r spec-58 tively . Then , we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . The holistic60 feature representation w.r.t . the input image Ii is achieved by concaten ting both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 Rd . Note that , we her by abstr c the af rementioned fine-gr ined62 feature learning process as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 2 R lat d Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section , we introduce the verall framework and notations of the proposed A2-NET model , as28 well as elaborating the detailed odules of A2-NET and its corr sponding optimization algorith .29 3.1 Overall Framework and Notations30 As illustrated in Figure ? ? , our A2-NET model cons sts of two crucial modules , i.e. , a fine-grain d31 representation learning module and an attribute-aware hash codes gene ating module . Given an32 input image Ii , based on its corresponding de p c ivation t nsor Ti 2 RC⇥H⇥W extracted by33 backbone CNN , a set of at ention guidance A = { Ac } is l arned for capturing fin -grained tail red34 local patterns T ci from Ti . To distill semantical cues and furt er generat the final at ribute-aware35 binary hash codes , we propose to transfor th se appearance-specific features T towards semantic-36 specific re resentations T̂ by performing a transform network ( · ) . Aft r aggregating T̂ , the obtained37 attentive local-level features xci are asso iated with the global-level feature x global i o form as a38 holistic featur representation i . In ord r to generate attribute-awar binary hash codes , we conduct39 a rec nstructing parad gm to project xi as vi in an attribute space wh re its data point corr spond to40 an attribute vector w.r.t . one kind of namable properties of fine-grained objects . Finally , hash code41 learning is performed up n vi to obtain the fin l attribute-aware bi ary codes ui.42 3.2 Fine-Grain d Representation Learning43 Attention plays an important role in human perception [ 3 , 7 ] , and ans exploit a sequ nce of44 partial glimps s and s lectively focus on s lient parts of an object or a e in orde to bette captur45 visual structure [ 10 ] . Inspired by this , we incorporate he attenti n ec anism into repr sentation46 learning to capture fine-grained local patterns for distinguishi g subtle ifferences between these47 subordinate categori s.48 In concretely , we extract the deep feature of its nput image Ii via a backbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Then , based on Ti , C attention guidance Ac 2 RH⇥W is generated as a set of attention maps , i.e. , A.51 The att ntion guidance Ac is designed to evaluate which d ep escriptors should be attended or even52 overlooked by conducting53 T ci = A Ti , ( 2 ) where is the element-wise Hadama d product . T obtain he final a tribute-awar binary cod s , it is54 desirable to transform these appearance-specific ( i.e. , low-level ) features T to semantic-specific ( i.e.,55 mid-level ) representations which are closer t the attribute space . Thus , transforming netwo k ( · ) ,56 which is equipped with a stack of conv lution layers , is performed on T as follows:57 T ci = ( i ; local ) , ( 3 ) T̂i = ( T ; ✓globa ) , ( 4 ) where ✓ presents the parameters of the corresponding transforming networks w.r.t . T ci and Ti , respec-58 tively . Then , we aggregate T̂ ci and T̂i by onducting glob l average-poolin and correspondingly59 obtain the attentive local-level features xci , as w ll as the gl b l-level featur x global i . Th holistic60 f ature r presentation w.r.t . t input image Ii is achiev d by concat nating both xci a d xglobali , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 Rd . Note that , we hereby bst ct the aforementione fin -grai d62 feature learning process as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 Aggregation where t e d coder ( i.e. , a counterpart of the encoder ) is r alized by W > to simplify the netw k.91 However , directly minimizing Eq . ( 6 ) with a hard constraint is hard to optimize . Therefore , we relax92 the constraint into a soft constraint , and the learning objective can be rewritten by93 min W kX W > V 0k2F + kWX V 0k2F . ( 7 ) 3.3.2 Attribute-Specifi F ur Decorrelation94 By conducting the aforementioned unsupervised attribute-guided learning , we can obtain the internal95 l tent vec rs V 0 as the attribut -sp cific features . I order to both en ance the discriminativ ability96 and remove the redundant correlation among these attribute-specific features , we introduce a featur97 decorrelation c n r int upon V 0 , which is formulat d by98 mi V 0 kV 0V 0 > nIk2F , ( 8 ) whe e I is t e i entity matrix . Such a feature ecorrelation co straint is pref rable to construct99 independent features and reduce redundant information . Therefore , based on unsupervised attribute-100 guided lear ing an ttribute-specific feature de orrelation , the final learne hash code are expected101 to be bo h ttribute-aware and hash-bit ind pend nt.102 3.3.3 H sh Co e L arning103 In the f llowing , we conduct the h sh code learni g ased on the obtained attribute-spe ific features.104 Assum that w have n q ry data p ints which ar d noted as { qi } i=1 , as w ll a m database points105 which a enoted as { vj } j=1 . Note hat , both qi and vi are belo gin to the attribute space V . Then,106 the corr sponding binary has codes ca be obt ined via107 ui = sgn ( tanh ( qi ) ) , ( 9 ) zj = sg ( tanh ( vj ) ) , ( 10 ) where ui , zj 2 { 1 , +1 } k. The goal of our hash code learning is to learn binary hash c des for both108 qu ry poin s and database points from { qi } ni=1 , { vj } mj=1 , and the pairwise supervised information,109 i.e. , S 2 { 1 , +1 } n m. To pr se ve the pairwise similarity , ad pt the ` 2 loss between the110 supervised inf rmati ( .k.a . similarity ) d t inner product f qu ry-database point binary code111 pairs . I can be fo mulated s foll ws:112 minU , Z Pn i=1 Pm j=1 u > i zj kSij 2 s.t . U 2 { 1 , +1 } n⇥k , Z 2 { 1 , +1 } m⇥k . ( 11 ) Overall , g t the fi a obj tiv of the propo ed A2-NET model by considering Eq . ( 7 ) , Eq . ( 8 ) and113 Eq . ( 11 ) toge a follows:114 min W , ⇥ L ( I ) = kX W > V 0k2F + kWX V 0k2F +↵kV 0V 0 > nIk2F + nX i=1 mX j=1 > i zj kSij 2 , ( 12 ) where , ↵ and are hyper-param ters as the trade-off.11 In practice , during training , it might be only available a set of database poin s { vj } mj=1 without query116 points . Thus , we randomly sample data poi ts from database to cons uct a query set , and denote117 the indices of all the databa e points as with the indices of the query set as ⌦ . Additionally , because11 we can not back-propagate the gradient to ⇥ due to the sgn ( · ) function , we o it the sgn ( · ) function119 and o ly apply t nh ( · ) for elaxation in Eq . ( 10 ) of the whole optimization process . Therefore , the120 optimization formulation of A2-NET can be rewritten with only database points { vj } mj=1 for training121 as:122 min W , ⇥ L ( I ) = kX W > V 0k2F + kWX V 0k2F + ↵kV 0V 0 > nIk2F + X i2⌦ X j2 ⇣ tanh ( W · F ( Ii ; ⇥ ) ) > zj kSij ⌘2 . ( 13 ) For opti ization , our A2-NET does not require co plicat d two-stage le rning algorithms , e.g. , the123 altern tive optimization strategy . We emplo the back-propagation algorithm and follow [ 8 ] to train124 the whol A2-NET mo el in a u ified end-to-end anner.125 4 “ Blue he d ” Figure 1 : Overall fr ewo k of the propos d A2-NET m el , which consists of two crucial modules , i.e. , fin -grained representa ion l arning and attribute-aware hash c d s generating . The whole etw rk c n be end-to-e d trai abl , d is generally riven by the nsupervi ed attribute-guided loss , the feature decorrelation loss and he hash cod learn g loss , cf . S ction 3.3. has th potenti l to help with large-sc le fine- ra ed d ta t o. Therefo , targeting the ig data35 challenge , fine-grained hashing [ 7 , 13 ] is a promisi g direction ort y of furth r explorations.36 low-lev to mid-lev l appe ra ce-specific t s a tic-specific37 obtain corresponding fine-grai ed obj ct attributes ( e.g. , xx ) , without significant annotati n cost38 2 Relat d ork39 In this section , we briefly review t e r lated work i the following t r e aspects , i.e. , n -g ained40 i age retrieval , learning to hash and the studies b ut visual attribu es in computer visi n problems.41 2.1 Fine-Grained Image Retrieval42 Fine-grained image retrieval as an integr l part of fine-grained image analysis [ 34 ] has gained more43 and more tractio i recent years [ 7 , 20 , 23 , 33 , 35 , 37 , 38 ] . What makes it challenging is that44 objects of fine-grained images ave only subtle differences , and often largely vary in pose , scale and45 orientation or can exhibit cross-modal d ff rences ( e.g. , sketch-based retrieval [ 20 ] ) .46 Depending on the type of query image , the most studied areas of fine-grained im ge retrieval47 can be s parated into two groups : fine-grained con ent-base image retrie al ( FG-CBIR ) and fine-8 grained sketch-based image retrieval ( FG-SBIR ) . More specifically , in FG-CBIR , u s pervised49 learning based [ 33 ] nd supervised learning based methods [ 36–38 ] were dev loped from different50 perspectives for dealing with fine-grained retrieval tasks , e . . , localizing fin -gr ined parts [ 33 ] ,51 hancing intra-class s parability with inter-class compactness [ 38 ] , and reducing the confidence52 of the fine-grained predictions [ 36 ] , etc . While , FG-SBIR needs to ot only capture fine-grained53 ch racteristics pres nt in the sketches , but also possess the ability to traverse the sketch and image54 domain gap . In the literature of FG-SBIR , the earlier works , e.g. , [ 17 , 27 , 35 ] , were mostly based55 on Siamese-triplet networks [ 3 ] to tackle the aforementioned challenges . Recently , some works56 tried to incorporate the advances of recent progress in self-supervised learning [ 5 ] and attention57 mecha isms [ 6 ] f further improving the retrieval ccuracy of FG-SBIR , e g. , [ 23 , 25 ] .58 However , although these fine-grained retrieval methods achieved good results , they still show the59 limitations in the face of large-scale data , i.e. , the searching time for exact nearest neighbor is typically60 expe siv or ven impossible for the given queries . To alleviate this issue , fine-grained hashing , which61 aims to generate compact binary codes to represent fine-grained images , as a promising direction62 has attracted the attention in the fine-grained community very recently [ 7 , 14 ] . More specifically,63 2 R d_head ? Blue_tail ? O _the_tr e ? Spotted_body ? YES NO YES YES 1 0 1 1 Att ibute-aware has codes Retrieval Top-k retrieved images 1 0 1 1 1 0 1 1 1 0 1 0 Figure 1 : K y idea f t propos A2-NET mo l , s w ll th ain proces of fine- rained hashi g ased o our attribute-a are h s c des . In co cr tely , for a query image Iq of Red bellied Woodp cke , aft r r tur ing all the correct results , th fine-grained image ( belonging to Red aded Wo pe k ) cl s s to t e qu ry ima e is al o etriev d. A2-NET : Le r i g Attr bu e-Aw re Has Codes for L g -Scal Fi -Gr Im ge R ieval Anonymous Author ( s ) Affiliation Addr ss email Abstract The task of fine-grained image retrieval aims to rank all the insta c s so that images1 depicting the concept of interests ( i.e. , the same sub-category labels ) are ranked2 highest based on the fine-grained details in the query . It is challenging due to the3 subtle differences between sub-category , as well as the fine-grained nature of small4 inter-clas variations d l rge intra- lass va iations . Espec ally for large-scale5 data , it is desirable to be ha dled in a bo h effective and efficient way . In this paper,6 we an Attribute-Aware hashing Network , i . . , A2-NET , for generating an attribute-7 awar binary cod s to not only speed up the retrieval process , but also corresponds8 to fine-grained attributes . Specifically , based on the learned attention maps , we9 design an auto-encoder to project the visual feature vector into an attribute space10 by following a reconstructing paradigm . Furthermore , A2-NET is also equipped11 with an orthogonal constraint upon the generated attributes to enhance their own12 representation abilities . Qualitative experiments on five benchmark fine-grained13 datasets show our superiority over c mpeting methods . Moreover , quantitative14 results r veal the obtained hash cod s corresp nd to the attributes of fine-g ained15 patterns.16 1 Introduction17 a other barrier also occurs because of xxx18 suffer from slow query speed and redundant storage costs due to both the explosive growth of massive19 fin -grain d data nd high-dimensional r al-valued features.20 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . Query image 101 1 Enc de 3.3 Attribute-Aware Hash Codes Ge erating64 How to generate attribute-awar h sh codes is the key of our A2-NET model . We laborate it i65 the following four aspects , i.e. , unsupervised attri ute- uided learning , attribute-specific feature66 decorrelation , hash code learni g and the opti izatio lea n ng algo it m.67 3.3.1 Unsupervised Attribute-Guided L r ing68 In r al-applic tio s , esp ially f r the large-scale d fi e-grained tasks , attribute annotations are69 always infea ible , whic limits the l ar ing p oces t be conducted in an unsupervised setting.70 Whil , i he literatur , th main go l of unsup rvis d learning is to capture regularities in data for the71 purp s f extracti g useful r presentations or for restoring corrupted d ta [ 4 ] . Many uns pervised72 m tho s explicitly r uce internal la e t n ts or c d s , fr m which the data is to be reconstru ted.73 Inspired by this , w d velop an u supervised att ibute-guided l rning component to pr ject the74 holistic representation xi of Ii into a latent space , i.e. , the attribute space V . In V , its high-level75 vectors are designed to have certain sirable pr per s , e.g. , o responding to se an ic properti s76 of fine-grained objects ( a.k.a . “ fine-gr ine at ributes ” ) .77 More s ecifically , in our A2-NET , the u supervise attribute-guid d learning is r alized by a rec n-78 struction paradi m wi a coder-d coder structur , s s own in F g e ? ? . I co cret ly , given79 a b tch of n tra i g d ta Ii , th h listic r p es ta ion X = { x1 ; x2 ; . . . ; xn } 2 Rd⇥n can be80 obtained as af remention d. By rmulation , the n ode projects X into the ttrib te space V with81 projection m trix W 2 Rk⇥d to get an inter al lat nt r pr se tation V Rk⇥n w.r.t . X . In82 particularly , we set that the dim nsion of atent represe tation k quals to th n ber of has its in83 the final binary hash code ui . Fu th r ore , each c lumn of V , i.e. , vi 2 Rk , an d riv i by84 = sg ( ta h v ) ) . ( 5 ) Meanwhil , regarding vi , w also co side to rec stru t its inpu xi by a de oder as a count rpart85 of the encoder . Therefor , on o hand , such reconstruction paradigm can reduce nd further i till86 high-level sem tic cues in the attribute space V . While , on the other h nd , it can drive the training87 of A2-NET by prese ving he s il rity etween queried hash co es and databas poi ts i terms of88 learning to hash ( cf . Section 3.3.3 ) .89 In concretely , the le rni g bjective of unsupervised att ibut -guided lear ing i written s follows:90 min W kX W > WXk2F s.t . WX = V 0 = t h ( V ) , ( 6 ) wh re the de oder ( i.e. , unter ar f the e coder ) is r alized by W > to simplify th etw rk.91 However , directly minimizing Eq . ( 6 ) with a hard constrai t s hard to optimize . Therefor , w r l x92 the constr int into a soft con traint , and th learning objective can be rewritten by93 min W kX W > V 0 2 + kWX V 0k2F . ( 7 ) 3.3.2 Attribute-Specific Feature D correlation94 By conducting the aforementioned unsupervised attribute-guided learni g , w ca obtain he internal95 latent vect rs V 0 as the attribute-specific features . In order to both e han e the di cr minativ bility96 and remove the redundant correlation among these a tribute-specific features , w introduce fea ur97 decorrelation constrain upon V 0 , which is formulated by98 min V 0 kV 0V 0 > kIk2F , ( 8 ) where I is the d ntity mat ix . Such a featur decorrelatio constraint is pr f rable to construct99 independent f atures and redu e red ndant informati n. Therefore , based on unsupervised attri t100 guided learning and attribute-specific feature decorrelation , the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102 3.3.3 Hash Code Learning103 In the following , we conduct the hash code learning based on t e obtained attribute-specific featu .104 Assume that we have n que y data points which are denoted as { qi } ni=1 , as well as m database point105 3 3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET mo el . We elaborate it in65 the following four aspects , i.e. , unsupervised attribute-guided learning , attribute-specific feature66 decorrelation , hash code learning and the optimization learning algorithm.67 3.3.1 Unsupervised A tribut -Guid d Lea ing68 In real-applications , especially for the large-scale a d fine-grained asks , attribute annotations are69 always infeasible , which limits the ea ning proc ss to be conduct d in a unsup rvi d setting.70 While , in the literature , the main goal of un upervised learning i to capture re ulari ies i ata for the71 purpose of extracting useful r presentations r for restoring corrupted d a [ 4 ] . Many unsup rvised72 methods explicitly produce internal latent units or codes , from whi h the data i to be r constructed.73 Inspired by this , we develop an unsupervised attribute-guided learning component to project the74 holistic representation xi of Ii into a latent space , i.e. , the attribute spa V . In V , i s high-l vel75 vectors are designed to have certain desirable properties , e.g. , corresponding to semantic properties76 of fine-grained objects ( a.k.a . “ fine-grai ed attributes ” ) .77 More specifically , in our A2-NET , th un upervised tt ibute-gui d l arn ng is r lized by recon-78 struction paradigm with a encoder-decoder structure , as show in Figure ? ? . In concretely , giv n79 a batch of n training data Ii , the h listic e esentations X = { x1 ; x2 ; . . . ; xn } 2 Rd⇥n an be80 obtained as aforementioned . By formulati n , th e o er pr jects X i to the attribute p ce V wi h81 a projection matrix W Rk⇥d to g t an int rnal latent epr se tatio V 2 Rk⇥ w. .t . X . In82 particularly , we set that the d en ion of l t nt r pr sent tio k equ ls to the number of hash bit in83 the final binary hash code ui . Furthermore , each colum of V , i . . , v 2 Rk , ca derive ui by84 ui = sgn ( tanh ( vi ) ) . ( 5 ) Meanwhile , regarding vi , we also consider to re onstruct its in t xi by dec der as a counterpart85 of the encoder . Therefore , on one hand , such a reconstructio paradi m c n r duc and fur h d still86 high-level semantic cues in the attribute sp c V . While , on th other hand , t c n rive the rai ing87 of A2-NET by pres ving he similarity b twe n queri d h h cod a d ataba o nts in t m of88 learning to hash ( cf . Section 3.3.3 ) .89 In concretely , the learning objective of unsupervised ttribut u d learning is writt s follows:90 min W kX W > WXk2F s.t . WX = V 0 = anh ( V ) , ( 6 ) where the decoder ( i.e. , a counterpart of the ncod r ) is realiz d by W > t implif th network.91 However , directly minimizing Eq . ( 6 ) with a hard constr int is hard to optimize . Therefore , we r lax92 the constraint into a soft constraint , nd the learning obj ctive can be rewrit by93 mi W kX W > V 0k2F + kWX V 0k2F . ( 7 ) 3.3.2 Attribute-Specific Feature Decorre a ion94 By conducting the afor mentio ed unsupervised attribute-guided le ning , we can obtain the internal95 latent vectors V 0 as the attribute-specific features . In ord r to both nhan e the discri inative ability96 and remove the redundant correlation among these attribute-specific features , we introduce a fe ture97 decorrelation constraint upon V 0 , which is formulated by98 mi V 0 kV 0V 0 > kIk2F , ( 8 ) where I is the identity matrix . Such a feature decorrelation constraint is preferable to construct99 independent features and reduce redundant information . Therefore , b sed on unsupervised attribute-100 guided learning and attribute-specific feature dec r elation , the final learned hash codes ar expected101 to be both attribute-aware and hash-bit independent.102 3.3.3 Hash Code Learning103 In the following , we conduct the hash code learning based on the obtained attribute-s c features.104 Assume that we have n query data points which are denote s { qi } ni=1 , as well as m database points105 3 3.3 Attribute-Awar H sh Codes G n ati g64 How to generate attribut -aware hash codes is the key of our A2-NET model . We elaborate it in65 the following four aspec s , i.e. , un u e vised attribu e-gui ed lear i g , attribute-s ecific feat re66 d c relati n , hash code learning and the ptimization learn g algorith .67 3.3.1 Unsup rvised Att ibute-Guided Learning68 In real-a plications , esp cially for the large-scale and fine-gr ined tasks , attribute an ot tions are69 always infeasible , which limits the learning proc ss to be con ucted in an u supervised setting.70 While , in the literature , the main goal of u supervised learning is to capture regul rities in data for the71 purpo e of extracti g useful repr s nt tions or for restoring orrupted d ta [ 4 ] . Many unsup rvi d72 methods explicitly produc internal lat nt units r c des , fr w ic t e dat is o be r constructed.73 Inspired by this , we develop an un upervise attribut -g ided learn ng compon t t r ject t74 holistic representatio xi of Ii into a latent space , i.e. , the attribut space V . In V , its high-level75 vectors are designed to have certain desirable propertie , e.g. , corresponding to s mantic prope ties76 of fine-grained objects ( a.k.a . “ fine-grained attribut s ” ) .77 More specifically , in our A2-NET , h un up rvised attribute-guided arning is re lized by a r co -78 structi n aradigm it an encoder-decod r structure , as shown in Figu e ? ? . In c ncre ely , given79 a batc of n tr in ng data Ii , the holi ic r pr se tation X = { x1 ; x2 ; . . . ; xn } 2 Rd⇥ n be80 obtained as aforemention d. By form lation , the enco er projects X into the t ribu e space V with81 a projectio matrix W 2 Rk⇥d to t an nternal lat nt pr ent tion V 2 Rk⇥n w.r.t . X . I82 particularly , w s t tha the dim nsio of late t rep se tati n k quals t the numb r of hash bits in83 the final binary ash c de ui . Furthermore , each lum of V , i . . , vi 2 Rk , can derive ui by84 ui = sgn ( tanh ( vi ) ) . ( 5 ) Meanwhile , regarding vi , we also consider to econstruct i s input xi by a decoder as a counterpart85 of the ncod . Th refore , on one hand , uch reconstructio par digm can reduce and further distil86 high-level semantic cues in the attribut space V . Whil , on the other hand , it can drive the training87 of A2-NET by preservin the similarity betwee queried hash codes and database points in terms of88 learning to hash ( cf . Section 3.3.3 ) .89 In con retely , the learn ng objective f supervised attribute-guided learni g is written as foll ws:90 min W kX W > WXk2F .t . WX = V 0 = anh ( V ) , ( 6 ) where the decoder ( i.e. , a counterpart f the encoder ) is r alized y W > to simplify the network.91 However , irectly minimizi g Eq . ( 6 ) with a hard con traint is hard to optimize . Therefore , we r lax92 the constraint into a soft constraint , and the l ar ing objective can be rewritte by93 min W kX W > V 0k2F + kWX V 0k2F . ( 7 ) 3.3.2 Attribute-Specific Featur Decorr lation94 By conducting the aforementioned unsupervised attribute-guided learning , we can obtain the internal95 latent vectors V 0 as the attr bute-specific features . In order to both enhance the discriminative ability96 and remove the redundan correlation among these attribute-specific features , we introduce a feature97 decorrelation constraint upon V 0 , which is formulated by98 min V 0 kV 0V 0 > kIk2F , ( 8 ) where I is the identity matrix . Such a feature decorrel tio constraint is prefer ble to construct99 independent features and reduce redundant information . Therefore , based on unsup rvised attribute-100 guided learning an attribute- p ific feature ecorrelation , the final l arned hash codes are expect101 to be both attribute-aware and hash-bit independent.102 3.3.3 Hash Code Learning103 In the following , we conduct the hash code learning based on the obtained attribut -spec fic featur s.104 Assume that we have n query data points which are denoted as { qi } ni=1 , as well as m da base points105 3 3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET model . W laborat it in65 the following four aspects , i.e. , unsupervised attribute-guided learning , attribute-specific feature66 decorrelation , hash code learning and the optimization learning algorithm.67 3.3.1 Unsupervised Attribute-Guided Learning68 In real-applications , especially for the large-sca e and fine-grained task , ttr bute an otation r69 always infeasible , which limits the learning process to be nducted in an unsu ervised setting.70 While , in the literature , the main goal of unsupervised learning is to capture regularities in dat for the71 purpose of extracting useful representations or for restoring corrupted data [ 4 ] . Many unsupervised72 methods explicitly produce internal latent units or codes , from which the data is to be reconstructed.73 Inspired by this , we develop an unsupervised attribute-guided learning compo ent to project the74 holistic representation xi of Ii into a latent space , i.e. , t tribute sp ce V . In V , its high-l75 vectors are designed to have certain desirable properties , e.g. , corresponding to semantic properties76 of fine-grained objects ( a.k.a . “ fine-grained attributes ” ) .77 More specifically , in our A2-NET , the unsupervised attribut -guided learning s r alize by a reco -78 struction paradigm with an encoder-decoder structure , as shown in Figure ? ? . In concretely , given79 a batch of n training data Ii , the holistic representations X = { x1 ; x2 ; . . . ; xn } 2 Rd⇥n can be80 obtained as aforementioned . By formulation , the encoder projects X into the attribute space V with81 a projection matrix W 2 Rk⇥d to get an internal latent representation V 2 Rk⇥n w.r.t . X . In82 particularly , we set that the dimension of latent representation k equals to the number of hash bits in83 the final binary hash code ui . Furthermore , each column of V , i.e. , vi 2 Rk , can derive ui by84 ui = sgn ( tanh ( vi ) ) . ( 5 ) Meanwhile , regarding vi , we also consider to reconstruc its input xi by decoder as a counterpart85 of the encoder . Therefore , on one hand , such a reconstruction paradigm can reduce nd further distill86 high-level semantic cues in the attribute space V . While , on the other hand , it can drive the training87 of A2-NET by preserving the similarity between queried hash codes and database points in terms of88 learning to hash ( cf . Section 3.3.3 ) .89 In concretely , the learning objective of unsupervised attribute-guided learning is written as follows:90 min W kX W > WXk2F s.t . WX = V 0 = tanh ( V ) , ( 6 ) where the decoder ( i.e. , a counterpart of the encoder ) is realized by W > to simplify the network.91 However , directly minimizing Eq . ( 6 ) with a hard constraint is hard to optimize . Therefore , we relax92 the constraint into a soft constraint , and the learning objective can be rewritten by93 min W kX W > V 0k2F + kWX V 0k2F . ( 7 ) 3.3.2 Attribute-Specific Feature Decorrelation94 By conducting the aforementioned unsupervised attribute-guided learning , we can btain the int rnal95 latent vectors V 0 as the attribute-specific features . In order to both enhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features , we introduce a feature97 decorrelation constraint upon V 0 , which is formulated by98 min V 0 kV 0V 0 > kIk2F , ( 8 ) where I is the identity matrix . Such a feature decorrelation constraint is pr f rable to construct99 independent features and reduce redundant information . Therefore , based on unsupervised ttri t -100 guided learning and attribute-specific feature decorrelation , the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102 3.3.3 Hash Code Learning103 In the following , we conduct the hash code learning based on the obtained attribute-sp cific feat res.104 Assume that we have n query data points which are denoted as { qi } ni=1 , as well as m database points105 3 D coder 3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET model . We elaborate it in65 the following four aspects , i.e. , unsupervised ttribute-guided learning , attribute-specific feature66 decorrelation , hash code learning and the optimization learning algorithm.67 3.3.1 Unsupervised Attribute-Guided Learning68 In real-applications , especially for the large-scale and fine-grained tasks , attribute annotations are69 always infeasible , which limits the learnin proc ss to be conducted in an unsupervised setting.70 While , in the literature , the main goal of unsupervised learning is to capture regularities in data for the71 purpose of extracting useful representations or for restoring corrupted data [ 4 ] . Many unsupervised72 methods explicitly produce internal latent units or codes , from which the data is to be reconstructed.73 Inspired by this , w develop an unsupervised attribute-guided learn ng component to project th74 holistic repr sentation xi of I into a late t space , i.e. , th attribute space V . In V , its high-level75 vectors are desig ed to have certain d sirable pr perties , e.g. , corresponding to semantic properties76 of fine-grain d objects ( a.k.a . “ fine-grained attributes ” ) .77 More specifically , in our 2-NET , the unsupervised attribute-guided learning is realized by a reco -78 s ruction paradigm with an ncode -decoder structure , as shown in Figure ? ? . In concret ly , give79 a batch of n training d ta Ii , the holistic represe tatio s X = { x1 ; x2 ; . . . ; xn } 2 Rd⇥n can be80 obtained as afor menti ned . By form atio , th enco r projects X into the attribute spac V with81 a proj ction matrix W 2 Rk⇥d to get an internal latent representation V 2 Rk⇥n w.r.t . X . In82 p rticularly , we se that the dim nsion of latent repr sentation k equals t th number of hash bits in83 the final binary hash c de i . Furthermore , each colum of V , i.e. , vi 2 Rk , ca derive ui by84 ui = sg ( tanh ( vi ) ) . ( 5 ) Meanwhile , regard ng vi , we l o conside to reco struct it input xi by a decod r s a counterpart85 f the ncoder . Ther fore , o o e ha , ch a rec nstruction paradigm can re uce and further distill86 high-level semantic cues in the attribute space V . While , on the ther hand , it can drive the training87 of A2-NET by p eserving the imila ity between queried h sh cod s and database po nts in t rms of88 lear ing to hash ( cf . Section 3.3.3 ) .89 In concretely , the learni g objec ive of nsupervised t ibute-guid d learni g is w itten as follows:90 min W kX W > WXk2F s.t . WX = V 0 = tan ( V ) , ( 6 ) where the decoder ( i.e. , a count rpart f the encod r ) is realized by W > to s mplify the network.91 How ve , di tly minimizing Eq . ( 6 ) with a h rd onstraint is hard to optimize . T erefore , rel x92 the constr i t int a soft constraint , and th l arning obj ctiv can b r written by93 mi W kX W > V 0k2F + kWX V 0k2F . ( 7 ) 3.3.2 A tribut -Specific F ure Decorrela ion94 By c ducting the a re e tioned unsupervis d attribute-g id d l arni g , we can obtain the in r l95 latent vectors V 0 as the attribute-specific fe ures . In rd r to both e hanc the iscrimi ative a il y96 and r m ve the re un nt co relation among these attribute-sp cific features , we introd ce a feat e97 decorrela ion co str int upon V 0 , which is formulat d by98 min V 0 kV 0V 0 > kIk2F , ( 8 ) where I is the identity matrix . Such a f ature d c rr lati n constraint is pr f rabl to onstruct99 independent features and reduce redundant information . Therefore , based on unsupervised attribute-100 uided l arning and attribute-specific feature decorrelat on , th fi al learned h sh c des are xpected101 to be both attribute-aw re and hash-bit ndepend nt.102 3.3.3 Hash Code Learning103 In the following , we conduct the hash code learning based on the obtained attribute-specific features.104 Assume that w hav query d ta poi ts which ar den ted as { qi } ni=1 , as w ll as m d tabase points105 3 which are denoted as { vj } mj=1 . Note that , both qi and vi are belonging to the attribute space V . Then,106 the corresponding binary hash codes can be obtained via107 ui = sgn ( tanh ( qi ) ) , ( 9 ) zj = sgn ( tanh ( vj ) ) , ( 10 ) where ui , zj 2 { 1 , +1 } k. The goal of our hash code learning is to learn binary hash codes for both108 query points and database points from { qi } ni=1 , { vj } mj=1 , and the pairwise supervised information,109 i.e. , S 2 { 1 , +1 } n⇥m . To preserve the pairwise similarity , we adopt the ` 2 loss between the110 supervised information ( a.k.a . similarity ) and the inner product of query-database point binary code111 pairs . It can be formulated as follows:112 minU , Z Pn i=1 Pm j=1 u > i zj kSij 2 s.t . U 2 { 1 , +1 } n⇥k , Z 2 { 1 , +1 } m⇥k . ( 11 ) Overall , we get the final objective of the proposed A2-NET model by considering Eq . ( 7 ) , Eq . ( 8 ) and113 Eq . ( 11 ) together as follows:114 min W , ⇥ L ( I ) = kX W > V 0k2F + kWX V 0k2F +↵kV 0V 0 > nIk2F + nX i=1 mX j=1 u > i zj kSij 2 , ( 12 ) where , ↵ an are hyper-parameters as the trade-off.115 In practice , uri g training , it might be only available a set of database points { vj } mj=1 without qu ry116 points . Thus , we randomly sample n data points from database to construct a query set , and denote117 th indices of all the database points as with the indices of the query set as ⌦ . Additionally , because118 we can not b ck-propagate the gradient to ⇥ due to the sgn ( · ) fu ction , we omit the sgn ( · ) function119 and only apply tanh ( · ) for relaxation in Eq . ( 10 ) of the wh le optimization process . T erefore , th120 optimization formul tion of A2-NET can be rewritten with only atabase points { vj } mj=1 for training121 as:122 min W , ⇥ L ( I ) = kX W > V 0k2F + kWX V 0k2F + ↵kV 0V 0 > nIk2F + X i2⌦ X j2 ⇣ tanh ( W · F ( Ii ; ⇥ ) ) > zj kSij ⌘2 . ( 13 ) For optimization , we e ploy the back-propagat on lgorithm and follow [ 8 ] t train the whole123 A2-NET m l in an e d-t - d m nn r.124 3.4 Out-of-Sample Extension125 After tr ining A2-NET , the learned model can be applie for genera ing binary codes f r query points126 in lud ng unseen query points in th training phase . Specifically , we can use the followi g equation127 t gen rate the binary code for Iq:128 uq = sg ( ta h ( W · F ( Iq ; ⇥ ) ) ) . ( 14 ) 4 Exp riments129 In this cti , we pr sent th impl men atio d tails , empiri al settings , main results , blati n studi s130 and al o some i t rest ng qualitative a lyses of the le rned attribute-aware binary hash codes.131 4.1 Datasets132 x0i By fol owi g ExchNet [ 4 ] , our expe iments are conducted on five fine-grained benchmark datasets,133 i.e. , CUB-200-2011 [ 16 ] , Aircraft [ 12 ] , Food101 [ 1 ] , NAB ds [ 15 ] and V gFru [ 6 ] . Specifically,134 CUB-200-2011 is on of th most pop lar used fi e-gr ined d tas ts . It contains 11,788 bird images135 from 200 bird sp i s and is officially s lit into 5,994 i g s f r training with 5,794 imag s f r136 test . Ai cr ft con a s 10,000 imag s nnin 100 aircr ft models with 3,334 for training , 3,333137 4 Fine-grained representation learning Un su pe rv is ed a ttr ib ut e- gu id ed lo ss ( S ec tio n 3 . 3 . 1 ) Feature decorrelation loss ( S ction 3.3.2 ) 3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET model . We elaborate it i65 the following four aspects , i.e. , unsupervised attribute-guided learning , attribute-specifi feature66 deco relation , hash code learning and the optimization lea ning algorithm.67 3.3.1 Unsupervised Attribute-Guided Learning68 In real-applications , especially for the large-scale and fine-grained tasks , attribute annotations are69 always infeasibl , which limits the learning process to be onducted in an unsupervise setti g.70 While , in the literature , the main goal of unsupervised learning is to capture regularities in data for the71 pu p se of xtracting useful re esentations or for restoring c rrup ed data [ 13 ] . Many unsupervised72 methods explicitly produc intern l latent units or codes , fr m which the da is to b reconstructed.73 Inspired by this , we develop an unsupervised attribute-guided learning c mponent to roject the74 holistic r presentation xi f Ii into a latent space , i.e. , the attribute space V . In V , its high-level75 vectors are designed to have certain desirable properties , e.g. , cor sponding to semantic properties76 of fine-grained objects ( a.k. . “ fine-grained att ibutes ” ) .77 Mor specifically , in our A2-NET , the unsupervised attribute-gu ded l arning is realized by a recon-78 struction paradigm with an encoder-decoder structure , as shown in Figure ? ? . In concretely , given79 a batch of n training data Ii , the holistic representations X = { x1 ; x2 ; . . . ; xn } 2 Rd⇥n ca be80 obtained as aforementione . By formulation , the encoder projects X into the attribute space V with81 a projection matrix W 2 Rk⇥d to get an internal lat nt r presentat on V 2 Rk⇥n w.r.t . X . In82 particularl , we set that the dime sion of latent representation k quals to the number of hash bits in83 the final binary h sh code ui . Furthermore , each column of V , i.e. , vi 2 Rk , can derive ui by84 ui = sgn ( tanh ( vi ) ) . ( 5 ) Meanwh le , regardi g vi , we als consider to reconstruct its input xi by decoder as a counterpart85 of th enco er . Th efore , o one hand , such a r construction paradigm can reduce an further distill86 high-level semantic cues in the attribute space V . While , on the other hand , it can driv the training87 of A2-NET by preserving e similarity betwee queried ha h codes and databas points in t rms of88 learning to hash ( cf . Section 3.3.3 ) .89 In concretely , the learning object v of unsupervised at ribute-guided learni g i written as follows:90 mi W kX W > WXk2F s. . WX = V 0 = tanh ( V ) , ( 6 ) where the decoder ( i.e. , a count rpart of the encoder ) is realized by W > to simplify the network.91 Howev , directly minimizing Eq . ( 6 ) with a hard constraint is hard to optimize . Therefore , we relax92 the constraint into a soft cons rai t , and the lea ning objective can be rewr tt n by93 min W kX W > V 0k2F + kWX V 0k2F . ( 7 ) 3.3.2 Attribute-Specific Feat re Dec rrelation94 By conducting the aforementioned u supervis d attribute-guided l arning , we can obtain the intern l95 l tent vectors V 0 as t e attribute-spe ific featur s. In r r to both enhance the discrimin tive ability96 and remove the redunda t correlatio among these attri ute-sp cific fe tu es , we introduce a f ature97 d correlation constraint upon V 0 , hich is formulated by98 min V 0 kV 0V 0 > nIk2F , ( 8 ) where I is the i entity matrix . Such a feature decorrelation constraint is preferable to construct99 independent features and reduce redundant information . Therefore , b sed on unsupervise attribute-100 guided learning and attribute-specific feature decorrelation , the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102 3.3.3 Hash Code Learni g103 In the following , we conduct the hash code lear ing based on the obtained attribute-specific features.104 Assume that we have n query data points which are denoted as { qi } ni=1 , s well as m database points105 3 H h code learning loss ( Section 3.3.3 ) Attribute-aware has c des g ne tin “ Red head ” “ On the tre ” “ Spotted b dy ” 2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section , we introduce the overall framework and notations of the proposed A2-NET mod l , as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29 3.1 Overall Framework and Notations30 As illustrated in Figure ? ? , our A2-NET model consists of two crucial modules , i.e. , a fine-grain d31 representation learning module and an attribute-aware hash codes generating module . Given an32 input image Ii , based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN , a set of attention guidance A = { Ac } is learned for capturing fine-grained tailored34 local patterns T ci from Ti . To distill semantical cues and further generate the final attribute-aware35 binary hash codes , we propose to transform these appearance-specific features T towards semantic-36 specific representations T̂ by performing a transform network ( · ) . After aggregating T̂ , the obtained37 attentive local-level features xci are associated with the global-level feature x global i to form as a38 holistic feature representation xi . In order to generate attribute-aware binary hash codes , we conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point corr spond o40 an attribute vector w.r.t . one kind of namable properties of fine-grained objects . Finally , hash code41 learning is performed upon vi to obtain the final attribute-a are binary codes ui.42 3.2 Fine-Grained Representation Learning43 Attention plays an important role in human perception [ 3 , 7 ] , and humans exploit a sequence of44 partial glimpses and s lective y focus on salient parts of an object or a s ene in order to bet r capture45 visual structure [ 10 ] . Inspired by this , we incorporat the attention mechanism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differ nces between these47 subordinate categories.48 In concretely , we extract the deep feature of its input image Ii via a backbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Then , based on Ti , C attention guidance Ac 2 RH⇥W is generated as a set of attention maps , i.e. , A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or eve52 overlooked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obtain the final attribu e-aware binary co es , it is54 desirable to transform these appearance-specific ( i.e. , low-level ) features T to semantic-specific ( i.e.,55 mid-level ) representations which are closer to the attribute space . Thus , a transforming network ( · ) ,56 which is equipped with a stack of convolution layers , is performed on T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ presents the para eters of the corresponding transfor ing networks w.r.t . T ci and Ti , respec-58 tively . Then , we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . The holistic60 feature representation w.r.t . the input image Ii is achieved by concatenating both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 R . Note that , we hereby abstract the aforementioned fine-grai ed62 feature learning process as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 Attention guidance 2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section , we introduce the overall framework and notations of the proposed A2-NET model , as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29 3.1 Overall Framework and Notations30 As illustrated in Figure ? ? , our A2-NET model consists of two crucial modul s , i.e. , a fine-graine31 representation learning module and an attribute-aware hash codes generating module . Given an32 input image Ii , based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN , a set of attention guidance A = { Ac } is le rned for capturing fi -grained tailored34 local patterns T ci from Ti . To distill semantical cues and further generate the final attribute-aware35 binary hash c des , we propose to transform these appearance-specific features T t wa ds semantic-36 specific representations T̂ by performing a transform network ( · ) . After aggregating T̂ , the obt ined37 attentive local-level features xci are associated with the global-level feature x global i to form as a38 holistic feature representation xi . In order to generate attribute-aware bi ary hash codes , e conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond to40 an attribute vector w.r.t . one kind of namable properties of fine-grained objects . Finally , hash code41 learning is performed upon vi o obtain the final attribute-aware binary codes ui.42 3.2 Fine-Grained Representation Learning43 Attention plays a imp rtant role in human perception [ 3 , 7 ] , and humans exploit a sequence of44 artial glimpses d selectively focus on salient parts of an object or a scene in or er to better capture45 visual structure [ 10 ] . Inspired by this , we incorporate the attention mechanism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48 In concretely , we extract the deep feature of its input image Ii via a backbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Then , based on Ti , C attention guidance Ac 2 RH⇥W is g nerated as a set of attention ma s , i.e. , A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obtain the final attribute-aware binary codes , it is54 desirable to transform these appearance-specific ( i.e. , low-level ) features T to semantic-specific ( i.e.,55 mid-level ) representations which are closer to the attribute space . Thus , a transformi g n twork ( · ) ,56 which is equipped with a stack of convolution layers , is performed on T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ presents the parameters of the corresponding transforming networks w.r.t . T ci and Ti , respec-58 tively . Then , we aggregate T̂ ci and T̂i by conducting global average-pooling d correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . The holistic60 feature representation w.r.t . the input image Ii is achieved by concatenati g both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 Rd . Note that , we hereby abstract the aforementioned fine-gra ned62 feature learning process as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section , we introduce the overall framework and notations of the proposed A2-NET model , as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29 3.1 Overall Framework and Notations30 As illustrated in Figure ? ? , our A2-NET model consists of two crucial modules , i.e. , a fine-grained31 representation learning module and an attribute-aware hash codes generating module . Given an32 input image Ii , based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN , a set of attention guidance A = { Ac } is learned for capturing fine-grained tailored34 local patterns T ci from Ti . To distill semantical cues and further generate the final attribute-aware35 binary hash codes , we propose to transform these appearance-specific features T towards semantic-36 specific representations T̂ by performing a transform network ( · ) . After aggregating T̂ , the obtained37 attentive local-level features xci are associated with the global-level feature x global i to form as a38 holistic feature representation xi . In order to generate attribute-aware binary hash codes , we conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond to40 an attribute vector w.r.t . one kind of namable properties of fine-grained objects . Finally , hash code41 learning is performed upon vi to obtain the final attribute-aware binary codes ui.42 3.2 Fine-Grained Representation Learning43 Attention plays an important role in human perception [ 3 , 7 ] , and humans exploit a sequence of44 partial glimpses and selectively focus on salient parts of an object or a scene in order to better capture45 visual structure [ 10 ] . Inspired by this , we incorporate the attention mechanism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48 In concretely , we extract the deep feature of its input image Ii via a backbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Then , based on Ti , C attention guidance Ac 2 RH⇥W is generated as a set of attention maps , i.e. , A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obtain the final attribute-aware binary codes , it is54 desirable to transform these appearance-specific ( i.e. , low-level ) f atures T to semantic-specific ( i.e.,55 mid-level ) representations which are closer to the attrib te space . Thus , a transforming n twork ( · ) ,56 which is equipped with a stack of convolution layers , is performed on T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ presents the parameters of the corresponding transfor ing networks w.r.t . T ci and Ti , respec-58 tively . Then , we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x glo al i . Th holistic60 feature representation w.r.t . the input image Ii is achieved by concatenating both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 Rd . Note that , we hereby abstract the aforementioned fine-grained62 feature learning process as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section , we introduce the overall framework and notations of the proposed A2-NET model , as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29 3.1 Overall Framework and Notations30 As illustrated in Figure ? ? , our A2-NET model consists of two crucial modules , i.e. , a fine-grai d31 representation learning module and an attribute-aware hash codes g nerating ul . Given an32 input image Ii , based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN , a set of attention guidance A = { Ac } is learned for capturing fine-grained tailored34 local patterns T ci from Ti . To distill semantical cues and further g nerate the final attribute-aware35 bi ary ash c des , we propose to transform these appe rance-specific features towar s semantic-36 specific representations T̂ by perform ng a transform network ( · ) . Aft r aggregating T̂ , th obtained37 attentive local-level features xci are associated with the global-level f ature x g obal i to for as a38 holistic feature representation xi . In order to generate attribute-aware binary hash codes , we conduct39 a reconstructing paradigm to project xi as vi in an attribute space wher its dat point corresp nd to40 an attribute vector w.r.t . one kind of namable propertie of fine-grained objects . Finally , hash co41 learning is performed upon vi to obtain the final attribute-aw e binary codes ui.42 3.2 Fine-Grained Representation Learning43 Attention plays an important role in human perception [ 3 , 7 ] , and humans exploit a sequence of44 partial glimpses and selectively focus on salient parts of an object or a scene in order to be ter capture45 visu l structure [ 10 ] . I spired by this , we incorporate th attention mech nism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48 In concretely , we extra t the deep feature of its input image Ii via a backbon CNN model CNN ( · ) 49 s50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Then , based on Ti , C attention guidance Ac 2 RH⇥W is generated s a set of attention maps , i.e. , A.51 The attention guidance Ac is designed to ev luate which deep descript rs should be attended or ven52 ov rlooked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obtain the final attribute- ware binary codes , it is54 desirable to transform these appearance-specific ( i.e. , low-level ) features T to semantic-specific ( i.e.,55 mid-level ) representations which are closer to the attribute space . Thus , a transforming network ( · ) ,56 which is equipped with a stack of convolution layers , is performed on as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ pr sents the parameters of the corresponding tr nsforming networks w.r.t . T ci d Ti , resp c-58 tively . Then , we aggregate T̂ ci and T̂i y conducting global average-pooling and cor spo dingly59 obtain the atte tive local-level f atures xci , as well as the global-level fe tur x global i . The holisti60 fe ture representation w.r.t . the input image Ii is achieved b concatenating both xci and xgl b li , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 Rd . Note hat , we hereby abstract the aforem ntion d fin -grain d62 featur learning process as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 … 2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this secti n , we int oduce the over ll framew rk and otations f the p oposed A2-NET ode , as28 well as elaborating the detailed modules of A2-NET a its corresp nding ptimiza io algorithm.29 3.1 Overall Frame ork and No ations30 As illustrated in Figure ? ? , our A2-NET model consists of two crucial modules , i.e. , a fi e-grained31 representation learning module and an attribute-aware ha h codes generating modul . G v n an32 input image Ii , based n its corr po d ng deep activa io ten o Ti 2 RC⇥H⇥W ex rac ed by a33 backbone CNN , a set of attention gui a ce A = { Ac } i learned for capt g fine-grained t ilored34 local patt ns T ci from Ti . To distill semantical cues and further generate the final attribute-aware35 bi ary has c des , we pro ose to transfo m these ppearan e-s ecific f atures T owards semantic-36 sp cific representations T̂ by performing a transform etwork ( · ) . Aft r aggre ating T̂ , h obta37 attentive local-level featur s xi ar associated with t e gl bal-level f ture x global i to for s a38 holistic featu representation xi . I order to ge erate attribute-awar binary hash codes , w conduct39 a reconstructing paradigm to projec xi a vi in an attribute space wh re ts data point co respond to40 an attribute vector w.r.t . one kind of amable properties of fine-grai ed obj cts . F n lly , hash cod41 learning is performed upon vi to obtain the final attribute-aw re binary codes ui.42 3.2 Fine-Grained Representation Learning43 Att ntion lays a important rol in hu an perce tion [ 3 , 7 ] , nd h mans ex loit a seque c44 partial glimpses and selectively focus on salient parts of an bj ct or a scene in d r to better captu e45 visual structure [ 10 ] . Inspired by this , we incorp rate th attentio mechanism into present t on46 lear ing to c pture fine-grain d local patt rns for distinguishin subtle differences betw en th se47 subordinate categories.48 In concretely , we extract the deep feature of its input image Ii via a backbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Then , based on Ti , C attention guida c Ac 2 RH⇥W is gen rat d as set of attention maps , i.e. , A.51 The ttent on guidance Ac s designed to ev luate which deep descriptors should be attended or even52 overl ked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obt in t e final attribute-aw re binary codes , it is54 desirable to transform these appeara ce-specific ( i.e. , low-level ) features T to semantic-specific ( i.e.,55 mid-l vel ) repres ntations which are closer to the attribute space . Thus , a transforming network ( · ) ,56 which is equipped with a stack of convolution layers , is performed n T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ prese ts the parameters of the corresponding transforming networks w.r.t . T ci and Ti , respec-58 tively . Then , we aggregat T̂ ci a d T̂i by conducting global average-po ling nd c rr spo dingly59 obtain the attentiv local-level f atures xci , as well as the global-level feature x global i . Th holistic60 feature representation w.r.t . the input imag Ii i achieved by concatenating both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 Rd . Note that , we h reby abstract the aforementioned fine-grained62 feature learning process as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section , we introduce the overall framework and notations of the proposed A2-NET model , as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29 3.1 Overall Framework and Notations30 As illustrated in Figure ? ? , our A2-NET model consists of two crucial modules , i.e. , a fine-grained31 representation learning module and an attribute-aware hash codes generating module . Given an32 input image Ii , based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN , a set of attention guidance A = { Ac } is learned for capturing fine-grained tailored34 local patterns T ci from Ti . To distill semanti al cu s d further generate the final attribute-aware35 binary hash codes , we propose to transform these appearance-specific features T towards semantic-36 specific representations T̂ by performing a transform network ( · ) . After aggregating T̂ , the obtained37 attentive local-level features xci are associated with the global-level feature x global i to form as a38 holistic feature representation xi . In order to generate attribute-aware binary hash codes , we conduct39 a r constructing paradigm to project xi as vi in an attribute sp ce where its data point correspond to40 an attribute vector w.r.t . one kind of namable properties of fine-grained objects . Finally , hash code41 lear ing is perfor ed upon vi to obtain the final attribute-aware binary codes ui.42 3.2 Fine-Grained Representation Learning43 Attention plays an important role in human percept on [ 3 , 7 ] , and humans exploit a s quence of44 partial glimpses and selectively focus on salient parts of an object or a scene in order to better capture45 visual structure [ 10 ] . Inspired by this , we inc rporate t e attention mechanism into repres ntation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48 In concretely , we extract the deep feature f its i put im ge Ii via a ba kbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Then , based on Ti , C attention guidance Ac 2 RH⇥W is generated as a set of ttention maps , i.e. , A.51 The attention guidance Ac is designed to evaluate which deep descrip ors sh uld be attended or even52 overlooked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obtain the final attribut -aware binary codes , t is54 desirable to tran for these appearance-specific ( i.e. , low-level ) features T to semantic-spec fic ( i . .,55 mid-level ) representations which are closer to the attribute space . Thus , a transforming network ( · ) ,56 which is equipped with a stack of convolution layers , is performed on T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ pr sents the parameters of he c rrespo ding transforming networks w.r.t . T ci a d Ti , respec-58 tively . Then , we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondi gly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . Th h lis ic60 feature rep esentation w.r.t . the input image Ii i achieved by concat ati g both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 Rd . Note that , we hereby abstract the aforem ntioned fine-grained62 f ature le ning pro ess as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 Aggregation … 2 Related Work23 2.1 Fine-Grained Image Retr val24 2.2 Le rning to Hash25 2.3 Visual Attribut s26 3 Method logy27 In this section , we introduc the ove ll framework nd n t ti ns of the prop s A2-NET model , as28 well as elaborating the detailed mo ules f A2-NET and it rr spo ding ptimization algorithm.29 3.1 Over l Fra work and Notations30 As illustrated in Figur ? ? , o r A2-NET model onsist of two crucial m dul s , i.e. , a fin -gr ined31 representation learning modul and a attribute-awar h sh c des ge e ating mo l . Given a32 input image Ii , based on its cor es onding d p activatio tens r Ti 2 RC⇥H⇥W extracted by a33 back o e CNN , a set of ttention guidance A = { Ac } l rned f c pt i fine-gra ned t i or d34 local patterns T ci from Ti . To d still semantical c es and further ge rate the fi al a tribut -aw re35 binary hash cod s , we propose to tr nsform these ppearance-specific f atures T tow rds se ntic-36 specific repres ntations T̂ erforming a tra sform n twork ( · ) . Aft aggr ga i g T̂ , the obtai d37 attentive local-level feat res xci are associated with the global-level featur x global i t f rm as a38 holistic e ture r pr s ntation xi . I order to g nerate a tribute- war binary hash codes , we condu t39 a econstructing p adigm to project xi as vi i an attri te space w re it da p i t co resp d t40 an ttribute tor w.r.t . o kin f n m ble r perti o fin in d bject . Fi ally , ash cod41 le rning is p rformed pon vi to obtain th fin l at t - war bi ry co es ui.42 3.2 Fi e-Grained Representation L rning43 Attention plays an important role in human perceptio [ 3 , 7 ] , and hu ans exploit a s qu nce of44 partial glimpses and selectiv ly focus on sali nt parts of an object or a scene in or e to b tt r capture45 visual structur [ 10 ] . Inspired by this , we incorporate he att ntion mech ism int represent tion46 learning to capture fin - rained l cal patterns f r disti guishing subtle differ n es betw e these47 s b rdi te ategories.48 In concr tely , w extract the de p feature of i s input image Ii vi a backbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Th n , based o i , C attention uidan e Ac 2 RH⇥W is g nerated as a set f att ntion m ps , i.e. , A.51 The atte ion i ance Ac is designed to evaluate which deep descriptors should b attended or ven52 overlooked by conducting53 T ci = c Ti , ( 2 ) where is the element-wise Hadamard product . To o tain the fina attribute-aware binary odes , it is54 desirable to tran form these appearance-specific ( i.e. , low-level ) features T to emantic-specific ( i . .,55 mid-level ) representations whi h are clos r to the attribute space . Thu , a tr nsforming network ( · ) ,56 which is equipped with a stack of convolution layers , is performed on T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) wher ✓ presents the parameters of the correspo ding transforming network w.r.t . T ci and i , respec-58 tively . Then , we ggrega e T̂ ci and T̂i by c ucting global av rage-pooli g nd corr pondingly59 o tain the attentive local-level featur s xci , as well a the global-l vel feature x global i . The holistic60 feature representation w.r.t . the i put mage Ii is chi ved by concatenating both xci and xgl bali , .e.,61 xi = h xci ; l al = F ( Ii ; ⇥ ) 2 Rd . Note that , we hereby abstract the aforementioned fine-grained62 feature learnin proc ss as a function F ( Ii ; ⇥ ) with its p ram t rs ⇥.63 2 2 Rel t d Work23 2.1 Fi -Grained Image Retrieval24 2.2 Learni g to Ha h25 2.3 Visu l Attributes26 3 Methodology27 In this section , we introduce the overall framework and notations of the proposed A2-NET model , as28 w ll as elaborating the detailed modules of A2-NET and its correspon ing optimization algorithm.29 3.1 Overall Fram work and Notation30 As illustrated in Figur ? ? , our A2-NET model consists of two cruc al mo ules , i.e. , fi e-grai ed31 r p tatio l arning modul nd an attribut - ware has codes generating module . Given an32 inp t image Ii , b sed o its respond ng deep a tivation tensor Ti 2 RC⇥H⇥W extracted by a33 ba kbone CNN , a set of attention g idance A = { Ac } is learned for captu ing fine-gr ined tailor d34 l l patterns T ci from Ti . To distill semantical cues and further en rate the final ttribute- ware35 binary hash odes , we prop se to transform these ppearance-specific features T towards semantic-36 specific representations T̂ by performing a transform netwo k ( · ) . After aggregat ng T̂ , he obt ined37 attentive local-level features xci re associated with the global-level feature x global i to form as a38 holistic feature representation xi . In order to generate attribute-aware binary hash codes , we onduct39 a reconstructing paradigm to project xi as vi in an attribute space where i s data po nt correspo d to40 an attribute vector w.r.t . one kind of namable properties of fine-grained obj cts . Finally , has c de41 learning is performed upon vi to obtain the fi al attribute-aware binary codes ui.42 3.2 Fine-Grained Representation Learning43 Attention plays a important role in human perception [ 3 , 7 ] , and humans exploit a sequence of44 partial glimpses and select vely focus on salient parts of an bject or a scene in order to be ter capture45 visual structure [ 10 ] . Inspired by this , we incorporate the attention mechanism into representation46 learning to c pture fine-grained local patterns for distinguishing subtle difference b tween th se47 subordinate c tegories.48 In concretely , we extract the deep feature of its input image Ii via a backbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Then , based on Ti , C attention guidance Ac 2 RH⇥W is generat d as a set of attention maps , i.e. , A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be tte d d or even52 overlooked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obtain the final attribute-aware binary codes , it is54 desirable to transform these appearance-specific ( i.e. , low-level ) features T to semantic-specific ( i.e.,55 mid-level ) representations which are closer to the attribute space . us , transf rming netw rk ( · ) ,56 which is equipped with a stack of convolution layers , is performed on T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ presents the parameters of the corresponding transformi g networks w.r. . T ci and Ti , r spec-58 tively . Then , we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . The holistic60 feature representation w.r.t . the input image Ii is achieved by concatenating both xci and xglobali , i . .,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 Rd . Note that , we her by abstr c the af rementioned fine-gr ined62 feature learning process as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section , we introduce the overall framework and notations of the proposed A2-NET model , as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorith .29 3.1 Overall Framework and Notations30 As illustrated in Figure ? ? , our A2-NET model cons sts of two crucial modules , i.e. , a fine-grain d31 representation learning module and an attribute-aware hash codes gene ating module . Given an32 input image Ii , based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN , a set of attention guidance A = { Ac } is learned for capturing fin -grained tail red34 local patterns T ci from Ti . To distill semantical cues and further generat the final at ribute-aware35 binary hash codes , we propose to transfor th se appearance-specific features T towards semantic-36 specific representations T̂ by performing a transform network ( · ) . Aft r aggregating T̂ , the obtained37 attentive local-level features xci are asso iated with the global-level feature x global i o form as a38 holistic featur representation xi . In ord r to generate attribute-awar binary hash codes , we conduct39 a reconstructing paradigm to project xi as vi in an attribute space wh re its data point corr spond to40 an attribute vector w.r.t . one kind of namable properties of fine-grained objects . Finally , hash code41 learning is performed upon vi to obtain the final attribute-aware binary codes ui.42 3.2 Fine-Grain d Representation Learning43 Attention plays an important role in human perception [ 3 , 7 ] , and humans exploit a sequence of44 partial glimps s and s lectively focus on s lient parts of an object or a scene in order to better capture45 visual structure [ 10 ] . Inspired by this , we incorporate the attention mechanism into repr sentation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48 In concretely , we extract the deep feature of its nput image Ii via a backbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Then , based on Ti , C attention guidance Ac 2 RH⇥W is generated as a set of attention maps , i.e. , A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadama d product . T obtain he final a tribute-aware binary cod s , it is54 desirable to transform these appearance-specific ( i.e. , low-level ) features T to semantic-specific ( i.e.,55 mid-level ) representations which are closer to the attribute space . Thus , a transforming netwo k ( · ) ,56 which is equipped with a stack of convolution layers , is performed on T as follows:57 T̂ ci = ( i ; local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ presents the parameters of the corresponding transforming networks w.r.t . T ci and Ti , respec-58 tively . Then , we aggregate T̂ ci and T̂i by onducting glob l average-poolin and correspondingly59 obtain the attentive local-level features xci , as w ll as the gl bal-level feature x global i . Th holistic60 feature representation w.r.t . the input image Ii is achiev d by concat nating both xci a d xglobali , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 Rd . Note that , we hereby abstract the aforementioned fine-grain d62 feature learning process as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 Aggregation where t e d coder ( i.e. , a counterpart of the encoder ) is realized by W > to simplify the netwo k.91 However , directly minimizing Eq . ( 6 ) with a hard constraint is hard to optimize . Therefore , we relax92 the constraint into a soft constraint , and the l rning objective ca be rewritten by93 min W kX W > V 0k2F + kWX V 0k2F . ( 7 ) 3.3.2 Attribute-Specifi F ur D orrelatio94 By conducting the afo em ntio d unsupervi ed att ibut -g ided learning , we an obtain the int rn l95 l tent vecto s V 0 as the ttribute-specific feature . I ord to b h e h nce th discrimi ativ bility96 and r move the r dundan correlation among thes ttribute-sp cific features , we introduce a feature97 decorrelation c n r int upon V 0 , which is formulat d by98 m n V 0 kV 0V 0 > Ik2F , ( 8 ) where I is the identity matrix . Such a featur ecor lation co strai t is pref rable to const uct99 independent features and reduce redu da t information . Therefore , ba ed on u supervised ttribut -100 guided learning d attribute-specific featur decorrelation , th fi a learn d hash codes a e expected101 to be bo h attribute-awar a d hash-bi ind pe d nt.102 3.3.3 H sh Co e L arning103 In the f ll wing , we conduct the hash code learning bas d on the obt ned at ribute-sp cific featur s.104 Assume that w have n qu ry data p ints which re enoted as { qi } i=1 , as w ll m databa points105 which a noted a { vj } j=1 . Note ha , both qi and vi ar belo gin o th att ibute space V . Th n,106 the corresponding bi ary has codes ca b obt ined via107 ui = sgn ( tanh ( qi ) ) , ( 9 ) zj = sgn ( tanh ( vj ) ) , ( 10 ) where ui , zj 2 { 1 , +1 } k. The g al f our h cod learning is to le r binary ash cod s f r t108 qu ry poin s and database points from { qi } =1 , { vj } mj=1 , and the pairwise sup vised information,109 i . . , S 2 { 1 , +1 } n m. To pr se ve the pair simil ri y , adopt t ` 2 loss b tween the110 supervised inf rmati ( .k.a . similarity ) d t inn r product f query-dat base point binary cod111 pairs . It can be formulated as follows:112 inU , Z P =1 P j=1 u > i zj kSij 2 s.t . U 2 { 1 , +1 } n⇥k , Z 2 { 1 , +1 } m⇥k . ( 11 ) Overall , e g t the fina obj tiv f t e propo ed A2-NET mod l by co sidering Eq . ( 7 ) , Eq . ( 8 ) and113 Eq . ( 11 ) toge a follows:114 min W , ⇥ L ( I ) = kX W > V 0k2F + kWX V 0k2F +↵kV 0V 0 > nIk2F + nX i=1 mX j=1 > i zj kSij 2 , ( 12 ) where , ↵ and are hyper-parameters as the tr de-of .115 In practice , during training , it ght b only ava labl et of at ase poin s { vj } mj=1 with ut query116 points . Thus , we randomly sample n ata point from database to construct a query set , and d te117 the indices of all the database points as with the indices of th query set as ⌦ . Additionally , because118 we can not back-propagate the gradient to ⇥ due to the sgn ( · ) function , w omit the sgn ( · ) function119 and o ly apply t nh ( · ) f r elaxation in Eq . ( 10 ) of the whole optimization process . Th refore , he120 optimization formulation of A2-NET can be rewritten with only database points { vj } mj=1 for rai ing121 as:122 min W , ⇥ L ( I ) = kX W > V 0k2F + kWX V 0k2F + ↵kV 0V 0 > nIk2F + X i2⌦ X j2 ⇣ tanh ( W · F ( Ii ; ⇥ ) ) zj kSij ⌘2 . ( 13 ) For optimization , our A2-NET does not require co plicated two-stage learning algorithms , e.g. , th123 alternative optimization strategy . W employ the back-propagation algorith and follow [ 8 ] to rain124 the whol A2-NET model in a u ified end-to-end a ner.125 4 “ Blue head ” Figure 1 : Overall fra ewo k of the roposed A2-NET m el , whic consis s f t o crucial modules , i.e. , fine-grained representation learni g and attribute-aware hash codes ge er ting . The whole netw rk c n be end-to-e d trainabl , d is gener lly riv n b th sup rvi d tr bu e-guided loss , the feature decorrelation loss and he hash cod le rn g loss , cf . S tio 3.3. has th potenti l to help with large-sc le fine- rai ed d ta t o . Therefore , targeting the ig data35 challenge , fine-grained hashing [ 7 , 13 ] is a promisi g direction orthy f further explorations.36 low-level to mid-level appearance-specific to s antic-specific37 obtain corresponding fine-grai ed obj ct attribut s ( e.g. , xx ) , ithout signifi ant annotati cost38 2 Relat d Work39 In this section , we briefly review t e r ated work i t following t ree aspects , i.e. , n -g ained40 image retrieval , learning to hash and the studies about visual attributes in co puter vision p oblems.41 2.1 Fine-Grained Image Retrieval42 Fine-grained image retrieval as an integral part of fine-grained image analysis [ 34 ] has gained more43 and more tractio i recent years [ 7 , 20 , 23 , 33 , 35 , 37 , 38 ] . What makes it challenging is that44 objects of fine-grained images have only subtle differences , and often largely vary n po e , scale , and45 orientation or can exhibit cross-modal differences ( e.g. , sketch-based retrieval [ 20 ] ) .46 Depending on the type of query image , the most studied areas of fine-grain d imag retrieval47 can be separated into two groups : fine-grained content-based image retrieval ( FG-CBIR ) and fine-48 grained sketch-based image retrieval ( FG-SBIR ) . More sp cifically , in FG-CBIR , unsupervis d49 learning based [ 33 ] and supervised l rning bas d methods [ 36–38 ] were developed from differe t50 perspectives for dealing with fine-grained retriev tasks , e.g. , localizing fine-grained parts [ 33 ] ,51 enhancing intra-class separability with inter-class compact ess [ 38 ] , and re uci g th confidence52 of the fine-grained predictions [ 36 ] , etc . While , FG-SBIR needs to not only capture ne-graine53 characteristics present in the sketches , but also possess the ability to traverse the sketc and image54 domain gap . In the literature of FG-SBIR , the earlier works , e.g. , [ 17 , 27 , 35 ] , were mostly based55 on Siamese-triplet networks [ 3 ] to tackle the aforementioned c allenges . Recently , some works56 tried to incorporate the advances of recent progress in self-supervised learning [ 5 ] and attention57 mechanisms [ 6 ] for further improving the retrieval accuracy of FG-SBIR , e.g. , [ 23 , 25 ] .58 However , although these fine-grained retrieval methods achieved good results , they till show the59 limitations in the face of large-scale data , i.e. , the searching time for exact nearest neighbor is typically60 expensive or even impossible for the given queries . To alleviate this issue , fine-gr ined hashing , which61 aims to generate compact binary codes to represent fine-grained images , as a promising direction62 has attracted the attention in the fine-grained community very recently [ 7 , 14 ] . More sp cifically,63 2 R d_head ? Blue_tail ? O _the_tr e ? Spotted_body ? YES NO YES YES 1 0 1 1 Att ibute- ware has cod s R t ieval Top-k retriev d images 1 0 1 1 1 0 1 1 1 0 1 0 Figure 1 : K y idea of t e propos A2-NET mo l , s w ll th ain proces of fine rai d ha hing ased o our attribute-aware has c des . In co cr t ly , for a qu y i Iq of R b llied Woodpecker , aft r r tur ing all the correct re ults , t fin -grained i age ( b lon i g to R d h d Wo dpecker ) closest to t e qu ry image is al o r triev . A2-NET : Le r ing Attri u -A re Has C es or L g -Scal Fi -Gr I ge Ret i v l Anonymous Author ( s ) Affiliation Address email Abst ct The task of fine-grained image retrieval aims to r nk all th instances so that images1 depicting the concept of interests ( i.e. , the same sub-category labels ) are ranked2 highest based on the fine-grained details in the query . It is challenging due to the3 subtle differences between sub-category , as well as the fine-grained nature of small4 inter-class variations and large intra-class variations . Especially for large-scale5 data , it is desirable to be ha dled in a bo h effective and efficient way . In this paper,6 we an Attribute-Aware hashing Network , i . . , A2-NET , for generating an attribute-7 aware binary codes to not only speed up the retrieval process , but also corresponds8 to fine-grained attributes . Specifically , based on the learned attention maps , we9 design an auto-encoder to project the visual feature vector into an attribute space10 by following a reconstructing paradigm . Furthermore , A2-NET is also equipped11 with an orthogonal constraint upon the generated attributes to enhance their own12 representation abilities . Qualitative experiments o five be c mark fine-grained13 datasets show our superiority over competing methods . Moreov r , quantitative14 results reveal the obtained hash codes correspond t the attribut s of fine-grained15 patterns.16 1 Introduction17 another barrier also occurs because of xxx18 suffer from slow query speed and redundant storage costs due to both the explosive growth of massive19 fine-grained data and high-dimensional real-valued features.20 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . F gur 1 : K y id a f ur A2-NET , as w ll s th mai oc s of fine-grained hashing based on ur t ibut - w h sh c des . I conc t ly , regar ing a query image Iq of Red bellied Wo d ck , af er ret r i g all the corr ct results , a fine-gra n d image b longing to Red headed Wo dp ck r closes to th qu y ag in erms of H mm distance s also retriev d . .g. , [ 8 , 18 ] , chi v g o retri val perf a c , the bit f th ir hash codes correspond to no seman ics , i.e. , fi -gra ned at ributes . While , such t ributes , e.g. , head color , ail color , male , f ale , living habits , a e great means of des ribing fine-grained objects , in a way both humans and computers understand . In this paper , to st blish an explicit correspondence between hash codes and visu l attri ut s for not only fu the impr vi g rge-scal fine-grained retrieval accuracy , but more importa tly integrati g interpr tation into deep l arning based hash methods , we propose a unified Attribute-Aw r hashing Ne work , te med s A2-NET ( cf . Figure 1 ) , for achieving these goals . In ou A2-NET , on d ring huge l b r cost f supervised attribute annotations , we restrict ourselves in an unsup rvised setting to automatically c pture di criminative visual attributes from still images an the correspond the final learned hash code representatio s to these attributes . Therefore , a has bit of l ar ed hash co s could be both d scriminative and intuitive . Additionally , thanks to h u supervis d setting , the attributes derived from A2-NET will be not restricted to pre-defined attributes like supervise -based attribut learning methods [ 15 , 21 , 42 , 45 ] . Moreover , it can distill the most useful properties of fine-grained objects as attribute-aware hash codes in such an end-to-end trai able m ner for accuracy retrieval among multiple similar subordinate categories . More specific lly , as the overall framework shown in Figure 2 , our A2-NET consists of a finegrai ed represe tation learni g module and an a tribute-aware hash codes generating module . It first leverages attention mechanisms to model fine-grained tailored patterns in terms of both global-level deep featur s Ti and local-level ues T ci from inp t imag Ii . Then , the appearance-specific features of th se vis l p t rns T are aggr gated and tran lat d into semantic-specific representations xi . After that , we formulate the aforementioned unsupervised attribute learning as a reconstruction task of projecting xi to an attribute vector vi by performing an encoder-decoder structure network . T erefor , it can be xpected that in the high-level at ribute space , vi could correspond to certain kinds of nameable properties of fine-grained objects . Moreover , a feature decorrelation constraint is further introduced upon vi to both enhance t e discriminative bility and remove the redundant correlation among these dimensions of attribute-specific features . Finally , our attribute-aware hash codes ui are enerated from vi by conducting the hash code learning procedure . To evaluate our model , we conduct extensive experiments using five benchmark fine-grained retrieval datasets for both accuracy and interpretability . Quantitative results of retrieval accuracy on these datasets show that the proposed A2-NET model obviously and consistently outperforms existing stateof-the-art methods . Qualitative visualization of the obtained attribute-aware hash codes demonstrates that these hash bits have strong correspondences to visual attributes of fine-grained objects ( cf . Figure 4 ) , even without employing attribute supervisions or part-level annotations . In addition , the ablation studies of these crucial components in A2-NET also validate their own effectiveness . 2 Related Work 2.1 Fine-Grained Image Retrieval Fine-grained image retrieval as an integral part of fine-grained image analysis [ 41 ] has gained more and more traction in recent years [ 8 , 25 , 29 , 40 , 43 , 47 , 48 ] . What makes it challenging is that 101 1 Encoder 3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET model . We elaborate it in65 the following four aspects , i.e. , unsupervised attribute-guided learning , attribute-specific feature66 decorrelation , hash code learning and the optimization learning algorithm.67 3.3.1 Unsupervised Attribute-Guided Learning68 In real-applications , especially for the large-scale and fine-grained tasks , attribute annotations are69 always infeasible , which limits the learning process to be conducted in an unsupervised setting.70 While , in the literature , the main goal of unsupervised learning is to capture regularities in data for the71 purpose of extracting useful representations or for restoring corrupted data [ 4 ] . Many unsupervised72 methods explicitly produce internal latent units or codes , from which the data is to be reconstructed.73 Inspired by this , we develop an unsupervised attribute-guided learning component to project the74 holistic representation xi of Ii into a latent space , i.e. , the attribute space V . In V , its high-level75 vectors are designed to have certain desirable properties , e.g. , corresponding to semantic properties76 of fine-grained objects ( a.k.a . “ fine-grained attributes ” ) .77 More specifically , in our A2-NET , the unsupervised attribute-guided learning is realized by a recon-78 struction paradigm with an encoder-decoder structure , as shown in Figure ? ? . In concretely , given79 a batch of n training data Ii , the holistic representations X = { x1 ; x2 ; . . . ; xn } 2 Rd⇥n can be80 obtained as aforementioned . By formulation , the encoder projects X into the attribute space V with81 a projection matrix W 2 Rk⇥d to get an internal latent representation V 2 Rk⇥n w.r.t . X . In82 particularly , we set that the dimension of latent representation k equals to the number of hash bits in83 the final binary hash code ui . Furthermore , each column of V , i.e. , vi 2 Rk , can derive ui by84 ui = sgn ( tanh ( vi ) ) . ( 5 ) Meanwhile , regarding vi , we also consider to reconstruct its input xi by a decoder as a counterpart85 of the encoder . Therefore , on one hand , such a reconstruction paradigm can reduce and further distill86 high-level semantic cues in the attribute space V . While , on the other hand , it can drive the training87 of A2-NET by preserving the similarity between queried hash codes and database points in terms of88 learning to hash ( cf . Section 3.3.3 ) .89 In concretely , the learning objective of unsupervised attribute-guided learning is written as follows:90 min W kX W > WXk2F s.t . WX = V 0 = tanh ( V ) , ( 6 ) where the decoder ( i.e. , a counterpart of the encoder ) is realized by W > to simplify the network.91 However , directly minimizing Eq . ( 6 ) with a hard constraint is hard to optimize . Therefore , we relax92 the constraint into a soft constraint , and the learning objective can be rewritten by93 min W kX W > V 0k2F + kWX V 0k2F . ( 7 ) 3.3.2 Attribute-Specific Feature Decorrelation94 By conducting the aforementioned unsupervised attribute-guided learning , we can obtain the internal95 latent vectors V 0 as the attribute-specific features . In order to both enhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features , we introduce a feature97 decorrelation constraint upon V 0 , which is formulated by98 min V 0 kV 0V 0 > kIk2F , ( 8 ) where I is the identity matrix . Such a feature decorrelation constraint is preferable to construct99 independent features and reduce redundant information . Therefore , based on unsupervised attribute-100 guided learning and attribute-specific feature decorrelation , the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102 3.3.3 Hash Code Learning103 In the following , we conduct the hash code learning based on the obtained attribute-specific features.104 Assume that we have n query data points which are denoted as { qi } ni=1 , as well as m database points105 3 3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET model . We elaborate it in65 the following four aspects , i.e. , unsupervised attribute-guided learning , attribute-specific feature66 decorrelation , hash code learning and the optimization learning algorithm.67 3.3.1 Unsupervised Attribute-Guided Learning68 In real-applications , especially for the large-scale and fine-grained tasks , attribute annotations are69 always infeasible , which limits the learning process to be conducted in an unsupervised setting.70 While , in the literature , the main goal of unsupervised learning is to capture regularities in data for the71 purpose of extracting useful representations or for restoring corrupted data [ 4 ] . Many unsupervised72 methods explicitly produce internal latent units or codes , from which the data is to be reconstructed.73 Inspired by this , we develop an unsupervised attribute-guided learning component to project the74 holistic representation xi of Ii into a latent space , i.e. , the attribute space V . In V , its high-level75 vectors are designed to have certain desirable properties , e.g. , corresponding to semantic properties76 of fine-grained objects ( a.k.a . “ fine-grained attributes ” ) .77 More specifically , in our A2-NET , the unsupervised attribute-guided learning is realized by a recon-78 struction paradigm with an encoder-decoder structure , as shown in Figure ? ? . In concretely , given79 a batch of n training data Ii , the holistic representations X = { x1 ; x2 ; . . . ; xn } 2 Rd⇥n can be80 obtained as aforementioned . By formulati n , th en oder pr jects X into the attribute sp ce V with81 a projection matrix W 2 Rk⇥d to get an internal latent representation V 2 Rk⇥n w. .t . X . In82 particularly , we set that the dimension of latent representation k equals to the number of hash bits in83 the final binary hash code ui . Furthermore , each column of V , i.e. , vi 2 Rk , ca derive ui by84 ui = sgn ( tanh ( vi ) ) . ( 5 ) Meanwhile , regarding vi , we also consider to reconstruct its inp t xi by a decoder as a cou terpart85 of the encoder . Therefore , on one hand , such a reconstruction paradigm can reduce and further distill86 high-level semantic cues in the attribute space V . While , on the other hand , it can drive the training87 of A2-NET by preserving the similarity between queried hash cod s and database points in terms of88 learning to hash ( cf . Section 3.3.3 ) .89 In concretely , the learning objective of unsupervised attribute-guided learning is written as follows:90 min W kX W > WXk2F s.t . WX = V 0 = tanh ( V ) , ( 6 ) where the decoder ( i.e. , a counterpart of the encoder ) is realized by W > to simplify the network.91 However , directly minimizing Eq . ( 6 ) with a hard constraint is hard to optimize . Therefore , we relax92 the constraint into a soft constraint , and the learning objective can be rewritten by93 min W kX W > V 0k2F + kWX V 0k2F . ( 7 ) 3.3.2 Attribute-Specific Feature Decorrelation94 By conducting the aforementioned unsupervised attribute-guided learning , we can obtain the internal95 latent vectors V 0 as the attribute-specific features . In order to both nhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features , we introduce a feature97 decorrelation constraint upon V 0 , which is formulated by98 mi V 0 kV 0V 0 > kIk2F , ( 8 ) where I is the identity matrix . Such a feature decorrelation constraint is preferable to construct99 independent features and reduce redundant information . Therefore , based on unsupervised attribute-100 guided learning and attribute-specific feature decorrelation , the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102 3.3.3 Hash Code Learning103 In the following , we conduct the hash code learning based on the obtained attribute-specific features.104 Assume that we have n query data points which are denoted as { qi } ni=1 , as well as m database points105 3 3.3 Attribute-Aware Hash C es Gen ating64 How to generate attribute-aware hash codes is the key of our A2-NET model . We elaborate it in65 the following four aspec s , i.e. , unsup rvised attribute-guided learning , attribute-specific feature66 decorrelati n , hash code learning and the optimization learning algorithm.67 3.3.1 Unsupervised Attribute-Guided Learning68 In real-applications , especially for the large-scale and fine-grained tasks , attribute annotations are69 always infeasible , which limits the learning process to be conducted in an unsupervised setting.70 While , in the literature , the main goal of unsupervised learning is to capture regularities in data for the71 purpose of extracting useful representations or for restoring corrupted data [ 4 ] . Many unsupervised72 methods explicitly produce internal latent u its or codes , from which the data is to be reconstructed.73 Inspired by this , we develop an unsupervised attribute-guided learning component to project the74 holistic representation xi of Ii into a latent space , i.e. , the attribute space V . In V , its high-level75 vectors are designed to have certain desirable properties , e.g. , corresponding to semantic properties76 of fine-grained objects ( a.k.a . “ fine-grained attributes ” ) .77 More specifically , in our A2-NET , the unsupervised attribute-guided learning is realized by a recon-78 structi n paradigm with an encoder-decod r structure , as shown in Figure ? ? . In concretely , given79 a batch of n training data Ii , the holistic representations X = { x1 ; x2 ; . . . ; xn } 2 Rd⇥n can be80 obtained as aforementioned . By formulation , the encoder projects X into the attribute space V with81 a projectio matrix W 2 Rk⇥d to get an nternal latent representation V 2 Rk⇥n w.r.t . X . I82 particularly , w set that the dimension of latent repre entation k equals t the number of hash bits in83 the final binary sh de ui . Furthermore , each column of V , i.e. , vi 2 Rk , can derive ui by84 ui = sgn ( tanh ( vi ) ) . ( 5 ) Meanwhile , regarding vi , we also consider to reconstruct its input xi by a decoder as a counterpart85 of the encod r. Therefore , on one hand , uch a reconstructio paradigm can reduce and further distill86 high-level semantic cues in the attribute space V . While , on the other hand , it can drive the training87 of A2-NET by preserving the similarity between queried hash codes and database points in terms of88 learning to hash ( cf . Section 3.3.3 ) .89 In con retely , the learn ng objective of u supervised attribute-guided learning is written as follows:90 min W kX W > WXk2F .t . WX = V 0 = tanh ( V ) , ( 6 ) where the decoder ( i.e. , a counterpart f the encoder ) is r alized y W > to simplify the network.91 However , irectly minimizi g Eq . ( 6 ) with a hard con traint is hard to optimize . Therefore , we relax92 the constraint into a soft constraint , and the learning objective can be rewritten by93 min W kX W > V 0k2F + kWX V 0k2F . ( 7 ) 3.3.2 Attribute-Specific Feature Decorrelation94 By conducting the aforementioned unsupervised attribute-guided learning , we can obtain the internal95 latent vectors V 0 as the attribute-specific features . In order to both enhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features , we introduce a feature97 decorrelation constraint upon V 0 , which is formulated by98 min V 0 kV 0V 0 > kIk2F , ( 8 ) where I is the identity matrix . Such a feature decorrelation constraint is preferable to construct99 independent features and reduce redundant information . Therefore , based on unsupervised attribute-100 guided learning and attribute-specific feature decorrelation , the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102 3.3.3 Hash Code Learning103 In the following , we conduct the hash code learning based on the obtained attribute-specific features.104 Assume that we have n query data points which are denoted as { qi } ni=1 , as well as m database points105 3 3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET model . We elaborat it in65 the following four aspects , i.e. , unsupervised attribute-guided learning , attribute-specific feature66 decorrelation , hash code learning and the optimization learning algorithm.67 3.3.1 Unsupervised Attribute-Guided Learning68 In real-applications , especially for the large-sca e and fine-grained tasks , attribute an otation re69 always infeasible , which limits the learning process to be nducted in an unsupervised setting.70 While , in the literature , the main goal of unsupervised learning is to capture regularities in data for the71 purpose of extracting useful representations or for restoring corrupted data [ 4 ] . Many unsupervised72 methods explicitly produce internal latent units or codes , from which the data is to be reconstructed.73 Inspired by this , we develop an unsupervised attribute-guided learning component to project the74 holistic representation xi of Ii into a latent space , i.e. , the attribute sp ce V . In V , its high-lev l75 vectors are designed to have certain desirable properties , e.g. , corresponding to semantic properties76 of fine-grained objects ( a.k.a . “ fine-grained attributes ” ) .77 More specifically , in our A2-NET , the unsupervised attribute-guided learning is realized by a recon-78 struction paradigm with an encoder-decoder structure , as shown in Figure ? ? . In concretely , given79 a batch of n training data Ii , the holistic representations X = { x1 ; x2 ; . . . ; xn } 2 Rd⇥n can be80 obtained as aforementioned . By formulation , the encoder projects X into the attribute space V with81 a projection matrix W 2 Rk⇥d to get an internal latent representation V 2 Rk⇥n w.r.t . X . In82 particularly , we set that the dimension of latent representation k equals to the number of hash bits in83 the final binary hash code ui . Furthermore , each column of V , i.e. , vi 2 Rk , can derive ui by84 ui = sgn ( tanh ( vi ) ) . ( 5 ) Meanwhile , regarding vi , we also consider to reconstruct its input xi by a decoder as a counterpart85 of the encoder . Therefore , on one hand , such a reconstruction paradigm can reduce and further distill86 high-level semantic cues in the attribute space V . While , on the other hand , it can drive the training87 of A2-NET by preserving the similarity between queried hash codes and database points in terms of88 learning to hash ( cf . Section 3.3.3 ) .89 In concretely , the learning objective of unsupervised attribute-guided learning is written as follows:90 min W kX W > WXk2F s.t . WX = V 0 = tanh ( V ) , ( 6 ) where the decoder ( i.e. , a counterpart of the encoder ) is realized by W > to simplify the network.91 However , directly minimizing Eq . ( 6 ) with a hard constraint is hard to optimize . Therefore , we relax92 the constraint into a soft constraint , and the learning objective can be rewritten by93 min W kX W > V 0k2F + kWX V 0k2F . ( 7 ) 3.3.2 Attribute-Specific Feature Decorrelation94 By conducting the aforementioned unsupervised attribute-guided learning , we can obtain the internal95 latent vectors V 0 as the attribute-specific features . In order to both enhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features , we introduce a feature97 decorrelation constraint upon V 0 , which is formulated by98 min V 0 kV 0V 0 > kIk2F , ( 8 ) where I is the identity matrix . Such a feature decorrelation constraint is pref rable to construct99 independent features and reduce redundant information . Therefore , based on unsupervise ttri t -100 guided learning and attribute-specific feature decorrelation , the final learned hash codes are expected101 to be both attribute-aware and hash-bit independent.102 3.3.3 Hash Code Learning103 In the following , we conduct the hash code learning based on the obtained attribute-sp cific feat res.104 Assume that we have n query data points which are denoted as { qi } ni=1 , as well as m database points105 3 Decoder 3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET model . We elaborate it in65 the following four aspects , i.e. , unsupervised attribute-guided learning , attribute-specific feature66 decorrelation , hash code learning and the optimization learning algorithm.67 3.3.1 Unsupervised Attribute-Guided Learning68 In real-applications , especially for the large-scale and fine-grained tasks , attribute annotations are69 always infeasible , which limits the learning process to be conducted in an unsupervised setting.70 While , in the literature , the main goal of unsupervised learning is to capture regularities in data for the71 purpose of extracting useful representations or for restoring corrupted data [ 4 ] . Many unsupervised72 methods explicitly produce internal latent units or codes , from which the data is to be reconstructed.73 Inspired by this , we develop an unsupervised attribute-guided learning component to project the74 holistic representation xi of Ii into a latent space , i.e. , the attribute space V . In V , its high-level75 vectors are designed to have certain desirable properties , e.g. , corresponding to semantic properties76 of fine-grained objects ( a.k.a . “ fine-grained attributes ” ) .77 More specifically , in our A2-NET , the unsupervised attribute-guided learning is realized by a recon-78 struction paradigm with an encoder-decoder structure , as shown in Figure ? ? . In concretely , given79 a batch of n training data Ii , the holistic representations X = { x1 ; x2 ; . . . ; xn } 2 Rd⇥n can be80 obtained as aforementioned . By formulation , the encoder projects X into the attribute space V with81 a projection matrix W 2 Rk⇥d to get an internal latent representation V 2 Rk⇥n w.r.t . X . In82 particularly , we set that the dimension of latent representation k equals to the number of hash bits in83 the final binary hash code ui . Furthermore , each column of V , i.e. , vi 2 Rk , can derive ui by84 ui = sgn ( tanh ( vi ) ) . ( 5 ) Meanwhile , regard ng vi , we lso consider to reco struct i s input xi by a decod r as a counterpart85 of the ncoder . Therefore , on one ha d , such a rec nstruction paradigm can reduce and further distill86 high-level semantic cues in the attribute space V . While , on the other hand , it can drive the training87 of A2-NET by preserving the similarity between queried hash codes and database points in terms of88 learning to hash ( cf . Section 3.3.3 ) .89 In concretely , the learni g objective of unsupervised attribute-guided learni g is written as follows:90 min W kX W > WXk2F s.t . WX = V 0 = tanh ( V ) , ( 6 ) where the decoder ( i.e. , a counterpart of the encod r ) is realized by W > to simplify the network.91 How ve , directly minimizing Eq . ( 6 ) with a hard constraint is hard to optimize . Therefore , we relax92 the constraint into a soft constraint , nd the learning objective can be rewritten by93 min W kX W > V 0k2F + kWX V 0k2F . ( 7 ) 3.3.2 Attribute-Specific Feature Decorrelation94 By conducting the aforementioned unsupervised attribute-guided learning , we can obtain the internal95 latent vectors V 0 as the attribute-specific features . In order to both enhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features , we introduce a feature97 decorrelation constraint upon V 0 , which is formulated by98 min V 0 kV 0V 0 > kIk2F , ( 8 ) where I is the identity matrix . Such a feature decorrelation constraint is preferable to construct99 independent features and reduce redundant information . Therefore , based on unsupervised attribute-100 guided learning and attribute-specific feature decorrelation , the final learned hash codes are expected101 to be both attribute-aware and hash-bit independ nt.102 3.3.3 Hash Code Learning103 In the following , we conduct the hash code learning based on the obtained attribute-specific features.104 Assume that we have n query data points which are denoted as { qi } ni=1 , as well as m database points105 3 which are denoted as { vj } mj=1 . Note that , both qi and vi are belonging to the attribute space V . Then,106 the corresponding binary hash codes can be obtained via107 ui = sgn ( tanh ( qi ) ) , ( 9 ) zj = sgn ( tanh ( vj ) ) , ( 10 ) where ui , zj 2 { 1 , +1 } k. The goal of our hash code learning is to learn binary hash codes for both108 query points and database points from { qi } ni=1 , { vj } mj=1 , and the pairwise supervised information,109 i.e. , S 2 { 1 , +1 } n⇥m . To preserve the pairwise similarity , we adopt the ` 2 loss between the110 supervised information ( a.k.a . similarity ) and the inner product of query-database point binary code111 pairs . It can be formulated as follows:112 minU , Z Pn i=1 Pm j=1 u > i zj kSij 2 s.t . U 2 { 1 , +1 } n⇥k , Z 2 { 1 , +1 } m⇥k . ( 11 ) Overall , we get the final objective of the proposed A2-NET model by considering Eq . ( 7 ) , Eq . ( 8 ) and113 Eq . ( 11 ) together as follows:114 min W , ⇥ L ( I ) = kX W > V 0k2F + kWX V 0k2F +↵kV 0V 0 > nIk2F + nX i=1 mX j=1 u > i zj kSij 2 , ( 12 ) where , ↵ and are hyper-parameters as the trade-off.115 In practice , uri g training , it might be only available a set of database points { vj } mj=1 without query116 points . Thus , we randomly sample n data points from database to construct a query set , and denote117 the indices of all the database points as with the indices of the query set as ⌦ . Additionally , because118 we can not back-propagate the gradient to ⇥ due to the sgn ( · ) function , we omit the sgn ( · ) function119 and only apply tanh ( · ) for relaxation in Eq . ( 10 ) of the whole optimization process . Therefore , the120 optimization formulation of A2-NET can be rewritten with only database points { vj } mj=1 for training121 as:122 min W , ⇥ L ( I ) = kX W > V 0k2F + kWX V 0k2F + ↵kV 0V 0 > nIk2F + X i2⌦ X j2 ⇣ tanh ( W · F ( Ii ; ⇥ ) ) > zj kSij ⌘2 . ( 13 ) For optimization , we employ the back-propagation algorithm and follow [ 8 ] to train the whole123 A2-NET m l in an end-to-end manner.124 3.4 Out-of-Sample Extension125 After training A2-NET , the learned model can be applied for generating binary codes for query points126 including unseen query points in th training phase . Specifically , we can use the following equation127 to generate the binary code for Iq:128 uq = sgn ( tanh ( W · F ( Iq ; ⇥ ) ) ) . ( 14 ) 4 Experiments129 In this section , we present the implementation details , empirical settings , main results , ablation studies130 and also some interesting qualitative a alyses of the learned attribute-aware binary hash codes.131 4.1 Datasets132 x0i By following ExchNet [ 4 ] , our experiments are conducted on five fine-grained benchmark datasets,133 i.e. , CUB-200-2011 [ 16 ] , Aircraft [ 12 ] , Food101 [ 1 ] , NABirds [ 15 ] and VegFru [ 6 ] . Specifically,134 CUB-200-2011 is one of the most popular used fine-grained datasets . It contains 11,788 bird images135 from 200 bird species and is officially split into 5,994 images for training with 5,794 images for136 test . Aircraft contains 10,000 images spanning 100 aircraft models with 3,334 for training , 3,333137 4 Fine-grained representation learning U ns up er vi se d at tr ib ut e- gu id ed lo ss ( S ec tio n 3 . 3 . 1 ) Feature ecorrelation loss ( S ction 3.3.2 ) 3.3 Attribute-Aware Hash Codes Generating64 How to generate attribute-aware hash codes is the key of our A2-NET model . We laborate it in65 the foll wing four aspects , i.e. , unsupervised attribute-guided learning , attribute-specifi feature66 decorrelation , hash code learning and the optimization learning algorithm.67 3.3.1 Unsupervised Attribute-Guided Learning68 In real-applications , especially for the large-scale and fine-grained tasks , attribute annotations are69 always infeasibl , which limits the learning process to be conducted in an unsupervised setti g.70 While , in the literature , the main goal of unsupervised learning is to capture regularities in data for the71 purp se of xtracting useful representations or for restoring corrupted data [ 13 ] . Many unsupervised72 methods explicitly produce internal latent units or codes , from which the data is to be reconstructed.73 Inspired by this , we develop an unsupervised attribute-guided learning component to project the74 holistic r presentation xi of Ii into a latent space , i.e. , the attribute space V . In V , its high-level75 vectors are designed to have certain desirable properties , e.g. , corresponding to semantic properties76 of fine-grained objects ( a.k.a . “ fine-grained attributes ” ) .77 More specifically , in our A2-NET , the unsupervised attribute-guided learning is realized by a recon-78 struction paradigm with an encoder-decoder structure , as shown in Figure ? ? . In concretely , given79 a batch of n training data Ii , the holistic representations X = { x1 ; x2 ; . . . ; xn } 2 Rd⇥n can be80 obtained as aforementioned . By formulation , the encoder projects X into the attribute space V with81 a projection matrix W 2 Rk⇥d to get an internal latent representation V 2 Rk⇥n w.r.t . X . In82 particularl , we set tha the dimension of lat nt representation k quals to the number of hash bits in83 the final binary hash code ui . Furthermore , each column of V , i.e. , vi 2 Rk , can derive ui by84 ui = sgn ( tanh ( vi ) ) . ( 5 ) Meanwhile , regarding vi , we als consider to reconstruct its input xi by decoder as a counterpart85 of the encoder . Therefore , o one hand , such a reconstruction paradigm can reduce and further distill86 high-level semantic cues in the attribute space V . While , on the other hand , it can drive the training87 of A2-NET by preserving the similarity between queried hash codes and database points in terms of88 learning to hash ( cf . Section 3.3.3 ) .89 In concretely , the learning objective of unsupervised attribute-guided learning is written as follows:90 min W kX W > WXk2F s. . WX = V 0 = tanh ( V ) , ( 6 ) where the decoder ( i.e. , a counterpart of the encoder ) is realized by W > to simplify the network.91 Howev r , directly minimizing Eq . ( 6 ) with a hard constraint is hard to optimize . Therefore , we relax92 the constraint into a soft constraint , and the learning objective can be rewritten by93 min W kX W > V 0k2F + kWX V 0k2F . ( 7 ) 3.3.2 Attribute-Specific Feature Decorrelation94 By conducting the aforementioned u supervised attribute-guided learning , we can obtain the internal95 latent vectors V 0 as the attribute-specific features . In order to both enhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features , we introduce a feature97 d correlation constraint upon V 0 , which is formulated by98 min V 0 kV 0V 0 > nIk2F , ( 8 ) where I is the i entity matrix . Such a feature decorrelation constraint is preferable to construct99 independent features and reduce redundant information . Therefore , b sed on unsupervise attribute-100 guided learning and attribute-specific feature decorrelation , the final learned hash codes are expected101 to be both attri ute-aware an hash-bit indepe dent.102 3.3.3 Hash Code Learning103 In the following , we conduct the hash code lear ing based on the obtained attribute-specific features.104 Assume that we have n query data points which are denoted as { qi } ni=1 , s well as m database points105 3 Hash code learning loss ( Section 3.3.3 ) Attribute-aware hash codes generating “ Red head ” “ On the tree ” “ Spotted body ” 2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section , we introduce the overall framework and notations of the proposed A2-NET model , as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29 3.1 Overall Framework and Notations30 As illustrated in Figure ? ? , our A2-NET model consists of two crucial modules , i.e. , a fine-grain d31 representation learning module and an attribute-aware hash codes generating module . Given an32 input image Ii , based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN , a set of attention guidance A = { Ac } is learned for capturing fine-grained tailored34 local patterns T ci from Ti . To distill semantical cues and further generate the final attribute-aware35 binary hash codes , we propose to transform these appearance-specific features T towards semantic-36 specific representations T̂ by performing a transform network ( · ) . After aggregating T̂ , the obtained37 attentive local-level features xci are associated with the global-level feature x global i to f rm as a38 holistic feature representation xi . In order to generate attribute-aware binary hash codes , we conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond o40 an attribute vector w.r.t . one kind of namable properties of fine-grained objects . Finally , hash code41 learning is performed upon vi to obtain the final attribute-a are binary codes ui.42 3.2 Fine-Grained Representation Learning43 Attention plays an important role in human perception [ 3 , 7 ] , and humans exploit a sequence of44 partial glimpses and s lective y focus on salient parts of an object or a s ene in order to bet er capture45 visual structure [ 10 ] . Inspired by this , we incorporat the attention mechanism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differ nces between these47 subordinate categories.48 In concretely , we extract the deep feature of its input image Ii via a backbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Then , based on Ti , C attention guidance Ac 2 RH⇥W is generated as a set of attention maps , i.e. , A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obtain the final attribu e-aware binary co es , it is54 desirable to transform these appearance-specific ( i.e. , low-level ) f atures T to semantic-specific ( i.e.,55 mid-level ) representations which are closer to the attribute space . Thus , a transforming network ( · ) ,56 which is equipped with a stack of convolution layers , is performed on T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ presents the para eters of the corresponding transfor ing networks w.r.t . T ci and Ti , respec-58 tively . Then , we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . The holistic60 feature representation w.r.t . the input image Ii is achieved by concatenating both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 R . Note that , we hereby abstract the aforementioned fine-grai ed62 feature learning process as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 Attention guidance 2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section , we introduce the overall framework and notations of the proposed A2-NET model , as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29 3.1 Overall Framework and Notations30 As illustrated in Figure ? ? , our A2-NET model consists of two crucial modules , i.e. , a fine-graine31 representation learning module and an attribute-aware hash codes generating module . Given an32 input image Ii , based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN , a set of attention guidance A = { Ac } is le rned for capturing fi -grained tailored34 local patterns T ci from Ti . To distill semantical cues and further generate the final attribute-aware35 binary hash c des , we propose to transform these appearance-specific features T towa ds semantic-36 specific representations T̂ by performing a transform network ( · ) . After aggregating T̂ , the obt ined37 attentive local-level features xci are associated with the global-level feature x global i to form as a38 holistic feature representation xi . In order to generate attribute-aware binary hash codes , we conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond to40 an attribute vector w.r.t . one kind of namable properties of fine-grained objects . Finally , hash code41 learning is performed upon vi o obtain the final attribute-aware binary codes ui.42 3.2 Fine-Grained Representation Learning43 Attention plays a imp rtant role in human perception [ 3 , 7 ] , and humans exploit a sequence of44 artial glimpses d selectively focus on salient parts of an object or a scene in or er to better capture45 visual structure [ 10 ] . Inspired by this , we incorporate the attention mechanism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48 In concretely , we extract the deep feature of its input image Ii via a backbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Then , based on Ti , C attention guidance Ac 2 RH⇥W is g nerated as a set of attention ma s , i.e. , A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obtain the final attribute-aware binary codes , it is54 desirable to transform these appearance-specific ( i.e. , low-level ) features T to semantic-specific ( i.e.,55 mid-level ) representations which are closer to the attribute space . Thus , a transforming network ( · ) ,56 which is equipped with a stack of convolution layers , is performed on T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ presents the parameters of the corresponding transforming networks w.r.t . T ci and Ti , respec-58 tively . Then , we aggregate T̂ ci and T̂i by conducting global average-pooling d correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . The holistic60 feature representation w.r.t . the input image Ii is achieved by concatenati g both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 Rd . Note that , we hereby abstract the aforementioned fine-grained62 feature learning process as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section , we introduce the overall framework and notations of the proposed A2-NET model , as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29 3.1 Overall Framework and Notations30 As illustrated in Figure ? ? , our A2-NET model consists of two crucial modules , i.e. , a fine-grained31 representation learning module and an attribute-aware hash codes generating module . Given an32 input image Ii , based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN , a set of attention guidance A = { Ac } is learned for capturing fine-grained tailored34 local patterns T ci from Ti . To distill semantical cues and further generate the final attribute-aware35 binary hash codes , we propose to transform these appearance-specific features T towards semantic-36 specific representations T̂ by performing a transform network ( · ) . After aggregating T̂ , the obtained37 attentive local-level features xci are associated with the global-level feature x global i to form as a38 holistic feature representation xi . In order to generate attribute-aware binary hash codes , we conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond to40 an attribute vector w.r.t . one kind of namable properties of fine-grained objects . Finally , hash code41 learning is performed upon vi to obtain the final attribute-aware binary codes ui.42 3.2 Fine-Grained Representation Learning43 Attention plays an important role in human perception [ 3 , 7 ] , and humans exploit a sequence of44 partial glimpses and selectively focus on salient parts of an object or a scene in order to better capture45 visual structure [ 10 ] . Inspired by this , we incorporate the attention mechanism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48 In concretely , we extract the deep feature of its input image Ii via a backbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Then , based on Ti , C attention guidance Ac 2 RH⇥W is generated as a set of attention maps , i.e. , A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obtain the final attribute-aware binary codes , it is54 desirable to transform these appearance-specific ( i.e. , low-level ) f atures T to semantic-specific ( i.e.,55 mid-level ) representations which are closer to the attrib te space . Thus , a transforming n twork ( · ) ,56 which is equipped with a stack of convolution layers , is performed on T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ presents the parameters of the corresponding transfor ing networks w.r.t . T ci and Ti , respec-58 tively . Then , we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x glo al i . Th holistic60 feature representation w.r.t . the input image Ii is achieved by concatenating both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 Rd . Note that , we hereby abstract the aforementioned fine-grained62 feature learning process as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section , we introduce the overall framework and notations of the proposed A2-NET model , as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29 3.1 Overall Framework and Notations30 As illustrated in Figure ? ? , our A2-NET model consists of two crucial modules , i.e. , a fine-grai d31 representation learning module and an attribute-aware hash codes g nerating modul . Given an32 input image Ii , based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN , a set of attention guidance A = { Ac } is learned for capturing fine-grained tailored34 local patterns T ci from Ti . To distill semantical cues and further generate the final attribute-aware35 bi ary ash c des , we propose to transform these appearance-specific features T towards semantic-36 specific representations T̂ by perform ng a transform network ( · ) . After aggregating T̂ , th obtained37 attentive local-level features xci are associated wi h the global-level f ature x global i to form as a38 holistic feature representation xi . In order to generate attribute-aware binary hash codes , we conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond to40 an attribute vector w.r.t . one kind of namable properties of fine-grained objects . Finally , hash code41 learning is performed upon vi to obtain the final attribute-aware binary codes ui.42 3.2 Fine-Grained Representation Learning43 Attention plays an important role in human perception [ 3 , 7 ] , and humans exploit a sequence of44 partial glimpses and selectively focus on salient parts of an object or a scene in order to b ter capture45 visu l structure [ 10 ] . I spired by this , we incorporate th attention mech nism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48 In concretely , we extra t the deep feature of its input image Ii via a backbon CNN model CNN ( · ) 49 s50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Then , based on Ti , C attention guidance Ac 2 RH⇥W is generated as a set of attention maps , i.e. , A.51 The attention guidance Ac is designed to ev luate which deep descript rs should be attended or ven52 ov rlooked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obtain the final attribute- ware binary codes , it is54 desirable to transform these appearance-specific ( i.e. , low-level ) features T to semantic-specific ( i.e.,55 mid-level ) representations which are closer to the attribute space . Thus , a transforming network ( · ) ,56 which is equipped with a stack of convolution layers , is performed on T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ pr sents the parameters of the corresponding tr nsforming networks w.r.t . T ci nd Ti , respec-58 tively . Then , we aggregate T̂ ci and T̂i y conducting global average-pooling and cor espo dingly59 obtain the atte tive local-level f atures xci , as well as the global-level fe tur x global i . The holisti60 fe ture representation w.r.t . the input image Ii is achieved b concatenating both xci and xgl bali , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 Rd . Note hat , we hereby abstract the aforementioned fin -grained62 featur learning process as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 … 2 Related Work23 2.1 Fine-Grained Imag Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this secti n , we int oduce the overall framew rk and otations of the p oposed A2-NET model , as28 well as elaborating the detailed modules of A2-NET a its corresp nding ptimiza ion algorithm.29 3.1 Overall Framework and No ations30 As illustrated in Figure ? ? , our A2-NET model consists of two crucial modules , i.e. , a fine-grained31 representation learning module and an attribute-aware hash codes generating module . Given an32 input image Ii , based n its corr po ding deep activa io ten o Ti 2 RC⇥H⇥W extracted by a33 backbone CNN , a set of attention guida ce A = { Ac } i learned for capt ring fine-grained t ilored34 local patterns T ci from Ti . To distill semantical cues and further generate the final attribute-aware35 bi ary has c des , we pro ose to transfo m these ppearan e-specific f atures T owards semantic-36 sp cific representations T̂ by performing a transform etwork ( · ) . Aft r aggre ating T̂ , h obtained37 attentive local-level features xi ar associated with t e gl bal-level feature x global i to form as a38 holistic feature representation xi . In order to ge erate attribute-aware binary hash codes , we conduct39 a reconstructing paradigm to projec xi a vi in an attribute space wh re ts data point correspond to40 an attribute vector w.r.t . one kind of namable properties of fine-grained obj cts . Finally , hash code41 learning is performed upon vi to obtain the final attribute-aware binary codes ui.42 3.2 Fine-Grained Representation Learning43 Att ntion plays an important role in hu an perception [ 3 , 7 ] , nd humans exploit a sequenc of44 partial glimpses and selectively focus on salient parts of an object or a scene in order to better capture45 visu l structure [ 10 ] . Inspired by this , we incorp rate the attentio mechanism into represent tion46 lear ing to capture fine-grain d local patt rns for distinguishing subtle differences between these47 subordinate categories.48 In concretely , we extract the deep feature of its input image Ii via a backbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Then , based on Ti , C attention guida c Ac 2 RH⇥W is gen rat d as set of attention maps , i.e. , A.51 The ttent on guidance Ac s designed to ev luate which deep descriptors should be attended or even52 overl ked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obt in the final attribute-aware binary codes , it is54 desirable to transform these appea a ce-specific ( i.e. , low-level ) features T to semantic-specific ( i.e.,55 mid-l vel ) repres ntations which are closer to the attribute space . Thus , a transforming network ( · ) ,56 which is equipped with a stack of convolution layers , is performed on T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ prese ts the parameters of the corresponding transforming networks w.r.t . T ci and Ti , respec-58 tively . Then , we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentiv local-level f atures xci , as well as the global-level feature x global i . Th holistic60 feature representation w.r.t . the input image Ii is achieved by concatenating both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 Rd . Note that , we hereby abstract the aforementioned fine-grained62 feature learning process as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section , we introduce the overall framework and notations of the proposed A2-NET model , as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorithm.29 3.1 Overall Framework and Notations30 As illustrated in Figure ? ? , our A2-NET model consists of two crucial modules , i.e. , a fine-grained31 representation learning module and an attribute-aware hash codes generating module . Given an32 input image Ii , based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN , a set of attention guidance A = { Ac } is learned for capturing fine-grained tailored34 local patterns T ci from Ti . To distill semanti al cu s d further generate the final attribute-aware35 binary hash codes , we propose to transform these appearance-specific features T towards semantic-36 specific representations T̂ by performing a transform network ( · ) . After aggregating T̂ , the obtained37 attentive local-level features xci are associated with the global-level feature x global i to form as a38 holistic feature representation xi . In order to generate attribute-aware binary hash codes , we conduct39 a r constructing paradigm to project xi as vi in an attribute sp ce where it data point correspond to40 an attribute vector w.r.t . one kind of namable properties of fine-grained objects . Finally , hash code41 lear ing is perfor ed upon vi to obtain the final attribute-aware binary codes ui.42 3.2 Fine-Grained Representation Learning43 Attention plays an important role in human percept on [ 3 , 7 ] , and humans exploit a sequence of44 partial glimpses and selectively focus on salient parts of an object or a scene in order to better capture45 visual structure [ 10 ] . Inspired by this , we inc rpor te t e attention mechanism into representation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48 In concretely , we extract the deep feature f its i put im ge Ii via a ba kbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Then , based on Ti , C attention guidance Ac 2 RH⇥W is generated as a set of ttentio maps , i.e. , A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obtain the final attribut -aware binary codes , it is54 desirable to tran for these appearance-specific ( i.e. , low-level ) features T to semantic-spec fic ( i.e.,55 mid-level ) representations which are closer to the attribute space . Thus , a transforming network ( · ) ,56 which is equipped with a stack of convolution layers , is performed on T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ pr sents the parameters of he c rrespo ding transforming networks w.r.t . T ci a d Ti , respec-58 tively . Then , we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . Th holistic60 feature rep esentation w.r.t . the input image Ii i achieved by concat at g both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 Rd . Note that , we hereby abstract the aforem ntioned fine-grained62 f ature le ning pro ess as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 Aggregation … 2 Related Work23 2.1 Fine-Grained Image Retri val24 2.2 Le rning to Hash25 2.3 Visual Attributes26 3 Method logy27 In this section , we introduce the ove ll framework nd notati ns of the propos A2-NET model , as28 well as elaborating the detailed modules of A2-NET and its correspon ing optimization algorithm.29 3.1 Over l Fra ework and Notations30 As illustrated in Figur ? ? , our A2-NET model consist of two crucial modules , i.e. , a fin -gr ined31 representation learning module and a attribute-aware hash c des ge erating mo ul . Given a32 input image Ii , based on its corres onding de p activatio tens r Ti 2 RC⇥H⇥W extracted by a33 back o e CNN , a set of ttention guidance A = { Ac } is learn d f capturing fine-grained tailored34 local patterns T ci from Ti . To distill semantical cues and further gen rate the fi al attribute-aware35 binary hash cod s , we propose to tr nsform these ppearance-specific features T towards semantic-36 specific repres ntations T̂ by performing a transform network ( · ) . After aggregating T̂ , the obtained37 attentive local-level features xci are associated with the global-level featur x global i to form as a38 holistic e ture r pr s ntation xi . In order to g nerate attribute-aware binary hash cod s , we conduct39 a econstructing p adigm to project xi as vi i an attribute space wh re it dat point correspond to40 an ttribute tor w.r.t . o e kind f n m ble properties of fin -grained objects . Finally , hash code41 le rning is p rformed pon vi to obtain th final attribute- ware bi ary codes ui.42 3.2 Fi e-Grained Representation Le rning43 Attention plays an important role in human perception [ 3 , 7 ] , and humans exploit a sequence of44 partial glimpses and selectively focus on sali nt parts of an object or a scene in or e to b tt r capture45 visual structur [ 10 ] . Inspired by this , we incorporate the attention mechanism into representati n46 learning to capture fin -grained local patterns for distinguishing subtle differences between these47 s bordi te tegories.48 In concretely , we extract the deep feature of its input image Ii via a backbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Th n , based o i , C attention uidan e Ac 2 RH⇥W is g nerated as a set f att ntion m ps , i.e. , A.51 The atte ion i ance Ac is designed to evaluate which deep descriptors should b attended or even52 overlooked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obtain the final attribute-aware binary codes , it is54 desirable to tran form these appearance-specific ( i.e. , low-level ) features T to semantic-specific ( i.e.,55 mid-level ) representations whi h are closer to the attribute space . Thu , a transforming network ( · ) ,56 which is equipped with a stack of convolution layers , is performed on T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) wher ✓ presents the parameters of the corresponding transforming networks w.r.t . T ci and Ti , respec-58 tively . Then , we ggrega e T̂ ci and T̂i by c n ucting global average-pooling and correspondingly59 o tain the attentive local-level featur s xci , as well a the global-l vel feature x global i . The holistic60 feature representation w.r.t . the i put mage Ii is chi ved by concatenating both xci and xglobali , i.e.,61 xi = h xci ; l al = F ( Ii ; ⇥ ) 2 Rd . Note that , we hereby abstract the aforementioned fine-grained62 feature learnin proc ss as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 2 Rel t d Work23 2.1 Fi -Grained Image Retrieval24 2.2 Lear i g to Ha h25 2.3 Visu l Attributes26 3 Methodology27 In this section , we introduce the overall framework and notations of the proposed A2-NET model , as28 w ll as elaborating the detailed modules of A2-NET and its correspon ing optimization algorithm.29 3.1 Overall Fram work and Notation30 As illustrated in Figur ? ? , our A2-NET model consists of two cruc al modules , i.e. , a fi e-grained31 r p tatio l arning modul nd an attribut - ware has codes generating module . Given an32 inp t image Ii , b sed o its respond ng deep a tivation tensor Ti 2 RC⇥H⇥W extracted by a33 ba kbone CNN , a set of attention g idance A = { Ac } is learned for capturing fine-grained tailored34 l l patterns T ci from Ti . To distill semantical cues and further en rate the final attribute-aware35 binary hash odes , we prop se to transform these ppearance-specific features T towards semantic-36 specific representations T̂ by performing a transform netwo k ( · ) . After aggregating T̂ , he obt ined37 attentive local-level features xci re associated with the global-level feature x global i to form as a38 holistic feature representation xi . In order to generate attribute-aware binary hash codes , w conduct39 a reconstructing paradigm to project xi as vi in an attribute space where its data point correspond to40 an attribute vector w.r.t . one kind of namable properties of fine-grained objects . Finally , has c de41 learning is performed upon vi to obtain the fi al attribute-aware binary codes ui.42 3.2 Fine-Grained Representation Learning43 Attention plays a important role in human perception [ 3 , 7 ] , and humans exploit a sequence of44 partial glimpses and select vely focus on salient parts of an bject or a scene in order to b ter capture45 visual structure [ 10 ] . Inspired by this , we incorporate the attention mechanism into representation46 learning to c pture fine-grained local patterns for distinguishing subtle differences b tween th se47 subordinate c tegories.48 In concretely , we extract the deep feature of its input image Ii via a backbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Then , based on Ti , C attention guidance Ac 2 RH⇥W is generat d as a set of attenti n maps , i.e. , A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be tte d d or even52 overlooked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obtain the final attribute-aware binary codes , it is54 desirable to transform these appearance-specific ( i.e. , low-level ) features T to semantic-specific ( i.e.,55 mid-level ) representations which are closer to the attribute space . T us , transf rming netw rk ( · ) ,56 which is equipped with a stack of convolution layers , is performed on T as follows:57 T̂ ci = ( T c i ; ✓local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ presents the parameters of the corresponding transformi g networks w.r.t . T ci and Ti , r spec-58 tively . Then , we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly59 obtain the attentive local-level features xci , as well as the global-level feature x global i . The holistic60 feature representation w.r.t . the input image Ii is achieved by concatenating both xci and xglobali , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 Rd . Note that , we her by abstr c the af rementioned fine-gr ined62 feature learning process as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 2 Related Work23 2.1 Fine-Grained Image Retrieval24 2.2 Learning to Hash25 2.3 Visual Attributes26 3 Methodology27 In this section , we introduce the overall framework and notations of the proposed A2-NET model , as28 well as elaborating the detailed modules of A2-NET and its corresponding optimization algorith .29 3.1 Overall Framework and Notations30 As illustrated in Figure ? ? , our A2-NET model cons sts of two crucial modules , i.e. , a fine-grain d31 representation learning module and an attribute-aware hash codes gene ating module . Given an32 input image Ii , based on its corresponding deep activation tensor Ti 2 RC⇥H⇥W extracted by a33 backbone CNN , a set of attention guidance A = { Ac } is learned for capturing fin -grained tail red34 local patterns T ci from Ti . To distill semantical cues and further generat the final at ribute-aware35 binary hash codes , we propose to transfor th se appearance-specific features T towards semantic-36 specific representations T̂ by performing a transform network ( · ) . Aft r aggregating T̂ , the obtained37 attentive local-level features xci are asso iated with the global-level feature x global i o form as a38 holistic featur representation xi . In ord r to generate attribute-awar binary hash codes , we conduct39 a reconstructing paradigm to project xi as vi in an attribute space wh re its data point corr spond to40 an attribute vector w.r.t . one kind of namable properties of fine-grained objects . Finally , hash code41 learning is performed upon vi to obtain the final attribute-aware binary codes ui.42 3.2 Fine-Grain d Representation Learning43 Attention plays an important role in human perception [ 3 , 7 ] , and humans exploit a sequence of44 partial glimps s and s lectively focus on s lient parts of an object or a scene in order to better capture45 visual structure [ 10 ] . Inspired by this , we incorporate the attention mechanism into repr sentation46 learning to capture fine-grained local patterns for distinguishing subtle differences between these47 subordinate categories.48 In concretely , we extract the deep feature of its nput image Ii via a backbone CNN model CNN ( · ) 49 as50 Ti = CNN ( Ii ) 2 RC⇥H⇥W . ( 1 ) Then , based on Ti , C attention guidance Ac 2 RH⇥W is generated as a set of attention maps , i.e. , A.51 The attention guidance Ac is designed to evaluate which deep descriptors should be attended or even52 overlooked by conducting53 T ci = A c Ti , ( 2 ) where is the element-wise Hadama d product . T obtain he final a tribute-aware binary cod s , it is54 desirable to transform these appearance-specific ( i.e. , low-level ) features T to semantic-specific ( i.e.,55 mid-level ) representations which are closer to the attribute space . Thus , a transforming netwo k ( · ) ,56 which is equipped with a stack of convolution layers , is performed on T as follows:57 T̂ ci = ( i ; local ) , ( 3 ) T̂i = ( Ti ; ✓global ) , ( 4 ) where ✓ presents the parameters of the corresponding transforming networks w.r.t . T ci and Ti , respec-58 tively . Then , we aggregate T̂ ci and T̂i by onducting glob l average-poolin and correspondingly59 obtain the attentive local-level features xci , as w ll as the gl bal-level feature x global i . Th holistic60 feature representation w.r.t . the input image Ii is achiev d by concat nating both xci a d xglobali , i.e.,61 xi = h xci ; x global i i = F ( Ii ; ⇥ ) 2 Rd . Note that , we hereby abstract the aforementioned fine-grained62 feature learning process as a function F ( Ii ; ⇥ ) with its parameters ⇥.63 2 Aggregation where t e d coder ( i.e. , a counterpart of the encoder ) is realized by W > to simplify the netwo k.91 However , directly minimizing Eq . ( 6 ) with a hard constraint is hard to optimize . Therefore , we relax92 the constraint into a soft constraint , and the learning objective can be rewritten by93 min W kX W > V 0k2F + kWX V 0k2F . ( 7 ) 3.3.2 Attribute-Specifi F ur Decorrelation94 By conducting the aforementioned unsupervised attribut -guided learning , we can obtain the internal95 l tent vectors V 0 as the attribute-specific features . I order to both enhance the discriminative ability96 and remove the redundant correlation among these attribute-specific features , we introduce a feature97 decorrelation c n r int upon V 0 , which is formulat d by98 mi V 0 kV 0V 0 > nIk2F , ( 8 ) where I is the identity matrix . Such a feature ecorrelation co straint is pref rable to construct99 independent features and reduce redundant information . Therefore , based on unsupervised attribute-100 guided learning and attribute-specific feature decorrelation , the final learned hash codes are expected101 to be bo h attribute-aware and hash-bit ind pend nt.102 3.3.3 H sh Co e L arning103 In the f llowing , we conduct the hash code learning based on the obtained attribute-specific features.104 Assume that w have n qu ry data p ints which are denoted as { qi } i=1 , as w ll a m database points105 which a noted as { vj } j=1 . Note hat , both qi and vi are belo gin to the attribute space V . Then,106 the corresponding binary has codes can be obt ined via107 ui = sgn ( tanh ( qi ) ) , ( 9 ) zj = sgn ( tanh ( vj ) ) , ( 10 ) where ui , zj 2 { 1 , +1 } k. The goal of our hash code learning is to learn binary hash codes for both108 qu ry poin s and database points from { qi } ni=1 , { vj } mj=1 , and the pairwise supervised information,109 i.e. , S 2 { 1 , +1 } n m. To pr se ve the pairwise similarity , adopt the ` 2 loss between the110 supervised inf rmati ( .k.a . similarity ) d t inner product f qu ry-database point binary code111 pairs . It can be formulated as follows:112 minU , Z Pn i=1 Pm j=1 u > i zj kSij 2 s.t . U 2 { 1 , +1 } n⇥k , Z 2 { 1 , +1 } m⇥k . ( 11 ) Overall , e g t the fina obj iv of the propo ed A2-NET model by considering Eq . ( 7 ) , Eq . ( 8 ) and113 Eq . ( 11 ) toge a follows:114 min W , ⇥ L ( I ) = kX W > V 0k2F + kWX V 0k2F +↵kV 0V 0 > nIk2F + nX i=1 mX j=1 > i zj kSij 2 , ( 12 ) where , ↵ and are hyper-parameters as the trade-off.115 In practice , during training , it might be only available a set of database poin s { vj } mj=1 without query116 points . Thus , we randomly sample n data points from database to construct a query set , and denote117 the indices of all the database points as with the indices of the query set as ⌦ . Additionally , because118 we can not back-propagate the gradient to ⇥ due to the sgn ( · ) function , we omit the sgn ( · ) function119 and o ly apply t nh ( · ) for elaxation in Eq . ( 10 ) of the whole optimization process . Therefore , the120 optimization formulation of A2-NET can be rewritten with only database points { vj } mj=1 for training121 as:122 min W , ⇥ L ( I ) = kX W > V 0k2F + kWX V 0k2F + ↵kV 0V 0 > nIk2F + X i2⌦ X j2 ⇣ tanh ( W · F ( Ii ; ⇥ ) ) > zj kSij ⌘2 . ( 13 ) For optimization , our A2-NET does not require co plicated two-stage learning algorithms , e.g. , the123 alternative optimization strategy . We employ the back-propagation algorithm and follow [ 8 ] to train124 the whol A2-NET model in a u ified end-to-end manner.125 4 “ Blue tail ” Figure 2 : Overall framewo k of the proposed A2-NET m el , which consists of two crucial modules , i.e. , fine-grained representation learning and attribute-aware hash codes generating . The whole netw rk c n be end-to-e d trainabl , d is generally riven by the nsupervi ed attribute-guided reconstruction loss , the featur decorrelation loss an the sh o e l arning loss , cf . Section 3.3. objects of fine-grained images have only subtle differences , and ften larg ly vary in pose , c l , and orientation or can exhibit cr ss-mo al differences ( e.g. , sketch-based retrieval [ 25 ] ) . Depe ding on he type of query image , the ost studied areas of fin -grai ed image retrieval can be separat into two groups : fine-grai e content-b s d imag retrieval ( FG-CBIR ) and finegrained sketch-based image retrieval ( FG-SBIR ) . More specifically , in G-CBIR , unsupervised learning based [ 40 ] a d supervise learning based methods [ 44 , 47 , 48 ] were developed from different perspe tives for handling fine-grained retrieval tasks e.g. , localizing fine-grained parts [ 40 ] , enhan ing intra-class separability with inter-class compactn ss [ 48 ] , an re ci g the co fid nce of the fine-grained predictions [ 44 ] , tc . While , FG-SBIR eeds to n t only captur fine- rained characteristic pr sent in th sket h s , but also poss ss he ability t traverse the sketch and image domain gap . In the literature of FG-SBIR , the earlier works , e.g. , [ 23 , 43 , 46 ] , were mostly based on Siamese-triplet networ s [ 3 ] to tackle the aforemention d challenges . Recently , s me works tried to incorporate the advances of rec nt progress in self-supervised learning [ 6 ] a d attention mechanisms [ 7 ] for further improving the retrieval accuracy of FG-SBIR , e.g. , [ 29 , 2 ] . However , although these fine-grained retrieval methods achieved good results , they still have the limitations in the face of large-scale data , i.e. , the searching time for exact nearest neighbor is typically expensive or even impossible for the given queries . To alleviate this issue , fine-grained hashing , which aims to generat comp ct binary codes o represent fine-grained images , as a promising direction has attracted the attention in the fine-grained community very recently [ 8 , 19 ] . More specifically , ExchNet [ 8 ] was the first to define the fine-grained hashing task and develop a fine-grained tailored method to firstly locate discriminative object parts and further learn binary hash codes for representing fine-grained images . In the same period , DSaH [ 19 ] was proposed to automatically mine salient regions and learn semantic-preserving hash codes simultaneously . Unfortunately , the learned hash bits of these methods lack any semantics which are more meaningful to fine-grained objects , and thus lack the model interpretability . Compared with them , our proposed A2-NET can not only outperform the previous fine-grained hashing methods , but more importantly , the learned hash codes of A2-NET are attribute-aware , i.e. , the hash bits of A2-NET have strong correspondence to semantic visual properties that are useful for fine-grained image retrieval . 2.2 Learning to Hash Hashing [ 38 ] is a widely-studied solution to approximate nearest neighbor search , which transforms the data item to a short code consisting of a sequence of bits ( i.e. , hash codes ) . The research efforts of hashing can be categorized into two groups , including data-independent hashing ( aka locality sensitive hashing [ 9 , 27 , 34 ] ) and data-dependent hashing ( aka learning to hash [ 4 , 12 , 17 , 33 ] ) . Specifically , locality sensitive hashing methods attempted to adjust hash learning from different perspectives , e.g. , the theory or machine learning views , to name a few : proposing random hash functions satisfying local sensitive property [ 9 ] , developing better search schemes [ 27 ] , providing faster computation of hash functions [ 34 ] , etc . While , compared with locality sensitive hashing methods , since data-dependent hashing methods learn hash functions from a specific dataset to achieve similarity preserving , they can generally obtain superior retrieval accuracy . Especially for capitalizing on advances in deep learning , many well-performing methods were proposed to integrate feature learning and hash code learning into an end-to-end framework based on deep networks , e.g. , [ 4 , 12 , 17 ] . In particular , very recently researchers in the vision community have begun to pay attention to the more challenging and practical hashing task , i.e. , fine-grained hashing [ 8 , 19 ] . To the best of our knowledge , this is the first work to equip these learned hash codes with strong correspondence to visual attributes for dealing with large-scale fine-grained image retrieval . 2.3 Visual Attributes Attributes are typically mid-level semantic properties of objects [ 10 ] , such as colors ( e.g. , “ red ” , “ blue ” ) , texture ( e.g. , “ striped ” , “ spotted ” ) , or even life habits of animals ( e.g. , “ living on the tree ” , “ living in the water ” ) . Visual attributes have exhibited their impact for strengthening various vision tasks , including facial verification [ 21 ] , fine-grained categorization [ 45 ] , zero-shot transfer [ 42 ] , scene understanding [ 30 ] , and so on . Most of the previous attribute learning methods are supervised by costly human-generated annotations and also are dependent on pre-defined attribute labels , e.g. , [ 15 , 21 , 42 , 45 ] . In consequence , for large-scale problems , these supervised methods might be not feasible due to the restriction caused by the cumbersomely obtained attribute annotations . Moreover , even for some tasks , their visual attributes are quite hard to define . In this paper , to alleviate the aforementioned issues , we propose an A2-NET model to formulate an unsupervised learning structure to project the learned visual features into an attribute space where it finally generates attribute-aware binary hash codes . Compared with previous attribute learning methods , our A2-NET is independent with pre-defined attribute labels , and could automatically learn discriminative attribute-aware hash codes in a unified end-to-end trainable fashion . Furthermore , our method can not only correspond hash bits to visual attributes tailored for fine-grained objects , which shows significant improvements of retrieval accuracy , but also offer an intuitive way of deep hashing interpretation . 3 Methodology In this section , we introduce the overall framework and notations of the proposed A2-NET model , as well as elaborating the key modules of A2-NET and its corresponding optimization algorithm . 3.1 Overall Framework and Notations As illustrated in Figure 2 , our A2-NET model consists of two crucial modules , i.e. , a fine-grained representation learning module and an attribute-aware hash codes generating module . Given an input image Ii , based on its corresponding deep activation tensor Ti ∈ RC×H×W extracted by a backbone CNN , a set of attention guidance A = { Ac } is learned for capturing fine-grained tailored local patterns T ci from Ti . To distill semantical cues and further generate the final attribute-aware binary hash codes , we propose to transform these appearance-specific features T towards semanticspecific representations T̂ by performing a transform network φ ( · ) . After aggregating T̂ , the obtained attentive local-level features xci are associated with the global-level feature x global i to form as a holistic feature representation xi . In order to generate attribute-aware binary hash codes , we conduct a reconstructing paradigm to project xi as vi in an attribute space where its data point corresponds to an attribute vector w.r.t . a certain kind of nameable properties of fine-grained objects ( e.g. , “ red head ” or “ spotted body ” ) . Furthermore , with the aid of feature decorrelation , vi is expected to be more discriminative by removing redundant correlation information . Finally , hash code learning is performed upon vi to obtain the final attribute-aware binary codes ui . 3.2 Fine-Grained Representation Learning Attention plays an important role in human perception [ 7 , 16 ] , and humans exploit a sequence of partial glimpses and selectively focus on salient parts of an object or a scene in order to better capture visual structure [ 22 ] . Inspired by this , we incorporate the attention mechanism into representation learning to capture fine-grained local patterns for distinguishing subtle differences between these subordinate categories . In concretely , we extract the deep feature of its input image Ii via a backbone CNN model ΦCNN ( · ) by Ti = ΦCNN ( Ii ) ∈ RC×H×W . ( 1 ) Then , based on Ti , C attention guidance Ac ∈ RH×W is generated as a set of attention maps , i.e. , A . The attention guidance Ac is designed to evaluate which deep descriptors [ 40 ] in these H ×W cells should be attended or even overlooked by conducting T ci = A c Ti , ( 2 ) where is the element-wise Hadamard product . To obtain the final attribute-aware binary codes , it is desirable to transform these appearance-specific ( i.e. , low-level ) features T to semantic-specific ( i.e. , mid-level ) representations which are closer to the attribute space . Thus , a transforming network φ ( · ) , which is equipped with a stack of convolution layers , is performed on T as follows : T̂ ci = φ ( T c i ; θlocal ) , ( 3 ) T̂i = φ ( Ti ; θglobal ) , ( 4 ) where θ presents the parameters of the corresponding transforming networks w.r.t . T ci and Ti , respectively . Then , we aggregate T̂ ci and T̂i by conducting global average-pooling and correspondingly obtain the attentive local-level features xci , as well as the global-level feature x global i . The holistic feature representation w.r.t . the input image Ii is achieved by concatenating both xci and xglobali , i.e. , xi = [ xci ; x global i ] = F ( Ii ; Θ ) ∈ Rd . Note that , we hereby abstract the aforementioned fine-grained feature learning process as a function F ( Ii ; Θ ) associated with its parameters Θ . 3.3 Attribute-Aware Hash Codes Generating How to generate attribute-aware hash codes is the key of our A2-NET model . We elaborate it in the following three aspects , i.e. , unsupervised attribute-guided learning , attribute-specific feature decorrelation , and hash code learning . 3.3.1 Unsupervised Attribute-Guided Learning In real-applications , especially for the large-scale and fine-grained tasks , attribute annotations are always infeasible , which limits the learning process to be conducted in an unsupervised setting . While , in the literature , the main goal of unsupervised learning is to capture regularities in data for the purpose of extracting useful representations or for restoring corrupted data [ 31 ] . Many unsupervised methods explicitly produce internal latent units or codes , from which the data is to be reconstructed . Inspired by this , we develop an unsupervised attribute-guided reconstruction component to project the holistic representation xi of Ii into a latent space , i.e. , the attribute space V . In V , its high-level vectors are designed to have certain desirable properties , e.g. , corresponding to semantic properties of fine-grained objects ( aka “ fine-grained attributes ” ) . More specifically , in our A2-NET , the unsupervised attribute-guided learning is realized by a reconstruction paradigm with an encoder-decoder structure , as shown in Figure 2 . In concretely , given a batch of n training data Ii , their holistic representations X = { x1 ; x2 ; . . . ; xn } ∈ Rd×n can be obtained as aforementioned . By formulation , the encoder projects X into the attribute space V with a projection matrix W ∈ Rk×d to get an internal latent representation V ∈ Rk×n w.r.t . X . In particularly , we set that the dimension of latent representation k equals the number of hash bits in the final binary hash code ui . Furthermore , each column of V , i.e. , vi ∈ Rk , can derive ui by ui = sgn ( tanh ( vi ) ) . ( 5 ) Meanwhile , regarding vi , we also consider to reconstruct its input xi by a decoder as a counterpart of the encoder . Therefore , on one hand , such a reconstruction paradigm can reduce and further distill high-level semantic cues in the attribute space V . While , on the other hand , it can drive the training of A2-NET by preserving the similarity between queried hash codes and database points in terms of hash code learning ( cf . Section 3.3.3 ) . In concretely , the learning objective of unsupervised attribute-guided reconstruction is written as follows : min W ‖X −W > WX‖2F s.t.WX = V ′ = tanh ( V ) , ( 6 ) where the decoder ( i.e. , a counterpart of the encoder ) is realized by W > to simplify the network . However , directly minimizing Eq . ( 6 ) with a hard constraint is difficult to optimize . Therefore , we relax the constraint into a soft constraint , and then the learning objective can be rewritten as min W ‖X −W > V ′‖2F + λ‖WX − V ′‖2F . ( 7 ) 3.3.2 Attribute-Specific Feature Decorrelation By conducting the aforementioned unsupervised attribute-guided learning , we can obtain the internal latent vectors V ′ as the attribute-specific features . In order to both enhance the discriminative ability and remove the redundant correlation among these attribute-specific features , we introduce a feature decorrelation constraint upon V ′ , which is formulated by min V ′ ‖V ′V ′ > − nI‖2F , ( 8 ) where I is the identity matrix and n is the batch size . Such a feature decorrelation constraint is preferable to construct independent features and reduce redundant information . Therefore , based on both unsupervised attribute-guided reconstruction and attribute-specific feature decorrelation , the final learned hash codes are expected to be both attribute-aware and hash-bit independent . 3.3.3 Hash Code Learning In the following , we conduct the hash code learning based on the obtained attribute-specific features . Assume that we have n query data points which are denoted as { qi } ni=1 , as well as m database points which are denoted as { vj } mj=1 . Note that , both qi and vi are belonging to the attribute space V . By following Eq . ( 5 ) , the corresponding binary hash codes can be obtained via ui = sgn ( tanh ( qi ) ) , ( 9 ) zj = sgn ( tanh ( vj ) ) , ( 10 ) where ui , zj ∈ { −1 , +1 } k. The goal of our hash code learning is to learn binary hash codes for both query points and database points from { qi } ni=1 , { vj } mj=1 , and the pairwise supervised information , i.e. , S ∈ { −1 , +1 } n×m . To preserve the pairwise similarity , we adopt the ` 2 loss between the supervised information ( aka similarity ) and the inner product of query-database point binary code pairs . It can be formulated as follows : minU , Z ∑n i=1 ∑m j=1 ( u > i zj − kSij ) 2 s.t . U ∈ { −1 , +1 } n×k , Z ∈ { −1 , +1 } m×k . ( 11 ) Overall , we get the final objective of the proposed A2-NET model by considering Eq . ( 7 ) , Eq . ( 8 ) and Eq . ( 11 ) together as follows : min W , Θ L ( I ) = ‖X−W > V ′‖2F +λ‖WX−V ′‖2F +α‖V ′V ′ > −nI‖2F +β n∑ i=1 m∑ j=1 ( u > i zj − kSij ) 2 , ( 12 ) where λ , α and β are hyper-parameters as the trade-off . In practice , during training , it might be only available a set of database points { vj } mj=1 without query points . Thus , we randomly sample n data points from database to construct a query set , and denote the indices of all the database points as Γ with the indices of the query set as Ω. Additionally , because we can not back-propagate the gradient to Θ due to the sgn ( · ) function , we omit the sgn ( · ) function and only apply tanh ( · ) for relaxation in Eq . ( 10 ) of the whole optimization process . Therefore , the optimization formulation of A2-NET can be rewritten with only database points { vj } mj=1 for training as : min W , Θ L ( I ) = ‖X −W > V ′‖2F + λ‖WX − V ′‖2F + α‖V ′V ′ > − nI‖2F + β ∑ i∈Ω ∑ j∈Γ ( tanh ( W · F ( Ii ; Θ ) ) > zj − kSij ) 2 . ( 13 ) For optimization , our A2-NET does not require complicated two-stage learning algorithms , e.g. , the alternative optimization strategy . In experiments , we employ the back-propagation algorithm and follow [ 17 ] to train the whole A2-NET model in a unified end-to-end manner . 3.4 Out-of-Sample Extension After training A2-NET , the learned model can be applied for generating binary codes for query points including unseen query points in the training phase . Specifically , we can use the following equation to generate the binary code for Iq : uq = sgn ( tanh ( W · F ( Iq ; Θ ) ) ) . ( 14 ) 4 Experiments 4.1 Datasets By following ExchNet [ 8 ] , our experiments are conducted on five fine-grained benchmark datasets , i.e. , CUB200-2011 [ 37 ] , Aircraft [ 28 ] , Food101 [ 2 ] , NABirds [ 35 ] and VegFru [ 14 ] . Specifically , CUB200-2011 is one of the most popular used fine-grained datasets . It contains 11,788 bird images from 200 bird species and is officially split into 5,994 images for training and 5,794 images for test . Aircraft contains 10,000 images spanning 100 aircraft models with 3,334 for training , 3,333 for validation and 3,333 for test . For large-scale datasets , Food101 contains 101 kinds of foods with 101,000 images , where for each class , 250 test images are checked manually for correctness while 750 training images still contain a certain amount of noises . NABirds is a high quality dataset which has 48,562 images of North American birds with 555 sub-categories , where 23,929 for training with 24,633 for test . VegFru is another large-scale fine-grained dataset covering 200 kinds of vegetables and 92 kinds of fruits with 29,200 for training , 14,600 for validation and 116,931 for test . 4.2 Baselines and Implementation Details Baselines In experiments , we compare our proposed model to the following competitive baselines , i.e. , ITQ [ 11 ] , SDH [ 33 ] , DPSH [ 24 ] , HashNet [ 5 ] , and ADSH [ 17 ] . Among them , DPSH , HashNet and ADSH are deep learning based methods , while ITQ and SDH are not . Furthermore , we also compare the results of our A2-NET with state-of-the-arts of fine-grained hashing methods , including ExchNet [ 8 ] . Additionally , another fine-grained hashing method , i.e. , DSaH [ 19 ] , also achieved good retrieval accuracy . However , due to its empirical settings quite distant from other existing fine-grained hashing methods , for fair comparisons , we strictly control empirical settings as the same as those of [ 19 ] and compare the results of our A2-NET with its results in the supplementary materials . Implementation Details For fair comparisons , we follow the efficient training setting in ExchNet [ 8 ] . In concretely , for CUB200-2011 , Aircraft and Food101 , we sample 2,000 images per epoch , while 4,000 samples are randomly selected for NABirds and VegFru . For the training details , regarding the backbone model , we can choose any network structure as the base network for the fine-grained representation learning module . While , by following [ 8 ] , ResNet-50 [ 13 ] is employed in experiments . The total number of training epochs is 20 , and the number of batch size is set as 16 . While , different from ExchNet , our model only requires a smaller iteration time until convergence . Specifically , for these datasets containing less than 20,000 training images , the iteration time Tmax is 60 , and the learning rate is divided by 10 at the 50th iteration . For other datasets , Tmax is set as 70 , and the learning rate is divided by 10 at the 60th iteration . The hyper-parameters , i.e. , λ , α and β in Eq . ( 13 ) , are set as 1 , 1n×k and 12 k , respectively . The number of attention guidance equals the number of hash bits . The optimizer is standard mini-batch stochastic gradient descent with the weight decay as 10−4 . All experiments are conducted with a GeForce RTX 2080 Ti GPU . 4.3 Main Results Table 1 presents the mean average precision ( mAP ) results of fine-grained retrieval on these five aforementioned fine-grained benchmark datasets . For each dataset , we report the results of four lengths of hash bits , i.e. , 12 , 24 , 32 , and 48 , for evaluations . As shown in that table , our proposed A2NET model significantly and consistently outperforms the other baseline methods on these datasets . In particular , compared with the state-of-the-art method ExchNet [ 8 ] , our A2-NET achieves 17.83 % and 17.88 % improvements over ExchNet of 24-bit and 32-bit experiments on Aircraft and Food-101 , respectively . Moreover , A2-NET also obtains superior results with an absolute value of about 80 % mAP on CUB200-2011 , Aircraft and Food101 with 48-bit hash codes . These observations validate the effectiveness of the proposed A2-NET model , as well as its promising practicality in real-applications of fine-grained retrieval . Additionally , in Figure 3 , we illustrate several retrieval results on CUB2002011 , which shows that A2-NET can retrieve well among multiple subordinate categories when the same species of birds with diverse variations appear in different kinds of background . Also , there also exist several failure cases , where quite tiny differences ( e.g. , caused by different views ) between the query image and the returned images are demanded by carefully observations . 4.4 Ablation Studies In this section , we demonstrate the effectiveness of these crucial components of the proposed A2-NET model , i.e. , the attention-based fine-grained representation learning component 3.2 , the unsupervised attribute-guided reconstruction component ( cf . Section 3.3.1 ) and the attribute-specific feature decorrelation component ( cf . Section 3.3.2 ) . In the ablation studies , we apply these components incrementally on a vanilla backbone ( i.e. , ResNet-50 ) as the baseline . As evaluated in Table 2 , by stacking these two components one by one , the retrieval results are steadily improved , which justifies the effectiveness of our proposed components in A2-NET . 4.5 Qualitative Analyses of Attribute-Aware Hash Codes We hereby discuss the quality of the learned attribute-aware hash codes ui of A2-NET . After obtaining ui , we visualize fine-grained images retrieved by a random single hash bit of ui to demonstrate the strong correspondence between visual attributes and the obtained hash bits . All the five datasets in experiments are used as examples to illustrate the quality . As observed in Figure 4 , images of each column have some similar fine-grained object properties , i.e. , visual attributes . Indeed , the learned hash codes are apparently attribute-aware , which could provide an explanation of the A2-NET ’ s success in fine-grained retrieval . Meanwhile , it also offers human-understandable interpretation for such a deep learning based fine-grained hashing method . 5 Conclusion In this paper , we proposed an Attribute-Aware hashing Network , i.e. , A2-NET , for dealing with the large-scale fine-grained image retrieval task . Particularly , A2-NET was designed as expected to be efficient , effective and more importantly interpretable . In concretely , by developing an unsupervised attribute-guided reconstruction method based on the obtained appearance-specific visual representation with attention , it can distill attribute-specific vectors in a high-level attribute space . After further performing feature decorrelation upon attribute-specific vectors , their discriminative ability is strengthened for representing a fine-grained object . Then , hash codes can be generated from these attribute-specific vectors and thus became attribute-aware . Both qualitative and quantitative experiments demonstrate the effectiveness of our A2-NET . Additionally , visual attributes have been shown to be useful in describing both known and unknown entities , which motivates us to study identifying unobserved sub-categories , i.e. , zero-shot fine-grained recognition , based on our attribute-aware hash codes as the future work . References [ 1 ] Thomas Berg , Jiongxin Liu , Seung Woo Lee , Michelle L. Alexander , David W. Jacobs , and Peter N. Belhumeur . Birdsnap : Large-scale fine-grained visual categorization of birds . In Proc . IEEE Conf . Comp . Vis . Patt . Recogn. , pages 2019–2026 , 2014 . [ 2 ] Lukas Bossard , Matthieu Guillaumin , and Luc Van Gool . Food-101 – mining discriminative components with random forests . In Proc . Eur . Conf . Comp . Vis. , pages 446–461 , 2014 . [ 3 ] Jane Bromley , Isabelle Guyon , Yann LeCun , Eduard Säckinger , and Roopak Shah . Signature verification using a “ siamese ” time delay neural network . In Proc . Advances in Neural Inf . Process . Syst. , pages 737–744 , 1993 . [ 4 ] Fatih Cakir , Kun He , Sarah Adel Bargal , and Stan Sclaroff . Hashing with mutual information . IEEE Trans . Pattern Anal . Mach . Intell. , 41 ( 10 ) :2424–2437 , 2019 . [ 5 ] Zhangjie Cao , Mingsheng Long , Jianmin Wang , and Philip S. Yu . HashNet : Deep learning to hash by continuation . In Proc . IEEE Int . Conf . Comp . Vis. , pages 5608–5617 , 2017 . [ 6 ] Ting Chen , Simon Kornblith , Mohammad Norouzi , and Geoffrey Hinton . A simple framework for contrastive learning of visual representations . In Proc . Int . Conf . Mach . Learn. , pages 1597–1607 , 2020 . [ 7 ] Maurizio Corbetta and Gordon L. Shulman . Control of goal-directed and stimulus-driven attention in the brain . Nature Reviews Neuroscience , 3:201–215 , 2002 . [ 8 ] Quan Cui , Qing-Yuan Jiang , Xiu-Shen Wei , Wu-Jun Li , and Osamu Yoshie . ExchNet : A unified hashing network for large-scale fine-grained image retrieval . In Proc . Eur . Conf . Comp . Vis. , pages 189–205 , 2020 . [ 9 ] Anirban Dasgupta , Ravi Kumar , and Tamas Sarlos . Fast locality-sensitive hashing . In Proc . ACM SIGKDD Int . Conf . Knowledge discovery & data mining , pages 1073–1081 , 2011 . [ 10 ] Vittorio Ferrari and Andrew Zisserman . Learning visual attributes . In Proc . Advances in Neural Inf . Process . Syst. , pages 433–440 , 2007 . [ 11 ] Yunchao Gong , Svetlana Lazebnik , Albert Gordo , and Florent Perronnin . Iterative quantization : A procrustean approach to learning binary codes for large-scale image retrieval . IEEE Trans . Pattern Anal . Mach . Intell. , 35 ( 12 ) :2916–2929 , 2012 . [ 12 ] Kun He , Fatih Cakir , Sarah Adel Bargal , and Stan Sclaroff . Hashing as tie-aware learning to rank . In Proc . IEEE Conf . Comp . Vis . Patt . Recogn. , pages 4023–4032 , 2018 . [ 13 ] Kaiming He , Xiangyu Zhang , Shaoqing Ren , and Jian Sun . Deep residual learning for image recognition . In Proc . IEEE Conf . Comp . Vis . Patt . Recogn. , pages 770–778 , 2016 . [ 14 ] Saihui Hou , Yushan Feng , and Zilei Wang . VegFru : A domain-specific dataset for fine-grained visual categorization . In Proc . IEEE Int . Conf . Comp . Vis. , pages 541–549 , 2017 . [ 15 ] Junshi Huang , Rogerio Feris , Qiang Chen , and Shuicheng Yan . Cross-domain image retrieval with a dual attribute-aware ranking network . In Proc . IEEE Int . Conf . Comp . Vis. , pages 1062–1070 , 2015 . [ 16 ] Laurent Itti , Christof Koch , and Ernst Niebur . A model of saliency-based visual attention for rapid scene analysis . IEEE Trans . Pattern Anal . Mach . Intell. , 20 ( 11 ) :1254–1259 , 1998 . [ 17 ] Qing-Yuan Jiang and Wu-Jun Li . Asymmetric deep supervised hashing . In Proc . Conf . AAAI , pages 3342–3349 , 2018 . [ 18 ] Sheng Jin , Hongxun Yao , Xiaoshuai Sun , Shangchen Zhou , Lei Zhang , and Xiansheng Hua . Deep saliency hashing for fine-grained retrieval . IEEE Trans . Image Process. , 29:5336–5351 , 2020 . [ 19 ] Sheng Jin , Hongxun Yao , Xiaoshuai Sun , Shangchen Zhou , Lei Zhang , and Xiansheng Hua . Deep saliency hashing for fine-grained retrieval . IEEE Trans . Image Process. , 29:5336–5351 , 2020 . [ 20 ] Jonathan Krause , Michael Stark , Jia Deng , and Li Fei-Fei . 3D object representations for fine-grained categorization . In ICCV Workshop on 3D Representation and Recognition , 2013 . [ 21 ] Neeraj Kumar , Alexander Berg , Peter N. Belhumeur , and Shree Nayar . Describable visual attributes for face verification and image search . IEEE Trans . Pattern Anal . Mach . Intell. , 33 ( 10 ) :1962–1977 , 2011 . [ 22 ] Hugo Larochelle and Geoffrey Hinton . Learning to combine foveal glimpses with a third-order boltzmann machine . In Proc . Advances in Neural Inf . Process . Syst. , pages 1243–1251 , 2010 . [ 23 ] Ke Li , Kaiyue Pang , Yi-Zhe Song , Timothy M. Hospedales , Tao Xiang , and Honggang Zhang . Synergistic instance-level subspace alignment for fine-grained sketch-based image retrieval . IEEE Trans . Image Process. , 26 ( 12 ) :5908–5921 , 2017 . [ 24 ] Wu-Jun Li , Sheng Wang , and Wang-Cheng Kang . Feature learning based deep supervised hashing with pairwise labels . In Proc . Int . Joint Conf . Artificial Intell. , pages 1711–1717 , 2015 . [ 25 ] Yi Li , Timothy M. Hospedales , Yi-Zhe Song , and Shaogang Gong . Fine-grained sketch-based image retrieval by matching deformable part models . In Proc . British Machine Vis . Conf. , pages 1–12 , 2014 . [ 26 ] Ziwei Liu , Ping Luo , Shi Qiu , Xiaogang Wang , and Xiaoou Tang . DeepFashion : Powering robust clothes recognition and retrieval with rich annotations . In Proc . IEEE Conf . Comp . Vis . Patt . Recogn. , pages 1096–1104 , 2016 . [ 27 ] Qin Lv , William Josephson , Zhe Wang , Moses Charikar , and Kai Li . Multi-probe LSH : Efficient indexing for high-dimensional similarity search . In Proc . Int . Conf . Very Large Data Bases , pages 950–961 , 2007 . [ 28 ] Subhransu Maji , Esa Rahtu , Juho Kannala , Mathhew Blaschko , and Andrea Vedaldi . Finegrained visual classification of aircraft . arXiv preprint arXiv:1306.5151 , 2013 . [ 29 ] Kaiyue Pang , Yongxin Yang , Timothy M. Hospedales , Tao Xiang , and Yi-Zhe Song . Solving mixed-modal jigsaw puzzle for fine-grained sketch-based image retrieval . In Proc . IEEE Conf . Comp . Vis . Patt . Recogn. , pages 10347–10355 , 2020 . [ 30 ] Genevieve Patterson , Chen Xu , Hang Su , and James Hays . The SUN attribute database : Beyond categories for deeper scene understanding . Int . J. Comput . Vision , 108 ( 1-2 ) :59–81 , 2014 . [ 31 ] Marc ’ Aurelio Ranzato , Y-Lan Boureau , Sumit Chopra , and Yann LeCun . A unified energy-based framework for unsupervised learning . In ICAIS , pages 371–379 , 2007 . [ 32 ] Aneeshan Sain , Ayan Kumar Bhunia , Yongxin Yang , Tao Xiang , and Yi-Zhe Song . Cross-modal hierarchical modelling for fine-grained sketch based image retrieval . In Proc . British Machine Vis . Conf. , pages 1–14 , 2020 . [ 33 ] Fumin Shen , Chunhua Shen , Wei Liu , and Heng Tao Shen . Supervised discrete hashing . In Proc . IEEE Conf . Comp . Vis . Patt . Recogn. , pages 37–45 , 2015 . [ 34 ] Anshumali Shrivastava and Ping Li . Densifying one permutation hashing via rotation for fast near neighbor search . In Proc . Int . Conf . Mach . Learn. , pages 557–565 , 2014 . [ 35 ] Grant Van Horn , Steve Branson , Ryan Farrell , Scott Haber , Jessie Barry , Panos Ipeirotis , Pietro Perona , and Serge Belongie . Building a bird recognition app and large scale dataset with citizen scientists : The fine print in fine-grained dataset collection . In Proc . IEEE Conf . Comp . Vis . Patt . Recogn. , pages 595–604 , 2015 . [ 36 ] Grant Van Horn , Oisin Mac Aodha , Yang Song , Yin Cui , Chen Sun , Alex Shepard , Hartwig Adam , Pietro Perona , and Serge Belongie . The iNaturalist species classification and detection dataset . In Proc . IEEE Conf . Comp . Vis . Patt . Recogn. , pages 8769–8778 , 2017 . [ 37 ] Catherine Wah , Steve Branson , Peter Welinder , Pietro Perona , and Serge Belongie . The Caltech-UCSD birds-200-2011 dataset . Tech . Report CNS-TR-2011-001 , 2011 . [ 38 ] Jingdong Wang , Heng Tao Shen , and Ting Zhang . A survey on learning to hash . IEEE Trans . Pattern Anal . Mach . Intell. , 40 ( 4 ) :769–790 , 2018 . [ 39 ] Xiu-Shen Wei , Quan Cui , Lei Yang , Peng Wang , and Lingqiao Liu . RPC : A large-scale retail product checkout dataset . arXiv preprint arXiv:1901.07249 , 2019 . [ 40 ] Xiu-Shen Wei , Jian-Hao Luo , Jianxin Wu , and Zhi-Hua Zhou . Selective convolutional descriptor aggregation for fine-grained image retrieval . IEEE Trans . Image Process. , 26 ( 6 ) :2868–2881 , 2017 . [ 41 ] Xiu-Shen Wei , Jianxin Wu , and Quan Cui . Deep learning for fine-grained image analysis : A survey . arXiv preprint arXiv:1907.03069 , 2019 . [ 42 ] Yongqin Xian , Christoph H. Lampert , Bernt Schiele , and Zeynep Akata . Zero-shot learning–a comprehensive evaluation of the good , the bad and the ugly . IEEE Trans . Pattern Anal . Mach . Intell. , 41 ( 9 ) :2251–2265 , 2019 . [ 43 ] Qian Yu , Feng Liu , Yi-Zhe Song , Tao Xiang , Timothy M. Hospedales , and Chen-Change Loy . Sketch me that shoe . In Proc . IEEE Conf . Comp . Vis . Patt . Recogn. , pages 799–807 , 2016 . [ 44 ] Xianxian Zeng , Yun Zhang , Xiaodong Wang , Kairui Chen , Dong Li , and Weijun Yang . Finegrained image retrieval via piecewise cross entropy loss . Image and Vision Computing , 93:2868– 2881 , 2020 . [ 45 ] Hua Zhang , Xiaochun Cao , and Rui Wang . Audio visual attribute discovery for fine-grained object recognition . In Proc . Conf . AAAI , pages 7542–7549 , 2018 . [ 46 ] Jingyi Zhang , Fumin Shen , Li Liu , Fan Zhu , Mengyang Yu , Ling Shao , Heng Tao Shen , and Luc Van Gool . Generative domain-migration hashing for sketch-to-image retrieval . In Proc . Eur . Conf . Comp . Vis. , pages 297–314 , 2018 . [ 47 ] Xiawu Zheng , Rongrong Ji , Xiaoshuai Sun , Yongjian Wu , Feiyue Huang , and Yanhua Yang . Centralized ranking loss with weakly supervised localization for fine-grained object retrieval . In Proc . Int . Joint Conf . Artificial Intell. , pages 1226–1233 , 2018 . [ 48 ] Xiawu Zheng , Rongrong Ji , Xiaoshuai Sun , Baochang Zhang , Yongjian Wu , and Feiyue Huang . Towards optimal fine grained retrieval via decorrelated centralized loss with normalize-scale layer . In Proc . Conf . AAAI , pages 9291–9298 , 2019 .

Summary: In this paper, authors propose to learn attribute-aware hash codes for fine-grained image retrieval, which establish the explicit correspondences between hash codes and visual attributes.   The attention mechanism is employed to extract the fine-grained tailored visual patterns from both global-level and local-level, and unsupervised reconstruction task is defined to distill the attribute related vectors through an encoder-decoder structure. Finally the hash codes are generated by the attribute vectors.