Learning to Parameterize Visual Attributes for Open-set Fine-grained Retrieval
Authors: Shijie Wang
Jianlong Chang
Haojie Li
Zhihui Wang
Wanli Ouyang
Qi Tian
Venue: NeurIPS 2023 poster (2023)
Keywords: Open-set Fine-grained Retrieval
Visual Attribute
Unknown Categories
TL;DR: A novel algorithm is proposed to transform retrieval models from category semantic extraction to attribute modeling.

Abstract: Open-set fine-grained retrieval is an emerging challenging task that allows to retrieve unknown categories beyond the training set. 
The best solution for handling unknown categories is to represent them using a set of visual attributes learnt from known categories, as widely used in zero-shot learning. Though important, attribute modeling usually requires significant manual annotations and thus is labor-intensive. Therefore, it is worth to investigate how to transform retrieval models trained by image-level supervision from category semantic extraction to attribute modeling. To this end, we propose a novel Visual Attribute Parameterization Network (VAPNet) to learn visual attributes from known categories and parameterize them into the retrieval model, without the involvement of any attribute annotations.
In this way, VAPNet could utilize its parameters to parse a set of visual attributes from unknown categories and precisely represent them.
Technically, VAPNet explicitly attains some semantics with rich details via making use of local image patches and distills the visual attributes from these discovered semantics. Additionally, it integrates the online refinement of these visual attributes into the training process to iteratively enhance their quality. Simultaneously, VAPNet treats these attributes as supervisory signals to tune the retrieval models, thereby achieving attribute parameterization. Extensive experiments on open-set fine-grained retrieval datasets validate the superior performance of our VAPNet over existing solutions.

Learning to Parameterize Visual Attributes for Open-set Fine-grained Retrieval
Shijie Wang', Jianlong Chang", Haojie Li Zhihui Wang', Wanli Ouyang', Qi Tian?
International School of Information Science & Engineering, Dalian University of Technology, China
Huawei Cloud & AI, China
Colleeee of Computer and Engineering, Shandong University of Science and Technology. China *Shanghai Artificial Intelligence Laboratory, China
Abstract
Open-set fine-grained retrieval is emerging challenging task that allows retrieve unknown categories beyond the training set. The best solution for handling unknown categories is represent them using set of visual attributes learnt from known categories, as widely used in zero-shot learning. Though important, attribute modeling usually requires significant manual annotations and thus is labor-intensive. Therefore, it is worth to investigate how to transform retrieval models trained by image-level supervision from category semantic extraction to attribute modeling. To this end, we propose novel Visual Attribute Parameterization Network (VAPNet) learn visual attributes from known categories and parameterize them into the retrieval model, without the involvement of any attribute annotations. In this way, VAPNet could utilize its parameters to parse a set of visual attributes from unknown categories and precisely represent them. Technically, VAPNet explicitly attains some semantics with rich details via making use of local image patches and distills the visual attributes from these discovered semantics. Additionally, it integrates the online refinement of these visual attributes into the training process iteratively enhance their quality. Simultaneously. VAPNet treats these attributes as supervisory signals to tune the retrieval models, thereby achieving attribute parameterizaaion. Extensive experiments open-set fine-gu tne-grained retrieval datasets validate the superior performance our VAPNet over existing solutions.
1 Introduction
Fine-grained image retrieval attempts to build a well-generalized embedding space where the vi- sual discrepancies among categories are clearly reflected. It plays vital role in numerous vision applications from fashion industry, e.g., retrieval of different types of shoe or clothes [22] I. to envirommental conservation, e.g., retrieval endangered species 66 38 36 32). However, real-world applications probably face the input of unknown categories, and the model will treat them as known ones. As result, the retrieval performance decays, which unbearable in real-world applications. Open-set fine-grained retrieval is thus proposed to conduct training on known categories but retrieve unknown ones during evaluation.
Facing the unknown inputs of novel categories, an intuitive way to identify them is to capture the discriminative discrepancies of unknown categories from these inputs for identifying them. However, most existing works [19] 24; 28; 20) still focus on discriminative concepts of known categories and only capture them from unknown instances, consequently making it hard precisely identify unknown categories. Interestingly, zero-shot learning [II 18 47 has proven that an unknown instance can be described integrally using variety of visual attributes, and these attributes can be "Corresponding author: hjli @ (dsuus.eduu.cn
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

discovered on multiple known cate- gories. For example, the unknown birds in Fig. 1 can be represented us- ing visual attributes discovered from seen instances, and the combination of these attributes can clearly reflect their discrepancies, thus alleviating the problem behind open-set settings. Though important, attribute modeling usually requires significant manual an- notations and thus is labor- inboorintensive. When attribute annotations are un- available, how to transform retrieval models trained by image-level super- vision from category semantic predic- tion to attribute modeling is worthy of investigation.
known Categories
Inpuate
(Cronn Heud)
Spor Wings)
Gallegoliis
@(Yellow_Fad)
o[Black Cluws)
Sun 1008
#(White Wing]
GNN
eCY @(Vellow_Beld
9(
Dianriminaiie Arribuos
Visual Attribute
Visual Attribure Exploration
Common Attrbutes
Paramatirs
Figure I: Motivation of the proposed VAPNet as well as the main process of retricving unknown categorics based on our visual at- tribute parameters. The key idea promoted throughout the paper is that the visual attributes within known categories promote the un- derstanding of unknown inputs of novel categories. The backhone network (CNN N) with this key idea can he transformed into a set of visual attribute parameters (()). Therefore, given visually similar images of unknown categories (Ca, Ca). we feed them into our VAPNet generate the retrieval embeddings containing various attributes, and then calculate their cosine similarity (Sim 0.08) to determine whether to be distinguished. VAPNet with specific parameters transformed by visual attributes can procure in-depth semantic pattern understanding for unknown categorics, improving the open-set retrieval performance eventually.
In this paper, we present a Visual Attribute Parameterization Network, termed as VAPNet, aiming to distill vi- sual attributes from various semantics presented on seen fine- grained objects, and utilize these attributes to tune the retrieval model. Consequently, VAP- Net describes the appearance of the input instances of novel categories based on its parameters tuned by visual attributes, thus trans- forming the retrieval model from category semantic prediction to attribute modeling. Notably, due to lacking of attribute annotations, the attributes derived from VAPNet will be not restricted to pre-defined attributes like supervised-beid
attribute learning works 10
Technically, VAPNet needs to parse various semantics presented in fine-grained objects, which is prerequisite for attribute modeling. However, feature extractor trained by image-level labels tends to focus on a few primary semantic regions (e.g., bird's head) while ignoring other visual clues (e.g. bird's body). We empirically observe that vision models object regions with rich details when taking local image patches as input compared to the whole image. Therefore, VAPNet attains some rich semantics presented in objects via parsing multiple local views randomly cropped from the input image. After that, we further apply an encoder to project these discovered semanties to a set of visual attributes. Nevertheless, due to lacking the attribute annotations, these attributes usually include some noisy patterns. To handle this limitation, we incorporate the online refinement of these attributes into the training process iteratively improve their quality and simultancously regard these attributes as supervision signals to tune the retrieval model, thus achieving attribute parameterization. Specifically, we design another encoder with the same structure as the above one to produce another set of visual attributes from the global features, which are used match the visual attributes inferred by local views for providing rich supervisory signal. To avoid optimizing two encoders instead of the retrieval model, we design the counterparts of two encoders by accumulating their parameters of all previous iterations to make supervisory signals provided by attributes tune the retrieval model directly. In this way, the features outputted by the retrieval model can be iteratively improved and fed into the optimized encoders to provide more accurate visual attributes, which, in turn, better tunes the retrieval model for visual attribute modelling.
discover these overlooked
Contributions of this paper are summarized as below:
To the best of our knowledge, we are the first to transform the retrieval model trained by image-level supervisions from category semantic prediction into attribute modeling, thus alleviating the problem behind open-set fine-grained retrieval settings.
We propose a novel Visual Attribution Parameterization Network, which distills visual attributes from various semantics discovered on seen fine-grained objects, and transcribes these attributes into parameters within the retrieval model, thus representing unknown categories precisely based on its parameters transformed by visual attributes.

---

Extensive experiments show that open-set fine-gra tne-ggained retrieval task can benefit from the proposed method, and thus our VAPNet obtains significant gains of 8.6% average accuracy over recent state-of-the-art work 33 on three open-set fine-grained retrieval benchmarks.
2 Related Work
Open-set fine-grained retrieval. Existing open-set fine-grained retrieval works can be roughly divided into several groups. The first group, localization based scheme, utilizes the supervision of category signals to learn discriminative embeddings [42] 24 40). CRL [52] designs an attractive object feature extraction strategy to facilitate the retrieval task. Despite the inspiring achievement, the shortcoming of these works is that they only focus on individual samples while neglecting the inter-class and intra-class correlations between subcategories, thus reducing the retrieval performance. Therefore, the second group. metric-based scheme, is learning an embedding space where similar examples are attracted, and dissimilar examples are repelled (33
15 27 51 14 50). NIA (28
enforces unique translatability of samples from their respective class proxies to bring the distance of samples with the same subcategory closer. However, they still capture the discriminative details of known categories from unknown instances but neglect more details on undiscovered semantic regions, consequently impairing the retrieval performance.
Unlike the above works, FRPT 35 steers a frozen pre-trained model to perform the fine-grained retrieval task from the perspectives of sample prompting and feature adaptation. PLEor 34 could leverage pre-trained CLIP model to infer the discrepancies encompassing both pre-defined and unknown subcategories, and transfer them to the backbone network trained in the close-set scenarios. Nevertheless, it is worth noting that both of these approaches typically require more computational resources to optimize the retrieval models. This can potentially limit their practical applicability in real-world scenarios. To alleviate the problem behind open-set scenarios, we design VAPNet to explore and exploit visual attributes learnt from known instances instead of learning discriminative clues to anticipate open-set class data, improving retrieval performance in open open-world scenarios accordingly.
Visual attributes. Attributes belong to intuitive properties of objects, which contain low-level semantics (e.g., color, texture and shape), high-level semantics (e.g., head, body and tail of objects), or even common sense (e.g., birds living on the tree) [7]. Utilizing visual attributes makes great progress various vision tasks, including image search (17), fine-grained recognition
49 37, scene
understanding 255) and so on. Most of the previous works based on attribute learning 10 49
usually require significant manual attribute annotations and therefore is labor-intensive. Besides, the attributes learnt by these works are also restrained to pre-defined attribute labels, consequently ignoring some potentially vital information lying in visual semantics. To alleviate the aforementioned issues, recent works [43(3] 39] formulate an unsupervised learning strategy to project the learnt features into an attribute space. However, although the two works achieve superior performance on their corresponding vision tasks, they still are rooted in the close-set scenarios and thus make it hard to handle unknown instances. Therefore, we propose VAPNet to process more challenging scenarios, i.e., oet fine- tne-grained retrieval tasks, by making full use of known data.
3 Methodology
The overall structure of VAPNet is shown in Fig. 2. It clear that our network is mainly organized by three modules: retrieval module, attribute exploration module and attribute parameterization module. The retrieval module could extract retrieval embeddings encompassing various attributes of input objects for retrieving visually similar objects. The attribute exploration module is designed to randomly extract visual attributes from known categories. In addition, the attribute parameterization module is responsible for improving visual attributes and utilizing them supervisory signals to tune the retrieval model.
3.1 Retrieval Module
The retrieval module aims at extracting basic image representations using the backbone network and producing retrieval embeddings. Thereby, the backbone network can be regarded as the retrieval model. Formally, given image X. let F R W be the C-dimensional with H x W feature
3

---

c: No learnable
Retrieval Module ($3. 1)
Attribute Parameterization Module ($3. ()
parameters
Global Features
local Attribute
Sampling
Feature
Encoder Ta
Encoder E(T)
T CHN
Fe
Bs
Reuse
Pooling
Accumulation
Purometer
Sampling
Retrieval Embeddings
Ra
Projection
Rosize
Local Attribute Feature
Attributes
Actribute
Encoder Tr
Encoder B(T))
Metric Space
FONN
Projection
Attribute Exploration Module ($3.
Porameter Accumulation
Figure 2: Detailed illustration of visual attribute parameterization framework. Our algorithm can be clearly divided into three main components: retrieval module, an attribute exploration module (AEM), and an attribute parameterization module (APM). The AEM is responsible for extracting set of visual attributes from local views, while the APM designed iteratively enhance the quality these attributes. These attributes are then used as supervisory signals tine-tune the parameters of the retrieval model. It is important to note that during testing, the single retrieval module acts as retrieval model. The AEM and APM are only utilized during the training phase to improve the attributes and fine-tune the retrieval model parameters.
planes encoded by a backbone network JONN (X). Thus the most common way for retrieval is embed the final feature F by using global average pooling operations (GAP), calculating mean values on the H x W feature plane and producing the final retrieval embeddings ER E RC. It should be clarified that VAPNet does not introduce extra computation overhead during evaluation.
3.2 Attribute Exploration Module
Facing the unknown input of novel categories, VAPNet aims to explore all the attributes presented in known instances much as possible and utilize them to understand unknown categories. A non-negligible problem is that feature extractor trained by image-level labels tends to focus few primary semantic regions (e.g., bird's head) while ignoring other visual clues (e.g. bird's body). Fortunately, feature extractor could discover object regions with rich details when replacing the input image with its local patches, as verified in Fig. 3. This suggests a proper way to focus on some overlooked object regions by making use of local image patches. Therefore, module is proposed to attain some semantic clues of an input object via randomly cropping local patches from the input image. These collected semantic clues could be translated into visual attributes describing pre-defined and unknown categories.
attribute exploration
Input sampling. Given an input image X, we equally split it into X n patches which have dimensions. Here, the granularities of patches are controlled by the hyper-parameter
3 x | In our experiments, respectively equals to 2, 4, 8 and 16, and thus the number of patches is 340. We randomly sample M (M & 340) patches from the candidate set each iteration to construct set of different local views V = [V, V..... VM). These local views are resized to the same scale as inputs via using bilinear interpolation. The magnification operation could directly highlight the subtle
discriminative details in the local views, further making the backbone network more sensitive to these subtle details.
Then, the local views V are passed through the backbone network FCNN:
FL = FCNN (V),
where FT. E MXCXHXW is the local attribute feature set.
It should be clarified that the input sampling step samples local patches randomly, so it can treat all object patches equally, regardless of their discriminative ability. However, should not be bad news for open-set fine-grained retrieval tasks, since the more diverse the visual attributes are, the

---

better the understanding of the unknown categories is. Furthermore, although this module picks out few patches each iteration, could effectively parse the holistic structures of objects based on the accumulation of selected patches by multiple iterations.
Attribute exploration. In real-world applications, especially for the fine-grained tasks with small inter-class variances, attribute annotations are expensive and therefore labor-intensive. Nevertheless, unsupervised learning [26] can capture regularities in data for the purpose of extracting useful knowledge or for restoring corrupted data. Therefore, many unsupervised works [26] 2 explicitly produce internal latent units or codes from feature representation. Inspired by this, we design visual attribute exploration to project these visual semantics attained by local views into a latent space, i.e., the attribute space:
AL T((F) = WL g(FL) + by,
where Tr. denotes the encoder with the weight matrix Wr, E Bexk and bias vector by E R to process the local features produced by local views, and g(-) is the global average pooling operation. By the projection operation, we successfully transform the visual semantics into the visual attributes [AL E R 1,... MJ. this way, these visual attributes could correspond to local semantic knowledge of fine- grained objects ( e.g., red head, spotted wings, etc.), so the group of certain attributes can clearly describe an unknown category and reflect its discriminative discrepancies.
3.3 Attribute Parameterization Module
Visual attributes produced by local views serve as supervisory signals tune the retrieval model, further make it be transformed from category semantic extraction to attribute modelling. Nevertheless, due lacking the attribute annotations, these visual attributes substantively include some noisy patterns. The retrieval model supervisied by these noisy attributes is harmful to parsing fine-grained objects. To this end, we propose an attribute parameterization module to incorporate the online refinement of these attributes into the training process to iteratively improve them and simultaneously regard these attributes as supervisory signals tune the retrieval model. In this way, the retrieval model could capture visual attributes from input instances, thus achieving attribute parameterization.
Attribute sampling. To match visual attributes provided by local views, we need to process the final features F inferred by an input image X to produce another group of attributes. which are used to match these visual attributes provided by local views. Concretely, forwarding F into sampler extracts the corresponding local attribute feature sets FG = [ F FM] according to four coordinates of local views. Specifically, we utilize RoIAlign operation proposed in Mask-RCNN [8] to accurately extract the corresponding local features. Then, we project these local representation in another attribute space:
A = (FF))) = Wa g(Fis) + bg:
where To represents an encoder with the weight matrix We E gexk and bias vector be E R* to process the local features produced by the final features, and A/ E R² denotes the i-th attribute in the attribute set AG E Mxk
With these visual attributes including AL and Ag and local features containing FL and FG, the attribute pairs A and local feature pairs Fp be organized
A [(A], A (A2, AG)........ (AM AM)
Fp [FL, F ), (FL. ......... FM)
Attribute parameterization constraint. This constraint is responsible for improving these visual attributes and utilizing them to tune the retrieval model. Specifically, as the local attribute features fed to each encoder come from global and local views, this encoder only distills visual attributes only from its corresponding view. Thus, given local attribute features, no matter which view they come from, if two encoders provide the same attribute, it means this feature can be regarded as from both two sources. other words, the feature discrepancy between the local attribute features inferred from both global and local views is effectively eliminated. Thereby, the above process could optimize two encoders to iteratively improve visual attributes, and utilize these attributes to modify the features inferred by local and global views, thus achieving attribute parameterization.
5

---

To this end, the attribute parameterization constraint Le can be formulated as:
M
(c(FL) + Tr. (F. (. A
F.( TT log
(5)
AL
This loss encourages the two encoders to produce the same attributes for the same visual content, no matter which view it comes from, thus achieving attribute parameterization. However, training the model with Eq. (5) directly will make the attributes provided by two encoders become similar quickly since the encoders learn the attributes from another view according Eq. (5). Therefore, using Eq. (5) more optimizes the parameters of two encoders, but has less impact on the parameters of the retrieval model.
To handle this limitation, we propose two mean encoders with the same structure as the above ones to produce attributes for features of another view. In this way, Eq. (5) can be written
(L + ETTL)( )log AL
(F)) A
- i=1
(6)
where E[TG] and EiT denote the mean encoders without learnable parameters, respectively. Their parameters can be updated in a temporal average manner. Concretely, at the 1-th iteration, parameters (OTG](OG) and ((7,.)(0r.) are accumulated by
E (Ta((O) (1 (1--a)a (l)a)E(t-a (1-a))((""))aa (T]]oc) + abc.
(TL = = - - + atL,
where [Ta]())))) (E (T. and ( (E.)'" respectively denote the parameters of mean encoders current iteration and last iteration, and 0g = (Wa, bg) and 8L (WL, bu. are the learnable parameters of Pg and PL at the current iteration, respectively. The mean encoders are initialized EX (0g) = OG and (TE](O) Oz. The hyper-parameter is the updating ratio within the range of [0, 1).
Since the two mean encoders do not introduce learnable parameters, the attribute parameterization constraint could directly penalize the retrieval model and make its parameters be adjusted through back propagation. More importantly, the mean encoders could consider the knowledge learned from all previous stages to form more robust attributes for the current stage. Therefore, they have another important property that could remain sensitive even for rare attributes, consequently staying well generalized when facing unknown categories. After the attribute parameterization operation, VAPNet will transform the retrieval model from category semantic extraction to attribute modeling, allowing the utilization visual attributes to anticipate open-set class data.
3.4 Loss Functions
The retrieval model with specific parameters supervised by visual attributes can extract the attributes presented in objects and further procure in-depth semantic pattern understanding. For fine-grained understanding, these extracted visual attributes should clearly reflect the discrepancies of objects, so that we can better identify visually similar objects. Thereby, we propose an auxiliary constraint based on the cross-entropy loss to ensure that these extracted visual attributes can contribute to decision boundary:
C. =
(
where denotes the ground-trute label of the corresponding input, and C(-) E x N is the number of category in the training set.
The total loss L of VAPNet can be formulated as:
XCc.
where À is the hyper- parameerr to balance the contributions of the individual loss item.

---

4 Experiments
4.1 Experimental Setup
Datasets. CUB-200-2011 dataset 5 contains 200 bird subcategories with 11,788 images. We utilize the first 100 classes (5,864 images) in training and the rest (5,924 images) in testing. The Stanford Cars dataset [16] contains 196 car models of 16, images. The spilt in Stanford Cars [16] is also similar CUB, which split into the first 98 classes (8,054 images) for training and the remaining classes (8, 131 images) for testing. FGVC Aircraft dataset 23 is divided into first 50 classes (5,000 images) for training and the rest 50 classes (5,000 images) for testing. Shop Clothes Retrieval (In-Shop) [2] contains 7,982 subcategories with 52, 712 images, and we use the 3,997 classes (25,882 images) in training and the rest 3,985 classes in testing. In-Shop is divided between a query (14,218 images) and gallery set (12, 162 images).
Evaluation protocols. We evaluate the retrieval performance by Recall@ K with cosine distance, which is average recall scores over all query images in the test set and strictly follows the setting in the pioneer work [3]]. Specifically, for each query, our model returns the top K similar images. In the top K returning images, the score will be if there exists at least one positive image, and O otherwise.
Implementation Details. For backbone network, we apply the widely-used Resnet-50 [ in our experiments with the pre-trained parameters. The input raw images resized to 256 x 256 and cropped into 224 x 224. We train our models using Stochastic Gradient Descent (SGD) optimizer with weight decay of 0.0001, momentum of 0.9, and batch size of 32. We adopt the commonly used data augmentation techniques, i.e., random cropping, left-right flipping, and color jittering for robust feature representations. Our model is trained end-to-end one NVIDIA 2080Ti GPUs for acceleration. The initial learning rate is set to 10 15, with exponential decay of 0.9 after every 5 epochs. The total number of training epochs is set to 200.
4.2 Ablation Study
The proposed VAPNet is optimized by a combination of two loss functions, an aux- iliary loss C. and an attribute parameter- ization constraint Le or Le. which play different roles in guiding our model to un- derstand unknown categories. Here, we perform thorough ablation experiments on CUB-200-2011 and Stanford Cars datasets to further validate the effective- ness of each loss function. Tab. I shows quantitative comparisons between differ- combinations of loss functions. The baseline method only using La obtains 69.5% and 89.3% Recall@ 1 accuracy on CUB-200-2 I and Stanford Cars datasets, respectively. The results reflect that the network only learns the discriminative object regions instead of the visual attributes, consequently impairing the retrieval performance of unknown categories. An addition of L. improves Recall @ I from 69.5% 71.4%. However, La optimizes the encoders to translate the semantics into visual attributes, instead of parameterizing the attributes into the retrieval model. During testing, our VAPNet still makes it hard to handle unknown categories, thus limiting the performance gains. To handle this limitation, we improve the attribute parameterization constraint to force this loss function to directly optimize the parameters
Table I: Comparison of performance and efficiency on CUB- 200-2011 and Stanford Cars datasets using different combina- tions of constraints. "R@ 1" denotes the Recall @ I retrieval performance. "Time" is the time of extracting retrieval embed- dings.
La
L. Ee
CUB R@1
CAR R@ I
Time
/
69.5%
89.3%
21.1ms
71.4%
91.2%
74.1%
92.7%
21.1 ms
76.2%
94.8%
within backbone network. As expected, E. can effectively make the backbone network sensitive to visual attributes and understand unknown categories accordingly. Furthermore, we also verify the effectiveness of Lu, which can ensure that these visual attributes learnt from known categories keep discriminative. As shown in Tab. 1, the proposed VAPNet achieves 76.2% and 94.8% Recall @ performance owing to the combination of L. and L. on two widely-used benchmarks. Additionally, during testing, the retrieval embedding extraction time remains the same as that of the baseline model, as the additional attribution exploration modules (AAM and APM) are only used during training.
7

---

Table 2: Comparison of different methods CUB-200-2011, Stanford Cars 196 and FGVC Aircraft datasets.
CUB-200
200-2011
Stanford Cars 196
FGVC Aircraft
Method
2
4
8
2
4
8
2
4
8
SCDA 42
57.3
70.2
81.0
88.4
48.3
60.2
71.8
81.8
56.5
67.7
77.6
85.7
PDDM 3
58.3
69.2
79.0
88.4
57.4
68.6
80.1
89.4
CRL 52
62.5
74.2
82.9
89.7
57.8
69.1
78.6
86.6
61.1
71.6
80.9
88.2
CEP 4
69.2
79.2
86.9
91.6
89.3
93.9
96,6
98.1
HDCL 46
69.5
79.6
86.8
92.4
84.4
90.1
94.1
96.5
71.1
81.0
88.3
93.3
DGCRI
67.9
79.1
86.2
91.8
75.9
83.9
89.7
94.0
70.1
79.6
88.0
93.0
DCML 50
68.4
77.9
86.1
91.7
85.2
91.8
96.0
98.0
DRML 51
68.7
78.6
86.3
91.6
86.9
92.1
95.2
97.4
DAS 20
69.2
79.3
87.1
92.6
87.8
93.2
96.0
97.9
IBC
70.3
80.3
87.6
92.7
88.1
93.3
96.2
98.2
NIA 28
70.5
80.6
89.1
93.4
Proxy
13
71.1
80.4
87.4
92.5
88.3
93.1
95.7
97.5
HIST 19
71.4
81.1
88.1
89.6
93.9
96.4
ETLR
14
72.1
81.3
87.6
89.6
94.0
96.5
PNCA
72.2
82.0
89.2
93.5
90.1
94.5
97.0
98.4
VAPNet
76.2
84.6
90.1
94.0
94.8
96.3
98.0
98.6
87.2
91.7
95.0
96.3
4.3 Comparison with the State-of-the-Art Methods
Open-set Fine-grained Object Retrieval. We compare VAPNet with ffate-of-the-ar approaches. In Tab. 2, the performance of different methods on CUB-200-2011 Stanford Cars- 196, and FGVC Aircraft datasets is reported, respectively. In the table from top bottom, the methods are roughly divided into three groups, i.e., localization-based networks, metric-based frameworks, and our VAPNet.
As shown in Tab. 2, it obvious that the retrieval performance obtained by our VAPNet is better than other methods matter whether the localization- based metric-baat based schemes are adopted. Concretely, existing works based on localization schemes, i.e., CEP 4 and HDCL [46], tend to project the final retrieval embeddings into category space. Despite the encouraging achievement, the shortcoming of these works that they only focus on individual samples while neglecting the correlations among subcategories, thus limiting the retrieval performance. To address this problem, the effectiveness of these models based on metric schemes, i.e., ETLR [1] and PNCA (33 can be largely attributed to their precise identification of negative/positive pairs through the manipulation of distances, which indirectly enhances the discriminative power of features. However, these existing works, e.g., CEP 4. HIST [19] and PNCA (33), follow close-set learning setting, where all the categories are pre-defined, to learn the discriminative and generalizable embeddings for identifying the visually similar objects of unknown subcategories.
thus very challenging for feature extractor trained in closed-set scenarios with classification or metric supervisions to capture discrimi- native discrepancies from unknown subcategories, consequently impairing the retrieval performance. To handle this limitation, our VAPNet focuses on learning visual attributes instead of discrim- inative clues to understand the unknown cate- gories and clearly reflect their discriminative dis- crepancies, thus achieving clear improvement of Mfate-o--hhe-at methods.
Table 3: Comparison of different state-of oftte-of-thearr meth- ods on In-shop dataset.
method
10
20
30
40
HDC 45
62.1
84.9
89.0
91.2
92.3
ABE 12
87.3
96.7
97.9
98.2
98.5
Large-scale Product Retrieval. Our VAPNet exceeds all the existing methods and achieves the best performance with retrieval accuracy of 93.9%, as shown in Tab. 3. Besides, we beat the second-best work CEP [ and get a better result with a relative accuracy improvement of 3.0%. By leveraging the visual attributes learned from known instances to identify category-specific discrepancies, our VAPNet demonstrates impressive generalization capabilities.
EPSHN
87.8
95.7
96.8
NSM 48
89.4
97.8
98.7
99.0
MS 41
89.7
97.9
98.5
98.8
99.1
CEP 4
90.6
98.0
98.6
98.9
99.1
PNCA 33
90.9
98.2
98.9
99.1
99.4
Our VAPNet
93.9
98.7
99.1
99.4
99.6
8

---

4.4 Discussions
Patch number M. Tab. 4 ablates the role of patch number & in the attribute exploration module ($3.2). The optimal value is M 4 (our default). More- over, VAPNet is robust when M is in [4,..., 16). showing that it is beneficial to spot visual attributes in a relatively many local regions. It is worth mentioning that when M is too large, the training time grows exponentially. However, when M is too small, the performance degrades due to easily overlooking some undiscovered regions. The results reveal that the local regions help the model attain accurate attributes, leading to better understanding unknown categories.
Table 4: The retrieval accuracy CUB-200-2011 of model trained with different number M of local views in $3.2.
Number M
4
16
R 73.6%
74.9%
76.2%
76.1
76.2%
Attribute dimension k. We investigate the necessity of diverse attribute dimen- sions for retrieval performance. As re- ported in Tab. 5, the dimension & stores the attribute knowledge. Although the large dimension could hold more infor- mation related to attributes and has less impact on retrieval performance, it is easy to contain more useless information and increase storage overhead. However, when the dimension is small, it is not enough to precisely represent visual attributes, leading to degraded performance. Therefore, the optimal dimension is k 256.
Table 5: Comparison of model trained with different dimension k of visual attributes on CUB-200-2011.
Dim
32
128
256
512
Recall@1 74.7%
75.9%
76.1%
76.2%
76.0%
Updating ration a. Tab. reports the accuracy of using diverse updating tios in Eq. (7). Notably, after increasing the updating ratio, the retrieval perfor- mance reduces progressively. These re- Recall@1 75.9%
Table Evaluation results on LUB-200-2011 of model trained with different updating ratio in Eq. (7).
Ratio a
0.1
0.2
0.4
0.6
76.2%
75.1
73.9%
72.6%
sults reveal that large updating ratio
quickly updates the projectors more relying on the learning parameters on the current stage, thus easily degrading the discrepancies between two projectors. Moreover, when using small updating ratio, the projectors keep sensitive to previous learning knowledge and easily keep different during optimization, thus extracting precisely visual attributes from given features.
Balanced parameter There is one balanced parameter in Eq. (10). The sensitivity analysis of the parameter are and the evaluation results are presented in Tab. It is observed that the performances of our VAPNet are not stable with the variation of À (from to 20). The retrieval performance increases À grows 10, the consistency constraint with a large balanced parameter would force the network to focus on visual attributes. However, this constraint has a larger balanced parameter, which makes the network neglect the discriminative ability of visual attributes, thus reducing retrieval performance. The upper bound of retrieval performance may saturate at À 10 for learning visual attributes from known categories.
Table 7: Quantitative performance of model trained with diffcrent weight À in loss function in Eq. (10) on CUB-200-2011. 1.
performed CUB-200-200
Weight À
10
15
20
Recall@1 73.8%
75.2%
76.2%
75.0%
74.9%
4.5 Visual Attribute Analysis
Interpreting visual attributes is difficult because these attributes are optimized in latent space. We resort to an indirect way to interpret these attributes by visualizing their sources (i.e., Fig. 3) to display the content within them, and the features influenced by them (i.e., Fig. 4) to indirectly track these visual attributes.
Our VAPNet distills visual attributes from some local regions randomly cropped from inputs. There- fore, we provide some activation maps generated by Grad-CAM (30 to display some visual clues of interest in the attributes. In Fig. 3, these semantics provided by local regions could grab some rich details, and thus attributes projected by them could clearly represent these regions. Besides, we can also observe that the response maps of the local views highlight more object details compared to that

---

[4]
19
Figure 3: Visualization of the source of attributes. The top row shows the global view. The second and third rows show the multiple image patches after random cropping (local views). (a), (c) and (c) denote the input views. (b), (d) and (f) show the response maps.
Figure 4: Illustration of class activation maps pro- duced by baseline and our VAPNet. (a) and (d) the input images. (b) and (e) are the referred class activation maps by baseline. (c) and (f) denote the class activation maps provided by our VAPNet.
of the global view. Based on the above observation, we draw a conclusion that vision models can discover more semantic clues when replacing the input image with its local patches.
We exhibit the visualization results to demonstrate the influence of visual attributes. The referred visualizations of baseline and our model are shown in Fig. 4. It is shown that our model focuses on multiple local parts (e.g., head, wings and abdomen, etc.) instead of the fixed part predicted by baseline, (e.g., head of birds, front of cars, and middle of aircraft, etc). This verifies that using visual attributes learnt from known categories reasonably is beneficial for describing novel categories, thus improving the retrieval performance under open-set scenarios. More importantly. as shown in Fig. (c) and (f), two sub-figures in the same row can roughly correspond to certain kinds of attributes of the fine-gre finegrrined objects, e.g., "wings of birds", "tires of cars", "head or wing of planes", etc. The results reflect that the activation of objects parts is apparently attribute-related and contains the visual discrepancies among unknown categories accordingly, which could provide a clear explanation the success in retrieving unknown categories.
5 Conclusion
In this paper, we propose novel Visual Attribution Parameterization Network (VAPNet) to handle unknown categories using visual attributes learnt from known instances in open-set fine-grained retrieval tasks. VAPNet focuses on distilling visual attributes from semantic clues presented in objects and utilizing these attributes as supervisory signals to tune the retrieval model. In this way, we could transform the retrieval model trained by image-level supervisions from category semantic extraction to attribute modeling, and precisely represent unknown categories based on its parameters supervised by visual attributes. Therefore, VAPNet successfully alleviates the problem behind facing instances from unseen novel categories. Last but not the least, the overall retrieval pipeline is simple and flexible. Extensive experiments demonstrate that our method outperforms the ofthe-of-the-art methods by significant margin, indicating the effectiveness of attribute modelling on facing unknown categories.
Limitations & Broader Impacts: By introducing VAPNet, we aim to extract visual attributes from seen classes without relying on attribute annotations to differentiate unseen classes. This innovation has the potential to greatly impact open-domain tasks. In particular, annotating a large number of attributes for unseen categories in open-domain tasks can be costly and time-consuming endeavor. By enabling the model to automatically capture knowledge about unseen classes, our approach reduces the reliance on attribute annotations, resulting in decreased manual labeling costs. Furthermore, our approach exhibits improved adaptability to data from domains resembling the training set, such as natural images or medical images. This heightened adaptability contributes to stronger generalization capabilities, allowing the model to perform well in real-world scenarios. Ultimately, our solution has the potential to propel the advancement of open-domain tasks and facilitate their practical applications.
10

---

Acknowledgements
This work supported in part by the National Natural Science Foundation of China (NSFC) under Grants (No.61976033 and No.61932020), and The Taishan Scholar Program of Shandong Province (tstp20221188).
References
[1] Kenan Ak, Ashraf A. Kassim, Joo-Hwee Lim, and Jo Yew Tham. Learning attribute representations with localization for flexible fashion search. In 2018 IEEE Conference on Computer Vision and Pattem Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 7708-7717 Computer Vision Foundation IEEE Computer Society, 2018
[2] Francesco Barbato, Marco Toldo, Umberto Michieli, and Pietro Zanuttigh. Latent space regularization for unsupervised domain adaptation in semantic segmentation. In CVPR, pages 2835-2845. Computer Vision Foundation IEFE, 2021.
[3] Scan Bell and Kavita Bala. Learning visual similarity for product design with convolutional neural networks. ACM Trans. Graph., 34(4):98:1-98 10, 2015.
[4] Malik Boudiaf. Jérôme Rony, Imtiaz Masud Ziko, Eric Granger. Marco Pedersoli, Pablo Piantanida, and Ismail Ben Ayed. A unifying mutual information view metric learning: Cross-entropy vs. pairwise losses, In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, cditors, ECCV, volume 12351 of Lecture Notes in Computer Science, pages 548-564. Springer, 2020.
[5] Steve Branson, Grant Van Horn. Serge J. Belongie, and Pietro Perona. Bird species categorization using pose normalized deep convolutional nets. CoRR, hass/140..295. 2014.
[6] Mohamed Elhoseiny, Yizhe Zhu, Han Zhang, and Ahmed M. Elgammal. Link the head to the "beak": Zero shot learning from noisy text description part precision. In 2017 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 6288 6297. IEEE Computer Society, 2017.
[7] Vittorio Ferrari and Andrew Zisserman. Learning visual attributes. In John C. Plau, Daphne Koller, Yoram Singer, and Sam T. Roweis, editors, Advances Neural Information Processing Systems 20, Proceedings of the Twenty-Firg First Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, December 3-6, 2007, pages 433-440. Curran Associates, Inc., 2007.
[8] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In JEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 2980-2988. TEEE Computer Society, 2017.
[9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770 778, 2016.
[10] Junshi Huang, Rogério Schmidt Feris, Qiang Chen, and Shuicheng Yan. Cross-domain image retrieval with a dual altribute-aware ranking network. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 1062-1070. IEEE Computer Society, 2015.
[11] Dat Huynh and Ehsan Elhamifar. Fine- Fine-grained generalized zero-shot learning via dense attribute- based attention. In CVPR, pages 4492- Computer Vision Foundation/ / IEEE, 2020.
[12] Dor Kedem, Stephen Tyree, Kilian Q. Weinberger. Sha, and Gert R. G. Lanckriet. Non-linear metric learning. In Peter L. Bartlett, Fernando C. N. Pereira, Christopher J. C. Burges, Léon Bottou, and Kilian Q. Weinberger, editors, NewrIPS, pages 2582-25525 2290,590,22 2012.
[13] Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. Proxy anchor loss for deep metric learning. In CVPR, pages 3235-3244. Computer Vision Foundation / IEEE, 2020.
[14] Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. Embedding transfer with label relaxation for improved metric learning. In CVPR, pages 3967-3976. Computer Vision Foundation IEEE, 2021.
[15] ByungSoo Ko, Geonmo Gu, Han-Gyu Kim, and ByungSoo Ko. Learning with memory- virtual classes for deep metrie leaming. In ICCV, pages 11772-11 11781. IEEE, 2021.
[16] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV Workshops 2013, Sydney, Australia, December 1-8, 2013, pages 554-561, 2013.
11

---

[17] Neeraj Kumar, Alexander C. Berg, Peter N. Belhumeur, and Shree K. Nayar. Describable visual attributes for face verification and image search. IEEE Trans. Puttern Anal. Mach. Intell., 33610)11962-1977 2011.
[18] Zechao Li. Hao Tang, Zhimao Peng. Guo-Jun Qi, and Jinhui Tang. Knowledge -oiided semantic transfer network for few-shot image recognition. IEEF Tronsactions Neural Networks and Learning Systems, pages 1-15, 2023.
[19] Jongin Lim, Sangdoo Yun, Seulki Park, and Jin Young Choi. Hypergraph-induued semantic tuplet loss for deep metrie learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. pages 212 222. IEEE, 2022.
[20] Lizhao Liu, Shangxin Huang, Zhuangwei Zhuang, Ran Yang, Mingkui Tan, and Yaowei Wang. DAS: densely-anchored sampling for deep metrie learning. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, ECCV, volume 13686 of Lecture Notes in Computer Science, pages 399-477. Springer, 2022.
[21] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj. and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In CVPR, pages 6738-6746. IEEE Computer Society, 2017.
[22] Ziwei Liu, Ping Luo. Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In 2016 IEFE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 1096-1104. IEFE Computer Society, 2016.
[23] Subhramsu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. CoRR, abs/1306.5151, 2013. 2013.
[24] Olga Moskvyak, Frédéric Maire, Feras Dayoub, and Mahsa Baktashmotlagh. Keypoint- allgnndd embeddings for image retrieval and re-identification. In WACV, pages 676 685. IEEE, 2021.
[25] Genevieve Patterson, Chen Xu, Hang Su, and James Hays. The SUN attribute database: Beyond categories for deeper scene understanding. Int. J. Comput. Vis., 108(1-2):555 22)5-888,,, 2014.
[26] Mare' Aurelio Ranzato, Y Y-Lan Boureau, Sumit Chopra, and Yann LeCun. A unified energy- based frame- work for unsupervised learning. In Marina Meila and Xiaotong Shen, editors, Proceedings of the Eleventh International Conference Artificial Intelligence and Statistics, AISTATS 2007, San Juan, Puerto Rico, March 21-24, 2007, volume 2 of JMLR Proceedings, pages 371-379 JMIR.org, 2007.
[27] Karsten Roth, Timo Milbich, Björn Ommer, Joseph Paul Cohen, and Marzyeh Ghassemi. Simultaneous iimilarity-basee self-distillation for deep metric learning. In Marina Meila and Tong Zhang, editors, ICML., volume 139 of Proceedings of Machine Learning Research, pages 9095-9106. PMLR, 2021.
[28] Karsten Roth, Oriol Vinyals. and Zeynep Akata, Non-isotropy regularization for proxy-basee deep metric learning. In IEEE/CVF Conference Computer Vision and Pottern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 1410-7420. IEEE, 2022.
[29] Jenny Seidenschwarz. Learning intra- intra-batch connections for deep metric learning. In Marina Meila and Tong Zhang, editors, ICML., volume 139 of Proceedings of Machine Learning Research, pages 9410-9421.
PMLR, 2021.
[30] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- localization. Int. J. Comput. Vis., 128(2):3363359 2020.
[31] Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric leamning via lifted structured feature embedding. CVPR, pages 4004-4012, IEEE Computer Society, 2016.
[32] Hao Tang. Chengeheng Yuan, Zechao Li, and Jinhui Tang. Learning attention-guided pyramidal features for few-shot fine-erained recognition. Puttern Recognia. 130:108792, 2022.
[33] Eu Wem Teh, Terrance DeVries, Graham W. Taylor, and Graham. Proxynca++: Revisiting and revitalizing proxy neighborhood component analysis. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan- Michael Frahm, editors, ECCV, volume 12369 Lecture Notes in Computer Science, pages 448-464. Springer, 2020.
[34] Shijie Wang, Jianlong Chang, Haojie Li. Zhihui Wang, Wanli Ouyang, and Qi Tian. Open-set fine-grained retrieval via prompting vision-languaen language evaluator. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages (9381-1939) IEFE, 2023.
12

---

[35] Shijie Wang, Jianlong Chang, Zhihui Wang. Haojie Li, Wanli Ouyang, and Qi Tian. Fine-grained retrieval prompt tuning. In Brian Williams, Yiling Chen, and Jennifer Neville, editors, SheeethSeventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifhh Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposiun on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pages 2644 2652. AAAI Press, 2023.
[36] Shijie Wang, Haojie i. Zhihui Wang, and Wanli Ouyang. Dynamic position-aware network for fine-grained image recognition. In Thirty-Fifth AAAI Conference Artificial Intelligence, AAAI 2021, Thirty Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages
2791-2799. AAAI Press, 2021.
[37] Shijie Wang. Zhihui Wang. Haojie Li, Jianlong Chang, Wanli Ouyang, and Qi Tian. Accurate fine-grained object recognition with sruuture-driven relation graph networks. In ICV, 2023.
[38] Shijie Wang, Zhihui Wang. Haojie Li, Jianlong Chang, Wanli Ouyang, and Qi Tian. Semantic-guide guided information alignment network for fine-grained image recognition. IEEE Transactions Circuits and Systems for Video Technology, pages 1-1, 2023.
[39] Shijie Wang. Zhihui Wang. Haojie Li, and Wanli Ouyang. Category- Category-speciffce semantic coherency learning for fine-grained image recognition. In Chang Wen Chen, Rita Cucchiara, Xian-Sheng Hua, Guo-Jun Qi, Elisa Ricci, Zhengyou Zhang, and Roger Zimmermann, editors, MM '20: The 28th ACM International Conference on Multimedia, Virtual Event / Seattle, WA, USA, October 12- 16, 2020, pages 174-183. ACM, 2020.
[40] Shijie Wang. Zhihui Wang, Haojie Li, and Wanli Ouyang. Category-specific nuance exploration network for fine-grained object retrieval. In Thirty-Sixth Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Andirty-Fourty Conference on Innovative Applications of Artifical Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 AAAI Press, 2022.
March 1, 2022, pages 2513-2521.
[41] Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew R. Scott. Multi-similarity loss with general pair weighting for deep metrie learning. CVPR, pages 5022-5030. Computer Vision Foundation IEEE, 2019.
[42] Xiu-Shen Wei, Jian-Hao Luo, Jianxin Wu, and Zhi-Hua Zhou. Selective convolutional descriptor aggrega- tion for fine- -eniggr image retrieval. IEEE Trans. Image Process., 26(6):2868 2881, 2017.
[43] Xiu-Shen Wei, Yang Shen, Xuhao Sun, Han-Jia Ye, and Jian Yang. 2$- Learning attribute-aware hash codes for large-scale fine-grained image retrieval. In Mare' Aurelio Ranzato. Alina Beygelzimer. Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6- 14, 2021, virtual, pages 5720-5722 5730, 2021.
[44] Yandong Wen, Kaipeng Zhang, Zhifeng Li,, and Yu Qian. discriminative feature learning approach for deep face recognition. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, ECCV, volume 9911 of Lecture Notes in Computer Science, pages 499-515. Springer, 2016.
[45] Zhirong Wu, Alexei A. Efros, and Stella X. Yu. Improving generalization via scalable neighborhood component analysis. In Vittorio Ferrari, Martial Hebert, Cristian Sminchiseseu, and Yair Weiss, editors, ECCV, volume i 11211 of Lecture Notes in Computer Science, pages 712 728. Springer, 2018
[46] Xianxian Zeng. Shun Liu, Xiaodong Wang, Yun Zhang. Kairui Chen, and Dong Li. Hard decorrelated centralized loss for fine-grained image retricval. Neurocomputingg 453:26-37, 2021.
[47] Zican Zha, Hao Tang, Yunlian Sun, and Jinhui Tang. Boosting few-shot fine- tine-grained recognition with background suppression and foreground alignment. IEEE Trans. Circuits Syst. Video Technol. 33(8):9947- 3961, 2023.
[48] Andrew Zhai and Hao- Yu Wu. Classification is strong baseline for deep metric learning. In BMVC, page 91. BMVA Press, 2019.
[49] Hua Zhang, Xiaochun Cao, and Rui Wang. Audio visual attribute discovery for fine-grained object recognition. In Sheila A. Mcllraith and Kilian Q. Weinberger, editors, Proceedings of the Thirty- Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), 18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 7542-7549, AAAI Press, 2018
13

---

[50] Wenzhao Zheng. Chengkun Wang, Jiwen Lu, and Jie Zhou. Deep compositional metric learning. In CVPR, pages 9320-9339. Computer Vision Foundation IEEE, 2021.
[51] Wenzhao Zheng, Borui Zhang, Jiwen Lu, and Jie Zhou. Deep relational metric learning, In ICCV, pages 12045 12054. IEEE, 2021.
[52] Xiawu Zheng, Rongrong Ji, Xiaoshuai Sun, Yongjian Wu, Feiyue Huang, and Yanhua Yang. Centralized ranking loss with weakly supervised localization for fine-grained object retrieval. In Jérôme Lang, editor, IJCAI, pages 1226 1233. ijcai.org. 2018.
[53] Xiawu Zheng, Rongrong Ji, Xiaoshuai Sun, Baochang Zhang, Yongjian Wu, and Feiyue Huang. Towards optimal fine grained retrieval via decorrelated centralized loss normalize-scale layer. In AAAI, pages 9291-9298. AAAI Press, 2019.
14


Reviews:
[{"review_id": "wli35xPLrW", "reviewer": "Reviewer_C9Cw", "summary": "This paper works on an open-set fine-grain retrieval task. To align novel categories and the basic categories into a unified space, this paper proposes a Visual Attribute Parameterization Network (VAPNet) to learn visual attributes from known categories and parameterize them into the retrieval model. Extensive experiments on open-set fine-grained retrieval datasets validate the superior performance of our VAPNet over existing solutions.", "questions": "See weaknesses.", "limitations": "No limitations discussion is provided in this paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation of this paper is sufficient and reasonable. Adopting a series of visual attributes to represent an image makes it easier to embed unknown categories and known categories into a unified space. \n\n- The experimental results show this paper has achieved SOTA performance across three datasets.\n", "weaknesses": "- The visual attributes seem to be a series of local discriminative patch-level features by contrastive learning. It is a very implicit manner to extract attributes. With the development of large language models (LLM), it is possible to represent visual attributes as a structural language description through LLM-driven multi-modal model. Please discuss it.\n\n- Additional attributes extraction model may bring additional computation. A detailed computation and parameters number discussion should be included.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for taking the time to read our paper and thanks for your valuable feedback. \n\n**W1** : Thank you for your insightful comments.  As highlighted by the reviewers, recent multimodal large models, such as CLIP (Radford et al., ICML 2021) and BLIP2 (Li et al., Arxiv 2023), have shown success in learning the mapping between visual content and language concepts by training on visual and language tasks together.\n\nHowever, directly representing the visual attributes extracted by VAPNet as structured language descriptions poses a challenge. For example, one of the main difficulties arises from the fact that VAPNet and the text encoder of CLIP operate in different representation spaces. This semantic gap between our VAPNet and CLIP model makes it challenging to determine the most suitable language description for the visual attributes based on similarity between attribute and text embeddings. Similarly, BLIP2 encounters a comparable challenge as CLIP. Due to the absence of a semantic association between our VAPNet and the large language model in BLIP2, we are unable to directly input the attribute information into the large language model to generate the language description of visual attributes.\n\nOne possible solution is to feed the patches used for visual attribute extraction into the image encoder of CLIP or BLIP2. This approach can help mitigate the modality gap problem. However, it is important to note that CLIP and BLIP2 are trained on image-text pairs that describe general categories rather than fine-grained attributes. This raises challenges in representing fine-grained visual attributes and may lead to ambiguities in language descriptions.\n\nWe appreciate the opportunity to discuss this forward-thinking question and eagerly look forward to further discussions. We welcome any guidance or input to improve our response.\n\n**W2**: Thank you for raising this important concern. We carefully analyze the computation and parameter numbers for our proposed VAPNet method and will include a detailed discussion in our revised manuscript.\n\nAs shown in the table below, we compare the parameters, retrieval embedding extraction times, and recall@1 performance of our VAPNet method with a baseline model.\n\nMethod | Parameters | Time | Recall@1 \n----------|---------------|--------|----------- \nBaseline| 23.50M | 21.7ms | 69.5%\nOur VAPNet| 24.55M|21.7ms | 76.2%\n\nFrom the comparison, we can observe that our VAPNet method achieves a recall@1 performance improvement of 6.7% while only adding an additional 1.05M parameters compared to the baseline model. It is important to note that the increase in parameters is relatively small in proportion to the overall model size. Additionally, during testing, the retrieval embedding extraction time remains the same as that of the baseline model, as the additional attribution exploration modules (AAM and APM) are only used during training.\n\nBased on these results, we believe that the increase in computation and model complexity introduced by our additional attribute exploration model is minimal and acceptable. We will add this discussion to our revised manuscript to provide a comprehensive analysis of the computation and parameter numbers.\n\nWe sincerely appreciate the time and effort you have invested in carefully reviewing our paper. We eagerly await your response and value your insights and feedback.\n\n"}, {"review_id": "9Io9ySycKG", "reviewer": "Reviewer_CR86", "summary": "The paper addresses the problem of fine-grained image retrieval, where the model must identify subtle object attributes in order to distinguish between visually similar classes.\nTo achieve this, the paper proposes a Visual Attribute Parameterization Network to localize object attributes in images to enhance performance.\nSpecifically, the paper designs Attribute Exploration Module that extracts image features into local patches and projects them into attribute space. \nMoreover, an Attribute Parameterization Module is introduced to iteratively refine visual features for fine-grained retrieval.\nThe paper conducts experiments on three datasets of CUB, Stanford Cars, and FGVC Aircraft for retrieval tasks.", "questions": "Please refer to the weakness section", "limitations": "Sufficiently addressed", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ The direction of fine-grained image retrieval without attribute annotation is interesting with impactful real-world applications.\n+ The idea of detecting attributes by using local image patches is sensible and has been demonstrated by prior works to be effective for fine-grained classification [6,11]\n+ The paper shows promising retrieval improvements on CUB and Cars datasets.", "weaknesses": "+ The claim that \"we are the first to transform the retrieval model trained by image-level supervisions from category semantic prediction into attribute modeling\" is not well supported. To be specific, many works approach the problem of fine-grained recognition without attribute annotations [A, B] which also leverage the ideas of attribute localization just like the proposed method. Thus, the main difference between the proposed method and SOTA is simply the retrieval tasks instead of classification tasks, which lack novelty.\n+ The reviewer is not yet convinced by the significance of the experimental results, as the paper lacks strong comparisons with appropriate baselines such as [A, B] that are specifically designed to capture fine-grained attributes without annotation. Thus, it is unclear to the reviewer how effective the proposed method is compared to SOTA.\n+ The experiment datasets are small in scale and might not be challenging enough for grained classification. Specifically, CUB, CAR, and Aircraft datasets might not have a diverse number of attributes per class. Thus, the reviewer has some doubts about the effectiveness of the proposed method. It would be more convincing if experiments on recent fine-grained datasets such as DeepFashion [19] or iNaturalist [C].\n\n[A] Lin et al., Bilinear CNN Models for Fine-Grained Visual Recognition\n[B] Ding et al., Selective Sparse Sampling for Fine-Grained Image Recognition\n[C] Grant Van Horn et al., The iNaturalist Species Classification and Detection Dataset", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive comments as well as constructive suggestions. Below, we discuss each of the reviewer's concerns, and explain how we plan to address them in the revised version of our manuscript.\n\n**W1**:  Thank you for raising this point and providing references to related works. We appreciate your feedback and apologize for any confusion caused by our previous claim.\n\nAs you correctly pointed out, it's true that there have been previous works that approach fine-grained recognition without attribute annotations and leverage ideas of attribute localization. However, we believe that our claim of being the first to transform the retrieval model trained by image-level supervisions from category semantic prediction into attribute modeling for identifying unknown categories is still well-supported. The key distinction lies in the open-set fine-grained retrieval task, which poses the challenge of **retrieving unseen subcategories using knowledge learned from seen subcategories**. This is different from the closed-set learning setting typically used in fine-grained recognition. Therefore, there are few studies that explore visual attributes from seen classes without attribute annotations and utilize these attributes to identify unknown classes.\n\nSpecifically, works like Bilinear CNN [A] and S3N [B] focus on capturing discriminative details and identifying seen subcategories. However, when directly applying these fine-grained recognition methods to the open-set fine-grained retrieval task, the models tend to **get stuck in the discriminative knowledge of seen subcategories** and may struggle to handle unseen subcategories effectively.\n\nIn VAPNet, we aim to address this challenge by transforming the retrieval model trained by image-level supervision into attribute modeling. By dissecting various fine-grained object semantics and capturing numerous attributes from seen categories, we aim to improve performance in recognizing unseen subcategories.\n\nWhile prior fine-grained recognition studies have also extracted visual attributes without additional attribute annotations, their acquired attributes are specific to seen categories and may not be useful for recognizing unseen subcategories. In contrast, our VAPNet method aims to capture attributes from seen data that aid in recognizing unseen subcategories.\n\nWe appreciate your feedback and will make the necessary adjustments to our claim to better reflect our contribution in the revised version of the manuscript. Thank you for bringing this to our attention, and we apologize for any confusion caused.\n \n**W2**: Thank you for your feedback and raising the concern about the significance of our experimental results. Although Bilinear CNN [A] and S3N [B] were originally devised for recognizing seen subcategories, using them directly in retrieval tasks for unseen classes may result in poor performance. Concretely, to provide a clearer comparison, we have included the recall@k results for Bilinear CNN and S3N, as well as our VAPNet method, in the table below:\n\nMethod | Recall@1 | Recall@2| Recall@4 | Recall@8 \n----------|----------|----------|----------|----------\nBilinear CNN | 67.4% | 78.7% | 86.3% | 91.2%\nS3N | 65.4%| 77.2%|85.6% | 90.7%\nOur VAPNet | 76.2% | 84.6%| 90.1% | 94.0% \n\nFrom the results, it is evident that using Bilinear CNN and S3N directly in the retrieval tasks for unseen subcategories leads to lower recall@k compared to our VAPNet method. This highlights the challenge of using feature extractors trained in closed-set scenarios with classification supervisions to detect distinguishing variations from unseen subcategories, which ultimately affects the retrieval performance.\n\nIn contrast, our VAPNet method focuses on acquiring visual attributes instead of relying solely on discriminative cues. This enables us to better comprehend the unknown categories and accurately represent their distinguishing variations, leading to a significant improvement.\n\n**W3**: Thank you for your valuable suggestions. We understand your doubts and agree that conducting experiments on recent fine-grained datasets, such as DeepFashion [19] or iNaturalist [C], would provide a more convincing evaluation.\n\nIn response to your suggestion, we have performed an additional experiment on the large-scale DeepFashion dataset to further validate the effectiveness of our proposed VAPNet. The DeepFashion dataset offers an open-set retrieval dataset, where the 3,997 classes are split into training and the remaining 3,985 classes are used for testing. However, due to time limitations, we will consider conducting experiments on iNaturalist in future work to further verify the effectiveness of VAPNet.\n\nTo provide a clearer comparison, we have included the recall@k results for our VAPNet method, as well as the state-of-the-art methods CEP [4] and PNCA [29], in the table below:\n\nMethod | Recall@1 | Recall@10| Recall@20 | Recall@30|Recall@40\n----------|----------|----------|----------|----------|----------\nCEP [4] ECCV20 | 90.6% | 98.0% | 98.6% | 98.9% | 99.1% \nPNCA [29] ECCV20| 90.9% | 98.2% | 98.9% | 99.1% | 99.4%\nOur VAPNet | 93.9% | 98.7% | 99.1% | 99.4% | 99.6% \n\nFrom the results, it is evident that our VAPNet method achieves superior performance compared to other state-of-the-art (SOTA) methods, CEP and PNCA, on the large-scale DeepFashion dataset. By leveraging the visual attributes learned from known instances to identify category-specific discrepancies, our VAPNet demonstrates impressive generalization capabilities.\n\nWe greatly appreciate your professional feedback, and we will incorporate these additional experiments in the revised version of the paper to further demonstrate the effectiveness and generalization ability of our proposed VAPNet method. Thank you for your valuable input.\n\n"}, {"review_id": "uKhH631mPa", "reviewer": "Reviewer_yK4j", "summary": "Firstly, they introduce a novel approach to address the problem of open-set fine-grained retrieval settings. They transform the retrieval model, which is typically trained using image-level supervisions for category semantic prediction, into attribute modeling. This transformation helps alleviate the challenges posed by open-set fine-grained retrieval.\n\nSecondly, they propose a Visual Attribution Parameterization Network (VAPNet) that distills visual attributes from various semantics observed in seen fine-grained objects. These attributes are then transcribed into parameters within the retrieval model. This parameterization allows for the precise representation of unknown categories based on their transformed parameters derived from visual attributes.\n\nThe authors conduct extensive experiments to evaluate their method's performance on open-set fine-grained retrieval tasks. The results demonstrate that their proposed approach, VAPNet, brings significant benefits. It achieves an average accuracy gain of 8.6% compared to the recent state-of-the-art work on three open-set fine-grained retrieval benchmarks.", "questions": "Please reply the concerns in the weakness section during rebuttal.", "limitations": "Limitations are not discussed in this paper.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The novelty presented in this paper revolves around the proposed Visual Attribution Parameterization Network (VAPNet) and its application in open-set fine-grained retrieval tasks.\n\nThe authors introduce VAPNet as a novel approach to handle unknown categories in such tasks. VAPNet focuses on distilling visual attributes from semantic clues observed in known instances. These visual attributes serve as supervisory signals to fine-tune the retrieval model. By doing so, the authors transform the retrieval model, originally trained with image-level supervisions for category semantic extraction, into attribute modeling. This transformation enables precise representation of unknown categories based on parameters supervised by visual attributes. As a result, VAPNet effectively addresses the challenges associated with encountering instances from unseen novel categories.\n\nFurthermore, the authors highlight the simplicity and flexibility of the overall retrieval pipeline enabled by VAPNet. They emphasize that the proposed method surpasses state-of-the-art approaches by a significant margin, demonstrating the effectiveness of attribute modeling when dealing with unknown categories in fine-grained retrieval tasks.", "weaknesses": "1.\tThe framework generally follows some contrastive learning works. The input pair – image and patch, can be regarded as two views in the contrastive learning, which has been used in RegionCL[1]. Additionally, using KL divergence as objectives for distillation has been proposed in RepDistiller [2].\n\n2.\tThe authors claim to solve the task of open-set fine-grained retrieval. However, the experiments on open-set image retrieval are missing in the paper.\n\n3.\tWhy most of SOTA work reported in Table 2 are proposed for deep metric learning instead of the image retrieval? Additionally, why some benchmark datasets for image retrieval, such as CIFAR, COCO, Oxford, are not used here?\n\n4.\tThere are lots of confusing concepts in the manuscript. For example, what does “c” and “k” in “… weight matrix W_L ∈ R^c x k …” in Line 179 and Line 194 refer to? What does “sampler” in L189 refer to? What does “L_s” in Eq. (8) refer to?\n\n5.\tThere are some typo errors. For example, “A” in Eq. (2) should be in bold.\n\n[1] Xu, Yufei, et al. \"Regioncl: exploring contrastive region pairs for self-supervised representation learning.\" ECCV 2022.\n\n[2] Yonglong Tian, et al. Contrastive Representation Distillation, ICLR 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments! Below, we discuss each of the reviewer's concerns, and explain how we plan to address them in the revised version of our manuscript.\n\n**W1**: Thank you for your insightful comments. I appreciate your corrections and would like to provide further clarification regarding the similarities and differences between our VAPNet framework and existing contrastive learning and knowledge distillation approaches.\n\nIndeed, our VAPNet framework shares some similarities with contrastive learning schemes, such as RegionCL [1]. The input pairs, consisting of an image and a patch, can be seen as two views in contrastive learning. However, the goals of our VAPNet differ from those of self-supervised learning schemes like RegionCL. While self-supervised learning aims to learn consistent probability distributions by attracting positive pairs and repelling negative pairs, our VAPNet utilizes local and global views to generate attribute pairs as supervisory signals. These attribute pairs are then used to transform the retrieval model from category prediction to attribute modeling.\n\nSimilarly, you correctly mentioned that KL divergence is commonly used in knowledge distillation, as seen in techniques like RepDistiller [2]. However, the purpose of knowledge distillation is to make the output of a student network mimic that of a teacher network by constraining their probability distributions using KL divergence. In contrast, in our VAPNet, we employ KL divergence to ensure the distribution consistency between attribute pairs generated by local and global views within a single network. This allows us to iteratively refine the visual attributes during training and simultaneously use them as supervisory signals to fine-tune the retrieval models, resulting in effective attribute parameterization.\n\nAlthough input pairs and KL divergence have been widely used in various contexts, our VAPNet effectively combines them within a joint network and leverages their collaboration to transform the retrieval model from category prediction to attribute modeling. This unique integration makes our VAPNet specifically designed for open-set tasks, enabling the effective utilization of knowledge learned from seen categories to identify unseen categories. These characteristics offer valuable insights and practical applications for real-world scenarios.\n\nThank you for your thoughtful feedback, and we will make sure to incorporate these additional explanations and clarifications in the revised version of our paper.\n\n**W2**: Thank you for bringing up this concern. We have indeed conducted experiments to evaluate the generalization ability of our VAPNet in open-set scenarios. As mentioned in Section 4.1, we have selected the CUB-200-2011, Stanford Cars, and FGVC Aircraft datasets, which are commonly used in open-set fine-grained retrieval tasks. These datasets are split into seen categories for training and unseen categories for evaluation, simulating real-world scenarios where new classes can emerge.\n\nIn the experiments section, we report all the results under the scenario where our VAPNet is trained on the seen classes but tested on the unseen classes. This setup allows us to assess the performance of our method in handling unknown or novel query images, which is a key aspect of open-set retrieval.\n\n**W3**: Thank you for your question and feedback. In Table 2, we present the results for retrieving unseen classes. Metric learning is designed to quantify the similarity between two images independent of their class information. As a result, metric learning can be trained on seen classes and utilized for retrieving unknown classes. However, traditional image retrieval methods often suffer from being trapped in the discriminative knowledge of seen classes and may struggle to handle unseen subcategories effectively. Therefore, most of the state-of-the-art works reported in Table 2 are proposed for deep metric learning rather than image retrieval.\n\nRegarding the choice of benchmark datasets, general image retrieval datasets such as CIFAR, COCO, and Oxford cover a wide range of object categories. and their settings usually share the same classes in both training and testing. However, in our experiments, we specifically focused on open-set settings to simulate real-world scenarios where new classes can emerge. Therefore, we choose fine-grained datasets containing visually similar classes, which can be easily split into seen categories for training and unseen categories for evaluation. \n\n**W4**: Thanks for your valuable corrections. In $W_L$, \"c\" denotes the channel number of input features, while \"k\" denotes the dimension of visual attributes. \"Sampler\" refers to the RoI align operations used to crop the local features from the global features based on the coordinates of local views. We apologize for mistakenly writing the auxiliary constraint \"$L_a$\" instead of $L_s$\". In the revised version, we will provide a clear explanation of these potentially confusing concepts and rectify any errors.\n\n**W5**: Thank you for your valuable feedback and for pointing out the typo errors in our manuscript. We appreciate your careful review and have made the necessary corrections, including ensuring that the \"A\" in Eq. (2) is in bold. \n\n"}, {"review_id": "dWTcoXSTEw", "reviewer": "Reviewer_uknX", "summary": "This paper proposes VAPNet to learn visual attributes from known fine-grained categories and parameterize them into final open-set retrieval. To learn visual attributes without attribute annotation, VAPNet explicitly attains some semantics with rich details via making use of local image patches and distills the visual attributes from these discovered semantics. Then, it incorporates the online refinement of these visual attributes into the training process to iteratively improve them and simultaneously regards these attributes as supervisory signals to tune the retrieval models. Experimental results on on open-set fine-grained retrieval benchmarks demonstrate improved performance compared to existing methods.\nHowever, I have some concerns about the motivation and the methodology employed. Please refer to the detailed comments below.", "questions": "- The corresponding relationship between local features and different patch of global features in line 191 is not clearly explained.\n- The global view in Figure 3(top row(b)(d)(f)) shows that the model only pays attention to some local parts, which is quite different from the VAPNet shown in Figure 4(top row(c)(f)), which focuses on more local parts.\n- Does the randomly cropped size have an effect on the model, and what is the relationship between the cropped size, the Patch number M, and visual attribute dimension k.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- It is commendable to introduce the visual attributes to open-set fine-grained retrieval.\n- This paper is well-written and easy to follow.", "weaknesses": "- The author do not explain the necessity of introducing visual attributes into open-set fine-grained retrieval.\n- In line 149, the attribute exploration module attain semantic clues of input object via randomly cropping local patches from the input image, but is hard to guarantee the complete coverage of all discriminative local regions of the input image by random cropping.\n- In addition to local attributes, visual attributes can also contain global attributes, such as shape. The design of attribute exploration module only focuses on local visual attributes while ignoring global visual attributes.\n- The artificially defined attributes correspond to different visual features and can be decoupled from each other. While the proposed method does not further constrain the relationship between visual attributes(i.e. Eq.(6) the attribute parameterization constraint Lc), and the extracted visual attributes cannot be guaranteed to have the characteristics of this aspect. It is more like to enhance the local representation ability of the model.\n- There are some errors in the details of the paper, such as Eq.(6) in line 322, which actually corresponds to Eq.(5).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1**: Thank you for your valuable feedback. In open-set fine-grained retrieval, the model is required to learn embedding from the seen classes and then to be capable of utilizing the learned knowledge to distinguish the unseen classes. Existing approaches commonly employ class supervisory signals to guide the model in capturing discriminative details that are useful for identifying the seen classes. However, these approaches tend to focus solely on capturing discriminative concepts of known categories from unknown instances, which can make it challenging to identify unknown categories.\n\nA noteworthy observation in the context of fine-grained objects is that visually similar objects from different subcategories often share common visual attributes. Therefore, an unknown instance can be described comprehensively using a range of visual attributes, which can be discovered across multiple known categories. Leveraging this insight, we can represent the unknown classes using visual attributes that have been discovered from the seen instances. By combining these attributes, we can effectively capture the subtle differences between the unseen classes, thus mitigating the challenges associated with open-set scenarios.\n\n**W2**: Thanks for your nice comment. Random cropping cannot always guarantee the completeness of all the parts which are smaller than the size of the patch. Although there could exist some parts which are smaller than the patch size, those still have chances of getting split. However, this should not be a concern for model training, since we adopt random cropping which is a standard data augmentation strategy before the random cropping. This strategy leads to the result that patches are different compared with those of previous iterations. Small discriminative parts, which are split at this iteration due to the random cropping, will not be always split in other iterations. This variability in patches brings an additional advantage when dealing with occluded visual attributes. It improves the generalization ability of our model, allowing it to better handle occlusions and enhancing its overall performance.\n\n**W3**: VAPNet successfully captures global attributes with a high probability. We achieve this by integrating large-scale patches that cover 1/4 of the original image within our attribute exploration module. When small objects lie within these patches, VAPNet can directly capture their global attributes. However, it is important to note that these attributes serve as supervisory signals rather than object descriptors. Consequently, VAPNet transforms these global attributes into parameters. This allows us to effectively capture the global attributes of other large-scale objects using the parameters fine-tuned by global attributes produced those small objects.\n\n**W4**: This is due to the lack of attribute annotations, which makes it challenging to establish explicit relationships between attributes and guarantee specific characteristics for attributes.\n\nHowever, these limitations do not hinder the exploration of visual attributes for describing unknown classes. VAPNet is designed to capture diverse visual attributes and convert them into parameters for retrieval models. When provided with a local view, it can translate it into a set of attributes. Although this attribute set is not directly used for object description, it serves as supervisory signals to fine-tune the retrieval model's parameters. This process allows the retrieval model to automatically disentangle and store the attributes in its parameters. Thus, the model can produce precise attributes to describe diverse visual content and effectively capture the discriminative differences among unknown classes.\n\nAs you correctly pointed out, VAPNet significantly enhances the local representation of the model. With this enhancement, VAPNet can accurately translate an input object into a set of visual attributes, leveraging its exceptional local perception. \n\n**W5**: Thanks for your valuable corrections. We will rectify this error in the revised version and meticulously refine our paper.\n\n**Q1**: Each local view includes its associated position information $(V_w, V_h, V_x, V_y)$, where $(V_w, V_h)$ represents the width and height of the local view, and $(V_x, V_y)$ denotes the center location of the local view. To extract local features from global features, we leverage the coordinate information of local views and employ the RoI Align operation proposed in Mask RCNN (He et al., ICCV 2017). \n\n**Q2**: The global view depicted in Fig. 3 is generated by the baseline network, rather than VAPNet. In contrast to Fig. 3, Fig. 4 is generated by VAPNet. The baseline network, which is supervised by class signals, tends to selectively learn partial regions that are easier to reduce the current training empirical risk for the seen categories. Consequently, it focuses more on specific local regions rather than capturing comprehensive details and information from all sides. \n\n**Q3**: The size of the randomly cropped patches does indeed have an impact on the performance. As the size of discriminative details may vary, it is important for the patch size to be flexible. Thus, we define four scales of patches to cover a wide range of detail scale. However, it is challenging to ensure that all scales of visual content are adequately covered solely through randomly cropped sizes.\n\nIt is important to note that the cropped size, the number of patches (M), and the dimension of visual attributes (k) are independent of each other. Specifically, VAPNet aims to collect diverse visual attributes and transform them into model parameters, rather than utilizing them as the final representation. Hence, the number of patches is not a critical factor. Additionally, the size of patches only determines the richness of visual semantics, while the dimension of visual attributes indicates the complexity of these attributes. Therefore, they are not unrelated."}, {"review_id": "GPCBeoYku7", "reviewer": "Reviewer_Xapw", "summary": "The work proposes a new approach for open-set fine-grained retrieval. Their main contribution is the new state-of-the-art method termed VAPNet, which exploits objects’ attributes to discriminate between known and unknown objects. Due to the absence of attribute annotations, their training loss includes a pipeline to extract and refine attributes in an unsupervised fashion.", "questions": "please refer to Weakness section", "limitations": "1. No limitation was reported. It may be compute intensive to extract multiple patches from the same image or for the overall training pipeline. Moreover, it could be that the evaluation process is slower due to the additional components introduced. It may be important to clearly explain any limitation to the work.\n2. Same for broader impact. It should be stated\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors’ idea to exploit visual attributes to discriminate between known and unknown classes is novel for the task.\n2. The approach to learn such attributes in an unsupervised fashion and in the open-set scenario is quite interesting.\n3. The quality of the manuscript is good.\n4. Their evaluation protocol makes sense and their method achieves the best results across three datasets when compared with related works. ", "weaknesses": "1. Section 3 is hard to follow, mainly the architectural choices. There are many components in the method. The authors could refer to Figure (2) whenever introducing some components to guide the reader’s understanding.\n2. The differences in the pipeline between training and evaluation are not stated explicitly. This could help in understanding the overall method better.\n3. Figure (2) could be improved to better convey the method (the current image is too crowded). From the text and the image it may be hard to understand whether the network represented in the retrieval module and the attribute exploration modules are the same or not. The retrieval embeddings returned by the pooling operations seems superfluous in the image. It may be best to either disentangle the components, or increase abstraction of some components. Moreover, the caption is too short and conveys no information. Ideally, the figure should be able to convey all that is required to understand the method without additional context.\n4. This is a minor suggestion. Some section names are misleading or could be improved. For instance, many titles contain the word “attribute” to the point that its presence means little. Moreover, the title for Section 3.4 is misleading and it may be best to rename it.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments! Below, we discuss each of the reviewer's concerns, and explain how we plan to address them in the revised version of our manuscript.\n\n**W1**: Thank you for your valuable feedback. \nWe appreciate your suggestion to refer to Figure (2) whenever introducing components in order to guide the reader's understanding. We agree that this will enhance the coherence of our narrative and make it easier for readers to follow along. We will make sure to incorporate these references in the revised version of our paper.\n\nThank you once again for your insightful comments, which will help us improve the clarity and readability of our work.\n\n**W2**: Thank you for your insightful feedback. In the revised version, we will provide a more detailed explanation of the differences in the pipeline between training and evaluation. Specifically, our VAPNet framework comprises a backbone network, an attribute exploration module, and an attribute parameterization module. During the training phase, all these components are utilized to explore and parameterize visual attributes. However, during the evaluation phase, we only employ the backbone network to extract retrieval embeddings. By clearly highlighting these differences, we hope to enhance the overall understanding of our proposed method.\n\n**W3**: We sincerely thank the reviewer for the insightful feedback on Figure 2. Your suggestions are valuable, and we recognize the need for refining the figure to better represent our method. Aligning with your advice, we will present an improved version of Figure 2, which will hopefully address these concerns more effectively.\n\n**W4**: Thank you for bringing up the issue of section names. To enhance the clarity of our method description, we have decided to remove sub-titles such as attribute exploration, attribute sampling, and attribute parameterization constraint from the method section. This modification allows for a more streamlined and concise presentation of our approach. Additionally, we agree that the title for Section 3.4 is misleading and will rename it as \"Loss Function\" to better reflect its content. Thank you again for your valuable suggestion.\n\n**L1**: Thank you for your insightful comments. We appreciate your suggestions regarding the limitations of our work. As you correctly pointed out, the extraction of multiple patches during training can indeed introduce additional computational overhead. This approach is necessary to capture diverse visual attributes, which are then used as supervisory signals for fine-tuning the backbone network. While this does require computational resources during training, we believe that the benefits outweigh the costs, especially considering the expense of labeling attribute annotations.\n\nDuring the evaluation phase, it is important to note that our VAPNet does not utilize the designed components or extract multiple patches. As a result, the evaluation process is not slower than the baseline. To provide a comprehensive analysis of complexity, we have reviewed previous test logs and compared retrieval embedding extraction times and model parameters. The results are summarized in the table below:\n\nMethod | Parameters | Time | Recall@1 \n----------|---------------|--------|----------- \nBaseline| 23.50M | 21.7ms | 69.5%\nOur VAPNet| 24.55M|21.7ms | 76.2%\n\nAs observed, our proposed VAPNet with AAM and APM modules achieves a performance improvement of 6.7% while only adding an additional 1.05M parameters compared to the baseline. Furthermore, since the AAM and APM modules are only utilized during training, the retrieval embedding extraction time during testing remains the same as that of the baseline. Therefore, the increase in algorithm complexity is minimal and is considered acceptable.\n\n**L2**: We appreciate your feedback regarding the broader impact of our work. \nBy introducing VAPNet, we aim to extract visual attributes from seen classes without relying on attribute annotations to differentiate unseen classes. This innovation has the potential to greatly impact open-domain tasks. \nIn particular, annotating a large number of attributes for unseen categories in open-domain tasks can be a costly and time-consuming endeavor. By enabling the model to automatically capture knowledge about unseen classes, our approach reduces the reliance on attribute annotations, resulting in decreased manual labeling costs.\nFurthermore, our approach exhibits improved adaptability to data from domains resembling the training set, such as natural images or medical images. This heightened adaptability contributes to stronger generalization capabilities, allowing the model to perform well in real-world scenarios.\nUltimately, our solution has the potential to propel the advancement of open-domain tasks and facilitate their practical applications. Thank you for bringing this to our attention.\n"}]

Author rebuttal:

Meta-review: After the rebuttal stage, there are two weak accept, two borderline accept, and one borderline reject scores. The reviewers praise the task and find the method interesting and effective. None of the concerns/questions asked in the reviews are critical; many are clarification questions. The borderline reject reviewer had the lowest confidence, and did not respond to the rebuttal, but their questions (mostly clarification, e.g. about the task) seem well addressed. The authors should improve the presentation as discussed during the rebuttal phase.
Decision: Accept (poster)