Title: Learning to Parameterize Visual Attributes for Open-set Fine-grained Retrieval
Venue: NeurIPS 2023
Accepted: yes
Keywords: Open-set Fine-grained Retrieval

Visual Attribute

Unknown Categories
Abstract: Open-set fine-grained retrieval is an emerging challenging task that allows to retrieve unknown categories beyond the training set. 
The best solution for handling unknown categories is to represent them using a set of visual attributes learnt from known categories, as widely used in zero-shot learning. Though important, attribute modeling usually requires significant manual annotations and thus is labor-intensive. Therefore, it is worth to investigate how to transform retrieval models trained by image-level supervision from category semantic extraction to attribute modeling. To this end, we propose a novel Visual Attribute Parameterization Network (VAPNet) to learn visual attributes from known categories and parameterize them into the retrieval model, without the involvement of any attribute annotations.
In this way, VAPNet could utilize its parameters to parse a set of visual attributes from unknown categories and precisely represent them.
Technically, VAPNet explicitly attains some semantics with rich details via making use of local image patches and distills the visual attributes from these discovered semantics. Additionally, it integrates the online refinement of these visual attributes into the training process to iteratively enhance their quality. Simultaneously, VAPNet treats these attributes as supervisory signals to tune the retrieval models, thereby achieving attribute parameterization. Extensive experiments on open-set fine-grained retrieval datasets validate the superior performance of our VAPNet over existing solutions.

Learning to Parameterize Visual Attributes for Open-set Fine-grained Retrieval

Abstract

Open-set fine-grained retrieval is an emerging challenging task that allows to retrieve unknown categories beyond the training set. The best solution for handling unknown categories is to represent them using a set of visual attributes learnt from known categories, as widely used in zero-shot learning. Though important, attribute modeling usually requires significant manual annotations and thus is labor-intensive. Therefore, it is worth to investigate how to transform retrieval models trained by image-level supervision from category semantic extraction to attribute modeling. To this end, we propose a novel Visual Attribute Parameterization Network (VAPNet) to learn visual attributes from known categories and parameterize them into the retrieval model, without the involvement of any attribute annotations. In this way, VAPNet could utilize its parameters to parse a set of visual attributes from unknown categories and precisely represent them. Technically, VAPNet explicitly attains some semantics with rich details via making use of local image patches and distills the visual attributes from these discovered semantics. Additionally, it integrates the online refinement of these visual attributes into the training process to iteratively enhance their quality. Simultaneously, V APNet treats these attributes as supervisory signals to tune the retrieval models, thereby achieving attribute parameterization. Extensive experiments on open-set fine-grained retrieval datasets validate the superior performance of our V APNet over existing solutions.

1 Introduction

Fine-grained image retrieval attempts to build a well-generalized embedding space where the visual discrepancies among categories are clearly reflected. It plays a vital role in numerous vision applications from fashion industry, e.g. , retrieval of different types of shoe or clothes [22; 1], to environmental conservation, e.g. , retrieval endangered species [6; 38; 36; 32]. However, real-world applications probably face the input of unknown categories, and the model will treat them as known ones. As a result, the retrieval performance decays, which is unbearable in real-world applications. Open-set fine-grained retrieval is thus proposed to conduct training on known categories but retrieve unknown ones during evaluation.

Facing the unknown inputs of novel categories, an intuitive way to identify them is to capture the discriminative discrepancies of unknown categories from these inputs for identifying them. However, most existing works [19; 24; 28; 20] still focus on discriminative concepts of known categories and only capture them from unknown instances, consequently making it hard to precisely identify unknown categories. Interestingly, zero-shot learning [11; 18; 47] has proven that an unknown instance can be described integrally using a variety of visual attributes, and these attributes can be

discovered on multiple known categories. For example, the unknown birds in Fig. 1 can be represented using visual attributes discovered from seen instances, and the combination of these attributes can clearly reflect their discrepancies, thus alleviating the problem behind open-set settings. Though important, attribute modeling usually requires significant manual annotations and thus is labor-intensive. When attribute annotations are unavailable, how to transform retrieval models trained by image-level supervision from category semantic prediction to attribute modeling is worthy of investigation.

In this paper, we present a Visual Attribute Parameterization Network, termed as VAPNet, aiming to distill visual attributes from various semantics presented on seen fine-grained objects, and utilize these attributes to tune the retrieval model. Consequently, VAPNet describes the appearance of the

Unknown

Categories

known Categories

⋯

⋯

⋯

⋯

Inputs

𝐶𝐴

𝐶𝐵

𝐶𝑁𝑁

Visual Attribute

Parameters

𝜃(𝐶𝑟𝑜𝑤𝑛_𝐻𝑒𝑎𝑑)

𝜃(𝑆𝑝𝑜𝑡_𝑊𝑖𝑛𝑔𝑠)

𝜃(𝑌𝑒𝑙𝑙𝑜𝑤_𝑇𝑎𝑖𝑙)

𝜃(𝐵𝑙𝑎𝑐𝑘_𝐶𝑙𝑎𝑤𝑠)

𝜃(𝑊ℎ𝑖𝑡𝑒_𝑊𝑖𝑛𝑔)

𝜃(𝑌𝑒𝑙𝑙𝑜𝑤_𝐵𝑒𝑙𝑙𝑦)

𝜃(𝐵𝑙𝑢𝑒_𝑀𝑜𝑢𝑡ℎ)

⋯

𝑆𝑖𝑚: 0.08

: Discriminative Attributes

: Common Attributes

Inputs

𝐶𝐴

𝐶𝐵

Embeddings

Embeddings

input instances of novel categories based on its parameters tuned by visual attributes, thus transforming the retrieval model from category semantic prediction to attribute modeling. Notably, due to lacking of attribute annotations, the attributes derived from V APNet will be not restricted to pre-defined attributes like supervised-based attribute learning works [10; 49].

Technically, VAPNet needs to parse various semantics presented in fine-grained objects, which is a prerequisite for attribute modeling. However, a feature extractor trained by image-level labels tends to focus on a few primary semantic regions (e.g., bird's head) while ignoring other visual clues (e.g. bird's body). We empirically observe that vision models can discover these overlooked object regions with rich details when taking local image patches as input compared to the whole image. Therefore, VAPNet attains some rich semantics presented in objects via parsing multiple local views randomly cropped from the input image. After that, we further apply an encoder to project these discovered semantics to a set of visual attributes. Nevertheless, due to lacking the attribute annotations, these attributes usually include some noisy patterns. To handle this limitation, we incorporate the online refinement of these attributes into the training process to iteratively improve their quality and simultaneously regard these attributes as supervision signals to tune the retrieval model, thus achieving attribute parameterization. Specifically, we design another encoder with the same structure as the above one to produce another set of visual attributes from the global features, which are used to match the visual attributes inferred by local views for providing a rich supervisory signal. To avoid optimizing two encoders instead of the retrieval model, we design the counterparts of two encoders by accumulating their parameters of all previous iterations to make supervisory signals provided by attributes tune the retrieval model directly. In this way, the features outputted by the retrieval model can be iteratively improved and fed into the optimized encoders to provide more accurate visual attributes, which, in turn, better tunes the retrieval model for visual attribute modelling.

Contributions of this paper are summarized as below:

To the best of our knowledge, we are the first to transform the retrieval model trained by image-level supervisions from category semantic prediction into attribute modeling, thus alleviating the problem behind open-set fine-grained retrieval settings.

We propose a novel Visual Attribution Parameterization Network, which distills visual attributes from various semantics discovered on seen fine-grained objects, and transcribes these attributes into parameters within the retrieval model, thus representing unknown categories precisely based on its parameters transformed by visual attributes.

Extensive experiments show that open-set fine-grained retrieval task can benefit from the proposed method, and thus our VAPNet obtains significant gains of 8.6% average accuracy over recent state-of-the-art work [33] on three open-set fine-grained retrieval benchmarks.

2 Related Work

Open-set fine-grained retrieval. Existing open-set fine-grained retrieval works can be roughly divided into several groups. The first group, localization-based scheme , utilizes the supervision of category signals to learn discriminative embeddings [42; 52; 24; 40]. CRL [52] designs an attractive object feature extraction strategy to facilitate the retrieval task. Despite the inspiring achievement, the shortcoming of these works is that they only focus on individual samples while neglecting the inter-class and intra-class correlations between subcategories, thus reducing the retrieval performance. Therefore, the second group, metric-based scheme , is learning an embedding space where similar examples are attracted, and dissimilar examples are repelled [33; 41; 4; 15; 27; 51; 14; 50]. NIA [28] enforces unique translatability of samples from their respective class proxies to bring the distance of samples with the same subcategory closer. However, they still capture the discriminative details of known categories from unknown instances but neglect more details on undiscovered semantic regions, consequently impairing the retrieval performance.

Unlike the above works, FRPT [35] steers a frozen pre-trained model to perform the fine-grained retrieval task from the perspectives of sample prompting and feature adaptation. PLEor [34] could leverage pre-trained CLIP model to infer the discrepancies encompassing both pre-defined and unknown subcategories, and transfer them to the backbone network trained in the close-set scenarios. Nevertheless, it is worth noting that both of these approaches typically require more computational resources to optimize the retrieval models. This can potentially limit their practical applicability in real-world scenarios. To alleviate the problem behind open-set scenarios, we design VAPNet to explore and exploit visual attributes learnt from known instances instead of learning discriminative clues to anticipate open-set class data, improving retrieval performance in open-world scenarios accordingly.

Visual attributes. Attributes belong to intuitive properties of objects, which contain low-level semantics (e.g., color, texture and shape), high-level semantics (e.g., head, body and tail of objects), or even common sense (e.g., birds living on the tree) [7]. Utilizing visual attributes makes great progress on various vision tasks, including image search [17], fine-grained recognition [49; 37], scene understanding [25], and so on. Most of the previous works based on attribute learning [10; 17; 49] usually require significant manual attribute annotations and therefore is labor-intensive. Besides, the attributes learnt by these works are also restrained to pre-defined attribute labels, consequently ignoring some potentially vital information lying in visual semantics. To alleviate the aforementioned issues, recent works [43; 39] formulate an unsupervised learning strategy to project the learnt features into an attribute space. However, although the two works achieve superior performance on their corresponding vision tasks, they still are rooted in the close-set scenarios and thus make it hard to handle unknown instances. Therefore, we propose VAPNet to process more challenging scenarios, i.e. , open-set fine-grained retrieval tasks, by making full use of known data.

3 Methodology

The overall structure of V APNet is shown in Fig. 2. It is clear that our network is mainly organized by three modules: retrieval module, attribute exploration module and attribute parameterization module. The retrieval module could extract retrieval embeddings encompassing various attributes of input objects for retrieving visually similar objects. The attribute exploration module is designed to randomly extract visual attributes from known categories. In addition, the attribute parameterization module is responsible for improving visual attributes and utilizing them as supervisory signals to tune the retrieval model.

3.1 Retrieval Module

The retrieval module aims at extracting basic image representations using the backbone network and producing retrieval embeddings. Thereby, the backbone network can be regarded as the retrieval model. Formally, given an image X , let F ∈ R W × H × C be the C -dimensional with H × W feature

ℱ𝐶𝑁𝑁

𝑷𝒐𝒐𝒍𝒊𝒏𝒈

§𝟑. 𝟏

)

𝐸𝑅

ℱ𝐶𝑁𝑁

Retrieval Embeddings

𝐹𝐿

Attribute Exploration Module (

§𝟑. 𝟐

)

𝑅𝑒𝑢𝑠𝑒

Local Attribute Feature

Attribute

Attributes

Attribute Parameterization Module (

§𝟑. 𝟑

)

𝒯 𝐺

𝒯 𝐿

𝑎1

𝑎2

𝑎3

𝑎4

𝑎5

Parameter

Accumulation

𝑎1

𝑎2

𝑎3

𝑎4

𝑎5

𝑎1

𝑎2

𝑎3

𝑎4

𝑎5

𝑎1

𝑎2

𝑎3

𝑎4

𝑎5

Parameter Accumulation

Attributes

Projection

𝐹

Global Features

𝐹𝐺

Local Attribute

Feature

𝑋

❄

❄

❄

: No learnable

parameters

planes encoded by a backbone network F = F CNN ( X ) . Thus the most common way for retrieval is to embed the final feature F by using global average pooling operations (GAP), calculating mean values on the H × W feature plane and producing the final retrieval embeddings E R ∈ R C . It should be clarified that our V APNet does not introduce extra computation overhead during evaluation.

3.2 Attribute Exploration Module

Facing the unknown input of novel categories, VAPNet aims to explore all the attributes presented in known instances as much as possible and utilize them to understand unknown categories. A non-negligible problem is that a feature extractor trained by image-level labels tends to focus on a few primary semantic regions ( e.g. , bird's head) while ignoring other visual clues ( e.g . bird's body). Fortunately, a feature extractor could discover object regions with rich details when replacing the input image with its local patches, as verified in Fig. 3. This suggests a proper way to focus on some overlooked object regions by making use of local image patches. Therefore, an attribute exploration module is proposed to attain some semantic clues of an input object via randomly cropping local patches from the input image. These collected semantic clues could be translated into visual attributes describing pre-defined and unknown categories.

Input sampling. Given an input image X , we equally split it into n × n patches which have 3 ×⌊ H n ⌋ × ⌊ W n ⌋ dimensions. Here, the granularities of patches are controlled by the hyper-parameter n . In our experiments, n respectively equals to 2, 4, 8 and 16, and thus the number of patches is 340. We randomly sample M ( M ≪ 340 ) patches from the candidate set at each iteration to construct a set of different local views V = [ V 1 , V 2 , · · · , V M ] . These local views are resized to the same scale as inputs via using bilinear interpolation. The magnification operation could directly highlight the subtle yet discriminative details in the local views, further making the backbone network more sensitive to these subtle details.

Then, the local views V are passed through the backbone network F CNN :

where F L ∈ R M × C × H × W is the local attribute feature set.

It should be clarified that the input sampling step samples local patches randomly, so it can treat all object patches equally, regardless of their discriminative ability. However, it should not be bad news for open-set fine-grained retrieval tasks, since the more diverse the visual attributes are, the

better the understanding of the unknown categories is. Furthermore, although this module picks out a few patches at each iteration, it could effectively parse the holistic structures of objects based on the accumulation of selected patches by multiple iterations.

Attribute exploration. In real-world applications, especially for the fine-grained tasks with small inter-class variances, attribute annotations are expensive and therefore labor-intensive. Nevertheless, unsupervised learning [26] can capture regularities in data for the purpose of extracting useful knowledge or for restoring corrupted data. Therefore, many unsupervised works [26; 2] explicitly produce internal latent units or codes from feature representation. Inspired by this, we design a visual attribute exploration to project these visual semantics attained by local views into a latent space, i.e. , the attribute space:

where T L denotes the encoder with the weight matrix W L ∈ R c × k and bias vector b L ∈ R k to process the local features produced by local views, and g( · ) is the global average pooling operation. By the projection operation, we successfully transform the visual semantics into the visual attributes [ A i L ∈ R k | i = 1 , · · · , M] . In this way, these visual attributes could correspond to local semantic knowledge of fine-grained objects ( e.g. , red head, spotted wings, etc. ), so the group of certain attributes can clearly describe an unknown category and reflect its discriminative discrepancies.

3.3 Attribute Parameterization Module

Visual attributes produced by local views serve as supervisory signals to tune the retrieval model, further make it be transformed from category semantic extraction to attribute modelling. Nevertheless, due to lacking the attribute annotations, these visual attributes substantively include some noisy patterns. The retrieval model supervisied by these noisy attributes is harmful to parsing fine-grained objects. To this end, we propose an attribute parameterization module to incorporate the online refinement of these attributes into the training process to iteratively improve them and simultaneously regard these attributes as supervisory signals to tune the retrieval model. In this way, the retrieval model could capture visual attributes from input instances, thus achieving attribute parameterization.

Attribute sampling. To match visual attributes provided by local views, we need to process the final features F inferred by an input image X to produce another group of attributes, which are used to match these visual attributes provided by local views. Concretely, forwarding F into a sampler extracts the corresponding local attribute feature sets F G = [ F 1 G , F 2 G , · · · , F M G ] according to four coordinates of local views. Specifically, we utilize RoIAlign operation proposed in Mask-RCNN [8] to accurately extract the corresponding local features. Then, we project these local representation in another attribute space:

where T G represents an encoder with the weight matrix W G ∈ R c × k and bias vector b G ∈ R k to process the local features produced by the final features, and A i G ∈ R k denotes the i -th attribute in the attribute set A G ∈ R M × k .

With these visual attributes including A L and A G and local features containing F L and F G , the attribute pairs A and local feature pairs F P can be organized as:

Attribute parameterization constraint. This constraint is responsible for improving these visual attributes and utilizing them to tune the retrieval model. Specifically, as the local attribute features fed to each encoder come from global and local views, this encoder only distills visual attributes only from its corresponding view. Thus, given local attribute features, no matter which view they come from, if two encoders provide the same attribute, it means this feature can be regarded as from both two sources. In other words, the feature discrepancy between the local attribute features inferred from both global and local views is effectively eliminated. Thereby, the above process could optimize two encoders to iteratively improve visual attributes, and utilize these attributes to modify the features inferred by local and global views, thus achieving attribute parameterization.

To this end, the attribute parameterization constraint L c can be formulated as:

This loss encourages the two encoders to produce the same attributes for the same visual content, no matter which view it comes from, thus achieving attribute parameterization. However, training the model with Eq. (5) directly will make the attributes provided by two encoders become similar quickly since the encoders learn the attributes from another view according to Eq. (5). Therefore, using Eq. (5) more optimizes the parameters of two encoders, but has less impact on the parameters of the retrieval model.

To handle this limitation, we propose two mean encoders with the same structure as the above ones to produce attributes for features of another view. In this way, Eq. (5) can be written as

where E[ T G ] and E[ T L ] denote the mean encoders without learnable parameters, respectively. Their parameters can be updated in a temporal average manner. Concretely, at the t -th iteration, parameters E[ T G ]( θ G ) and E[ T L ]( θ L ) are accumulated by

where E (t) [ T G ]( θ G ) , ( E (t) [ T L ]( θ L ) ) and E (t -1) [ T G ]( θ G ) , ( E (t -1) [ T L ]( θ L ) ) respectively denote the parameters of the mean encoders in current iteration and last iteration, and θ G = (W G , b G ) and θ L = (W L , b L ) are the learnable parameters of P G and P L at the current iteration, respectively. The mean encoders are initialized as E (0) [ T G ]( θ G ) = θ G and E (0) [ T L ]( θ L ) = θ L . The hyper-parameter α is the updating ratio within the range of [0 , 1) .

Since the two mean encoders do not introduce learnable parameters, the attribute parameterization constraint could directly penalize the retrieval model and make its parameters be adjusted through back propagation. More importantly, the mean encoders could consider the knowledge learned from all previous stages to form more robust attributes for the current stage. Therefore, they have another important property that could remain sensitive even for rare attributes, consequently staying well generalized when facing unknown categories. After the attribute parameterization operation, V APNet will transform the retrieval model from category semantic extraction to attribute modeling, allowing the utilization of visual attributes to anticipate open-set class data.

3.4 Loss Functions

The retrieval model with specific parameters supervised by visual attributes can extract the attributes presented in objects and further procure in-depth semantic pattern understanding. For fine-grained understanding, these extracted visual attributes should clearly reflect the discrepancies of an objects, so that we can better identify visually similar objects. Thereby, we propose an auxiliary constraint based on the cross-entropy loss to ensure that these extracted visual attributes can contribute to decision boundary:

where y denotes the ground-truth label of the corresponding input, and C( · ) ∈ R c × N , N is the number of category in the training set.

The total loss L of VAPNet can be formulated as:

where λ is the hyper-parameter to balance the contributions of the individual loss item.

4 Experiments

4.1 Experimental Setup

Datasets. CUB-200-2011 dataset [5] contains 200 bird subcategories with 11,788 images. We utilize the first 100 classes (5,864 images) in training and the rest (5,924 images) in testing. The Stanford Cars dataset [16] contains 196 car models of 16,185 images. The spilt in Stanford Cars [16] is also similar to CUB, which is split into the first 98 classes (8,054 images) for training and the remaining classes (8,131 images) for testing. FGVC Aircraft dataset [23] is divided into first 50 classes (5,000 images) for training and the rest 50 classes (5,000 images) for testing. In Shop Clothes Retrieval (In-Shop) [21] contains 7,982 subcategories with 52, 712 images, and we use the 3,997 classes (25,882 images) in training and the rest 3,985 classes in testing. In-Shop is divided between a query (14,218 images) and a gallery set (12,162 images).

Evaluation protocols. We evaluate the retrieval performance by Recall@K with cosine distance, which is average recall scores over all query images in the test set and strictly follows the setting in the pioneer work [31]. Specifically, for each query, our model returns the top K similar images. In the top K returning images, the score will be 1 if there exists at least one positive image, and 0 otherwise.

Implementation Details. For backbone network, we apply the widely-used Resnet-50 [9] in our experiments with the pre-trained parameters. The input raw images are resized to 256 × 256 and cropped into 224 × 224 . We train our models using Stochastic Gradient Descent (SGD) optimizer with weight decay of 0.0001, momentum of 0.9, and batch size of 32. We adopt the commonly used data augmentation techniques, i.e. , random cropping, left-right flipping, and color jittering for robust feature representations. Our model is trained end-to-end on one NVIDIA 2080Ti GPUs for acceleration. The initial learning rate is set to 10 -5 , with exponential decay of 0.9 after every 5 epochs. The total number of training epochs is set to 200.

4.2 Ablation Study

The proposed VAPNet is optimized by a combination of two loss functions, an auxiliary loss L a and an attribute parameterization constraint L c or ˆ L c , which play different roles in guiding our model to understand unknown categories. Here, we perform thorough ablation experiments on CUB-200-2011 and Stanford Cars datasets to further validate the effectiveness of each loss function. Tab. 1 shows quantitative comparisons between different combinations of loss functions. The

baseline method only using L a obtains 69.5% and 89.3% Recall@1 accuracy on CUB-200-2011 and Stanford Cars datasets, respectively. The results reflect that the network only learns the discriminative object regions instead of the visual attributes, consequently impairing the retrieval performance of unknown categories. An addition of L c improves Recall@1 from 69.5% to 71.4%. However, L c optimizes the encoders to translate the semantics into visual attributes, instead of parameterizing the attributes into the retrieval model. During testing, our VAPNet still makes it hard to handle unknown categories, thus limiting the performance gains. To handle this limitation, we improve the attribute parameterization constraint to force this loss function to directly optimize the parameters within backbone network. As expected, ˆ L c can effectively make the backbone network sensitive to visual attributes and understand unknown categories accordingly. Furthermore, we also verify the effectiveness of L a , which can ensure that these visual attributes learnt from known categories keep discriminative. As shown in Tab. 1, the proposed VAPNet achieves 76.2% and 94.8% Recall@1 performance owing to the combination of ˆ L c and L a on two widely-used benchmarks. Additionally, during testing, the retrieval embedding extraction time remains the same as that of the baseline model, as the additional attribution exploration modules (AAM and APM) are only used during training.

4.3 Comparison with the State-of-the-Art Methods

Open-set Fine-grained Object Retrieval. We compare our VAPNet with some state-of-the-art approaches. In Tab. 2, the performance of different methods on CUB-200-2011, Stanford Cars-196, and FGVC Aircraft datasets is reported, respectively. In the table from top to bottom, the methods are roughly divided into three groups, i.e. , localization-based networks, metric-based frameworks, and our VAPNet.

As shown in Tab. 2, it is obvious that the retrieval performance obtained by our VAPNet is better than other methods no matter whether the localization-based or metric-based schemes are adopted. Concretely, existing works based on localization schemes, i.e. , CEP [4] and HDCL [46], tend to project the final retrieval embeddings into a category space. Despite the encouraging achievement, the shortcoming of these works is that they only focus on individual samples while neglecting the correlations among subcategories, thus limiting the retrieval performance. To address this problem, the effectiveness of these models based on metric schemes, i.e. , ETLR [14] and PNCA [33], can be largely attributed to their precise identification of negative/positive pairs through the manipulation of distances, which indirectly enhances the discriminative power of features. However, these existing works, e.g. , CEP [4], HIST [19] and PNCA [33], follow a close-set learning setting, where all the categories are pre-defined, to learn the discriminative and generalizable embeddings for identifying the visually similar objects of unknown subcategories. It is thus very challenging for a feature extractor trained in closed-set scenarios with classification or metric supervisions to capture discriminative discrepancies from unknown subcategories, consequently impairing the retrieval performance.

To handle this limitation, our V APNet focuses on learning visual attributes instead of discriminative clues to understand the unknown categories and clearly reflect their discriminative discrepancies, thus achieving a clear improvement of state-of-the-art methods.

Large-scale Product Retrieval. Our VAPNet exceeds all the existing methods and achieves the best performance with a retrieval accuracy of 93.9%, as shown in Tab. 3. Besides, we beat the second-best work CEP [4] and get a better result with a relative accuracy improvement of 3.0%. By leveraging the visual attributes learned from

Table 3: Comparison of different state-of-the-art methods on In-shop dataset.

known instances to identify category-specific discrepancies, our V APNet demonstrates impressive generalization capabilities.

4.4 Discussions

Patch number M . Tab. 4 ablates the role of patch number k in the attribute exploration module (§3.2). The optimal value is M = 4 (our default). Moreover, VAPNet is robust when M is in [4 , · · · , 16] , showing that it is beneficial

to spot visual attributes in a relatively many local regions. It is worth mentioning that when M is too large, the training time grows exponentially. However, when M is too small, the performance degrades due to easily overlooking some undiscovered regions. The results reveal that the local regions help the model attain accurate attributes, leading to better understanding unknown categories.

Attribute dimension k . We investigate the necessity of diverse attribute dimensions k for retrieval performance. As reported in Tab. 5, the dimension k stores the attribute knowledge. Although the large dimension could hold more infor-

mation related to attributes and has less impact on retrieval performance, it is easy to contain more useless information and increase storage overhead. However, when the dimension is small, it is not enough to precisely represent visual attributes, leading to degraded performance. Therefore, the optimal dimension is k = 256 .

Updating ration α . Tab. 6 reports the accuracy of using diverse updating ratios in Eq. (7). Notably, after increasing the updating ratio, the retrieval performance reduces progressively. These results reveal that a large updating ratio

quickly updates the projectors more relying on the learning parameters on the current stage, thus easily degrading the discrepancies between two projectors. Moreover, when using a small updating ratio, the projectors keep sensitive to previous learning knowledge and easily keep different during optimization, thus extracting precisely visual attributes from given features.

Balanced parameter λ . There is one balanced parameter in Eq. (10). The sensitivity analysis of the parameter are performed on CUB-200-2011 and the evaluation results are presented in Tab. 7. It is observed that the performances

of our VAPNet are not stable with the variation of λ (from 1 to 20). The retrieval performance increases as λ grows to 10, the consistency constraint with a large balanced parameter would force the network to focus on visual attributes. However, this constraint has a larger balanced parameter, which makes the network neglect the discriminative ability of visual attributes, thus reducing retrieval performance. The upper bound of retrieval performance may saturate at λ = 10 for learning visual attributes from known categories.

4.5 Visual Attribute Analysis

Interpreting visual attributes is difficult because these attributes are optimized in a latent space. We resort to an indirect way to interpret these attributes by visualizing their sources ( i.e. , Fig. 3) to display the content within them, and the features influenced by them ( i.e. , Fig. 4) to indirectly track these visual attributes.

Our VAPNet distills visual attributes from some local regions randomly cropped from inputs. Therefore, we provide some activation maps generated by Grad-CAM [30] to display some visual clues of interest in the attributes. In Fig. 3, these semantics provided by local regions could grab some rich details, and thus attributes projected by them could clearly represent these regions. Besides, we can also observe that the response maps of the local views highlight more object details compared to that

(a)

(b)

(c)

(d)

(e)

(f)

(a)

(b)

(c)

(d)

(e)

(f)

of the global view. Based on the above observation, we draw a conclusion that vision models can discover more semantic clues when replacing the input image with its local patches.

We exhibit the visualization results to demonstrate the influence of visual attributes. The referred visualizations of baseline and our model are shown in Fig. 4. It is shown that our model focuses on multiple local parts ( e.g. , head, wings and abdomen, etc.) instead of the fixed part predicted by baseline, ( e.g. , head of birds, front of cars, and middle of aircraft, etc). This verifies that using visual attributes learnt from known categories reasonably is beneficial for describing novel categories, thus improving the retrieval performance under open-set scenarios. More importantly, as shown in Fig. 4 (c) and (f), two sub-figures in the same row can roughly correspond to certain kinds of attributes of the fine-grained objects, e.g. , 'wings of birds", 'tires of cars", 'head or wing of planes", etc. The results reflect that the activation of objects parts is apparently attribute-related and contains the visual discrepancies among unknown categories accordingly, which could provide a clear explanation of the success in retrieving unknown categories.

5 Conclusion

In this paper, we propose a novel Visual Attribution Parameterization Network (V APNet) to handle unknown categories using visual attributes learnt from known instances in open-set fine-grained retrieval tasks. V APNet focuses on distilling visual attributes from semantic clues presented in objects and utilizing these attributes as supervisory signals to tune the retrieval model. In this way, we could transform the retrieval model trained by image-level supervisions from category semantic extraction to attribute modeling, and precisely represent unknown categories based on its parameters supervised by visual attributes. Therefore, V APNet successfully alleviates the problem behind facing instances from unseen novel categories. Last but not the least, the overall retrieval pipeline is simple and flexible. Extensive experiments demonstrate that our method outperforms the state-of-the-art methods by a significant margin, indicating the effectiveness of attribute modelling on facing unknown categories.

Limitations & Broader Impacts: By introducing VAPNet, we aim to extract visual attributes from seen classes without relying on attribute annotations to differentiate unseen classes. This innovation has the potential to greatly impact open-domain tasks. In particular, annotating a large number of attributes for unseen categories in open-domain tasks can be a costly and time-consuming endeavor. By enabling the model to automatically capture knowledge about unseen classes, our approach reduces the reliance on attribute annotations, resulting in decreased manual labeling costs. Furthermore, our approach exhibits improved adaptability to data from domains resembling the training set, such as natural images or medical images. This heightened adaptability contributes to stronger generalization capabilities, allowing the model to perform well in real-world scenarios. Ultimately, our solution has the potential to propel the advancement of open-domain tasks and facilitate their practical applications.

References

Kenan E. Ak, Ashraf A. Kassim, Joo-Hwee Lim, and Jo Yew Tham. Learning attribute representations with localization for flexible fashion search. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018 , pages 7708-7717. Computer Vision Foundation / IEEE Computer Society, 2018.

Francesco Barbato, Marco Toldo, Umberto Michieli, and Pietro Zanuttigh. Latent space regularization for unsupervised domain adaptation in semantic segmentation. In CVPR , pages 2835-2845. Computer Vision Foundation / IEEE, 2021.

Sean Bell and Kavita Bala. Learning visual similarity for product design with convolutional neural networks. ACM Trans. Graph. , 34(4):98:1-98:10, 2015.

Malik Boudiaf, Jérôme Rony, Imtiaz Masud Ziko, Eric Granger, Marco Pedersoli, Pablo Piantanida, and Ismail Ben Ayed. A unifying mutual information view of metric learning: Cross-entropy vs. pairwise losses. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, ECCV , volume 12351 of Lecture Notes in Computer Science , pages 548-564. Springer, 2020.

Steve Branson, Grant Van Horn, Serge J. Belongie, and Pietro Perona. Bird species categorization using pose normalized deep convolutional nets. CoRR , abs/1406.2952, 2014.

Mohamed Elhoseiny, Yizhe Zhu, Han Zhang, and Ahmed M. Elgammal. Link the head to the "beak": Zero shot learning from noisy text description at part precision. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 , pages 6288-6297. IEEE Computer Society, 2017.

Vittorio Ferrari and Andrew Zisserman. Learning visual attributes. In John C. Platt, Daphne Koller, Yoram Singer, and Sam T. Roweis, editors, Advances in Neural Information Processing Systems 20, Proceedings of the Twenty-First Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, December 3-6, 2007 , pages 433-440. Curran Associates, Inc., 2007.

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017 , pages 2980-2988. IEEE Computer Society, 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016 , pages 770-778, 2016.

Junshi Huang, Rogério Schmidt Feris, Qiang Chen, and Shuicheng Yan. Cross-domain image retrieval with a dual attribute-aware ranking network. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015 , pages 1062-1070. IEEE Computer Society, 2015.

Dat Huynh and Ehsan Elhamifar. Fine-grained generalized zero-shot learning via dense attribute-based attention. In CVPR , pages 4482-4492. Computer Vision Foundation / IEEE, 2020.

Dor Kedem, Stephen Tyree, Kilian Q. Weinberger, Fei Sha, and Gert R. G. Lanckriet. Non-linear metric learning. In Peter L. Bartlett, Fernando C. N. Pereira, Christopher J. C. Burges, Léon Bottou, and Kilian Q. Weinberger, editors, NeurIPS , pages 2582-2590, 2012.

Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. Proxy anchor loss for deep metric learning. In CVPR , pages 3235-3244. Computer Vision Foundation / IEEE, 2020.

Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. Embedding transfer with label relaxation for improved metric learning. In CVPR , pages 3967-3976. Computer Vision Foundation / IEEE, 2021.

ByungSoo Ko, Geonmo Gu, Han-Gyu Kim, and ByungSoo Ko. Learning with memory-based virtual classes for deep metric learning. In ICCV , pages 11772-11781. IEEE, 2021.

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV Workshops 2013, Sydney, Australia, December 1-8, 2013 , pages 554-561, 2013.

Neeraj Kumar, Alexander C. Berg, Peter N. Belhumeur, and Shree K. Nayar. Describable visual attributes for face verification and image search. IEEE Trans. Pattern Anal. Mach. Intell. , 33(10):1962-1977, 2011.

Zechao Li, Hao Tang, Zhimao Peng, Guo-Jun Qi, and Jinhui Tang. Knowledge-guided semantic transfer network for few-shot image recognition. IEEE Transactions on Neural Networks and Learning Systems , pages 1-15, 2023.

Jongin Lim, Sangdoo Yun, Seulki Park, and Jin Young Choi. Hypergraph-induced semantic tuplet loss for deep metric learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 212-222. IEEE, 2022.

Lizhao Liu, Shangxin Huang, Zhuangwei Zhuang, Ran Yang, Mingkui Tan, and Yaowei Wang. DAS: densely-anchored sampling for deep metric learning. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, ECCV , volume 13686 of Lecture Notes in Computer Science , pages 399-417. Springer, 2022.

Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In CVPR , pages 6738-6746. IEEE Computer Society, 2017.

Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016 , pages 1096-1104. IEEE Computer Society, 2016.

Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. CoRR , abs/1306.5151, 2013.

Olga Moskvyak, Frédéric Maire, Feras Dayoub, and Mahsa Baktashmotlagh. Keypoint-aligned embeddings for image retrieval and re-identification. In WACV , pages 676-685. IEEE, 2021.

Genevieve Patterson, Chen Xu, Hang Su, and James Hays. The SUN attribute database: Beyond categories for deeper scene understanding. Int. J. Comput. Vis. , 108(1-2):59-81, 2014.

Marc'Aurelio Ranzato, Y-Lan Boureau, Sumit Chopra, and Yann LeCun. A unified energy-based framework for unsupervised learning. In Marina Meila and Xiaotong Shen, editors, Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics, AISTATS 2007, San Juan, Puerto Rico, March 21-24, 2007 , volume 2 of JMLR Proceedings , pages 371-379. JMLR.org, 2007.

Karsten Roth, Timo Milbich, Björn Ommer, Joseph Paul Cohen, and Marzyeh Ghassemi. Simultaneous similarity-based self-distillation for deep metric learning. In Marina Meila and Tong Zhang, editors, ICML , volume 139 of Proceedings of Machine Learning Research , pages 9095-9106. PMLR, 2021.

Karsten Roth, Oriol Vinyals, and Zeynep Akata. Non-isotropy regularization for proxy-based deep metric learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 7410-7420. IEEE, 2022.

Jenny Seidenschwarz. Learning intra-batch connections for deep metric learning. In Marina Meila and Tong Zhang, editors, ICML , volume 139 of Proceedings of Machine Learning Research , pages 9410-9421. PMLR, 2021.

Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. Int. J. Comput. Vis. , 128(2):336-359, 2020.

Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In CVPR , pages 4004-4012. IEEE Computer Society, 2016.

Hao Tang, Chengcheng Yuan, Zechao Li, and Jinhui Tang. Learning attention-guided pyramidal features for few-shot fine-grained recognition. Pattern Recognit. , 130:108792, 2022.

Eu Wern Teh, Terrance DeVries, Graham W. Taylor, and Graham. Proxynca++: Revisiting and revitalizing proxy neighborhood component analysis. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and JanMichael Frahm, editors, ECCV , volume 12369 of Lecture Notes in Computer Science , pages 448-464. Springer, 2020.

Shijie Wang, Jianlong Chang, Haojie Li, Zhihui Wang, Wanli Ouyang, and Qi Tian. Open-set fine-grained retrieval via prompting vision-language evaluator. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023 , pages 19381-19391. IEEE, 2023.

Shijie Wang, Jianlong Chang, Zhihui Wang, Haojie Li, Wanli Ouyang, and Qi Tian. Fine-grained retrieval prompt tuning. In Brian Williams, Yiling Chen, and Jennifer Neville, editors, Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023 , pages 2644-2652. AAAI Press, 2023.

Shijie Wang, Haojie Li, Zhihui Wang, and Wanli Ouyang. Dynamic position-aware network for fine-grained image recognition. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021 , pages 2791-2799. AAAI Press, 2021.

Shijie Wang, Zhihui Wang, Haojie Li, Jianlong Chang, Wanli Ouyang, and Qi Tian. Accurate fine-grained object recognition with structure-driven relation graph networks. In IJCV , 2023.

Shijie Wang, Zhihui Wang, Haojie Li, Jianlong Chang, Wanli Ouyang, and Qi Tian. Semantic-guided information alignment network for fine-grained image recognition. IEEE Transactions on Circuits and Systems for Video Technology , pages 1-1, 2023.

Shijie Wang, Zhihui Wang, Haojie Li, and Wanli Ouyang. Category-specific semantic coherency learning for fine-grained image recognition. In Chang Wen Chen, Rita Cucchiara, Xian-Sheng Hua, Guo-Jun Qi, Elisa Ricci, Zhengyou Zhang, and Roger Zimmermann, editors, MM'20: The 28th ACM International Conference on Multimedia, Virtual Event / Seattle, WA, USA, October 12-16, 2020 , pages 174-183. ACM, 2020.

Shijie Wang, Zhihui Wang, Haojie Li, and Wanli Ouyang. Category-specific nuance exploration network for fine-grained object retrieval. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 March 1, 2022 , pages 2513-2521. AAAI Press, 2022.

Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew R. Scott. Multi-similarity loss with general pair weighting for deep metric learning. In CVPR , pages 5022-5030. Computer Vision Foundation / IEEE, 2019.

Xiu-Shen Wei, Jian-Hao Luo, Jianxin Wu, and Zhi-Hua Zhou. Selective convolutional descriptor aggregation for fine-grained image retrieval. IEEE Trans. Image Process. , 26(6):2868-2881, 2017.

Xiu-Shen Wei, Yang Shen, Xuhao Sun, Han-Jia Ye, and Jian Yang. A$ˆ2$-net: Learning attribute-aware hash codes for large-scale fine-grained image retrieval. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 5720-5730, 2021.

Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, ECCV , volume 9911 of Lecture Notes in Computer Science , pages 499-515. Springer, 2016.

Zhirong Wu, Alexei A. Efros, and Stella X. Yu. Improving generalization via scalable neighborhood component analysis. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, ECCV , volume 11211 of Lecture Notes in Computer Science , pages 712-728. Springer, 2018.

Xianxian Zeng, Shun Liu, Xiaodong Wang, Yun Zhang, Kairui Chen, and Dong Li. Hard decorrelated centralized loss for fine-grained image retrieval. Neurocomputing , 453:26-37, 2021.

Zican Zha, Hao Tang, Yunlian Sun, and Jinhui Tang. Boosting few-shot fine-grained recognition with background suppression and foreground alignment. IEEE Trans. Circuits Syst. Video Technol. , 33(8):39473961, 2023.

Andrew Zhai and Hao-Yu Wu. Classification is a strong baseline for deep metric learning. In BMVC , page 91. BMVA Press, 2019.

Hua Zhang, Xiaochun Cao, and Rui Wang. Audio visual attribute discovery for fine-grained object recognition. In Sheila A. McIlraith and Kilian Q. Weinberger, editors, Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018 , pages 7542-7549. AAAI Press, 2018.

Wenzhao Zheng, Chengkun Wang, Jiwen Lu, and Jie Zhou. Deep compositional metric learning. In CVPR , pages 9320-9329. Computer Vision Foundation / IEEE, 2021.

Wenzhao Zheng, Borui Zhang, Jiwen Lu, and Jie Zhou. Deep relational metric learning. In ICCV , pages 12045-12054. IEEE, 2021.

Xiawu Zheng, Rongrong Ji, Xiaoshuai Sun, Yongjian Wu, Feiyue Huang, and Yanhua Yang. Centralized ranking loss with weakly supervised localization for fine-grained object retrieval. In Jérôme Lang, editor, IJCAI , pages 1226-1233. ijcai.org, 2018.

Xiawu Zheng, Rongrong Ji, Xiaoshuai Sun, Baochang Zhang, Yongjian Wu, and Feiyue Huang. Towards optimal fine grained retrieval via decorrelated centralized loss with normalize-scale layer. In AAAI , pages 9291-9298. AAAI Press, 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work proposes a new approach for open-set fine-grained retrieval. Their main contribution is the new state-of-the-art method termed VAPNet, which exploits objects’ attributes to discriminate between known and unknown objects. Due to the absence of attribute annotations, their training loss includes a pipeline to extract and refine attributes in an unsupervised fashion.\n\nSTRENGTHS:\n1. The authors’ idea to exploit visual attributes to discriminate between known and unknown classes is novel for the task.\n2. The approach to learn such attributes in an unsupervised fashion and in the open-set scenario is quite interesting.\n3. The quality of the manuscript is good.\n4. Their evaluation protocol makes sense and their method achieves the best results across three datasets when compared with related works.\n\nWEAKNESSES:\n1. Section 3 is hard to follow, mainly the architectural choices. There are many components in the method. The authors could refer to Figure (2) whenever introducing some components to guide the reader’s understanding.\n2. The differences in the pipeline between training and evaluation are not stated explicitly. This could help in understanding the overall method better.\n3. Figure (2) could be improved to better convey the method (the current image is too crowded). From the text and the image it may be hard to understand whether the network represented in the retrieval module and the attribute exploration modules are the same or not. The retrieval embeddings returned by the pooling operations seems superfluous in the image. It may be best to either disentangle the components, or increase abstraction of some components. Moreover, the caption is too short and conveys no information. Ideally, the figure should be able to convey all that is required to understand the method without additional context.\n4. This is a minor suggestion. Some section names are misleading or could be improved. For instance, many titles contain the word “attribute” to the point that its presence means little. Moreover, the title for Section 3.4 is misleading and it may be best to rename it.\n\nQUESTIONS:\nplease refer to Weakness section\n\nLIMITATIONS:\n1. No limitation was reported. It may be compute intensive to extract multiple patches from the same image or for the overall training pipeline. Moreover, it could be that the evaluation process is slower due to the additional components introduced. It may be important to clearly explain any limitation to the work.\n2. Same for broader impact. It should be stated", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes VAPNet to learn visual attributes from known fine-grained categories and parameterize them into final open-set retrieval. To learn visual attributes without attribute annotation, VAPNet explicitly attains some semantics with rich details via making use of local image patches and distills the visual attributes from these discovered semantics. Then, it incorporates the online refinement of these visual attributes into the training process to iteratively improve them and simultaneously regards these attributes as supervisory signals to tune the retrieval models. Experimental results on on open-set fine-grained retrieval benchmarks demonstrate improved performance compared to existing methods.\nHowever, I have some concerns about the motivation and the methodology employed. Please refer to the detailed comments below.\n\nSTRENGTHS:\n- It is commendable to introduce the visual attributes to open-set fine-grained retrieval.\n- This paper is well-written and easy to follow.\n\nWEAKNESSES:\n- The author do not explain the necessity of introducing visual attributes into open-set fine-grained retrieval.\n- In line 149, the attribute exploration module attain semantic clues of input object via randomly cropping local patches from the input image, but is hard to guarantee the complete coverage of all discriminative local regions of the input image by random cropping.\n- In addition to local attributes, visual attributes can also contain global attributes, such as shape. The design of attribute exploration module only focuses on local visual attributes while ignoring global visual attributes.\n- The artificially defined attributes correspond to different visual features and can be decoupled from each other. While the proposed method does not further constrain the relationship between visual attributes(i.e. Eq.(6) the attribute parameterization constraint Lc), and the extracted visual attributes cannot be guaranteed to have the characteristics of this aspect. It is more like to enhance the local representation ability of the model.\n- There are some errors in the details of the paper, such as Eq.(6) in line 322, which actually corresponds to Eq.(5).\n\nQUESTIONS:\n- The corresponding relationship between local features and different patch of global features in line 191 is not clearly explained.\n- The global view in Figure 3(top row(b)(d)(f)) shows that the model only pays attention to some local parts, which is quite different from the VAPNet shown in Figure 4(top row(c)(f)), which focuses on more local parts.\n- Does the randomly cropped size have an effect on the model, and what is the relationship between the cropped size, the Patch number M, and visual attribute dimension k.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFirstly, they introduce a novel approach to address the problem of open-set fine-grained retrieval settings. They transform the retrieval model, which is typically trained using image-level supervisions for category semantic prediction, into attribute modeling. This transformation helps alleviate the challenges posed by open-set fine-grained retrieval.\n\nSecondly, they propose a Visual Attribution Parameterization Network (VAPNet) that distills visual attributes from various semantics observed in seen fine-grained objects. These attributes are then transcribed into parameters within the retrieval model. This parameterization allows for the precise representation of unknown categories based on their transformed parameters derived from visual attributes.\n\nThe authors conduct extensive experiments to evaluate their method's performance on open-set fine-grained retrieval tasks. The results demonstrate that their proposed approach, VAPNet, brings significant benefits. It achieves an average accuracy gain of 8.6% compared to the recent state-of-the-art work on three open-set fine-grained retrieval benchmarks.\n\nSTRENGTHS:\nThe novelty presented in this paper revolves around the proposed Visual Attribution Parameterization Network (VAPNet) and its application in open-set fine-grained retrieval tasks.\n\nThe authors introduce VAPNet as a novel approach to handle unknown categories in such tasks. VAPNet focuses on distilling visual attributes from semantic clues observed in known instances. These visual attributes serve as supervisory signals to fine-tune the retrieval model. By doing so, the authors transform the retrieval model, originally trained with image-level supervisions for category semantic extraction, into attribute modeling. This transformation enables precise representation of unknown categories based on parameters supervised by visual attributes. As a result, VAPNet effectively addresses the challenges associated with encountering instances from unseen novel categories.\n\nFurthermore, the authors highlight the simplicity and flexibility of the overall retrieval pipeline enabled by VAPNet. They emphasize that the proposed method surpasses state-of-the-art approaches by a significant margin, demonstrating the effectiveness of attribute modeling when dealing with unknown categories in fine-grained retrieval tasks.\n\nWEAKNESSES:\n1.\tThe framework generally follows some contrastive learning works. The input pair – image and patch, can be regarded as two views in the contrastive learning, which has been used in RegionCL[1]. Additionally, using KL divergence as objectives for distillation has been proposed in RepDistiller [2].\n\n2.\tThe authors claim to solve the task of open-set fine-grained retrieval. However, the experiments on open-set image retrieval are missing in the paper.\n\n3.\tWhy most of SOTA work reported in Table 2 are proposed for deep metric learning instead of the image retrieval? Additionally, why some benchmark datasets for image retrieval, such as CIFAR, COCO, Oxford, are not used here?\n\n4.\tThere are lots of confusing concepts in the manuscript. For example, what does “c” and “k” in “… weight matrix W_L ∈ R^c x k …” in Line 179 and Line 194 refer to? What does “sampler” in L189 refer to? What does “L_s” in Eq. (8) refer to?\n\n5.\tThere are some typo errors. For example, “A” in Eq. (2) should be in bold.\n\n[1] Xu, Yufei, et al. \"Regioncl: exploring contrastive region pairs for self-supervised representation learning.\" ECCV 2022.\n\n[2] Yonglong Tian, et al. Contrastive Representation Distillation, ICLR 2020.\n\nQUESTIONS:\nPlease reply the concerns in the weakness section during rebuttal.\n\nLIMITATIONS:\nLimitations are not discussed in this paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of fine-grained image retrieval, where the model must identify subtle object attributes in order to distinguish between visually similar classes.\nTo achieve this, the paper proposes a Visual Attribute Parameterization Network to localize object attributes in images to enhance performance.\nSpecifically, the paper designs Attribute Exploration Module that extracts image features into local patches and projects them into attribute space. \nMoreover, an Attribute Parameterization Module is introduced to iteratively refine visual features for fine-grained retrieval.\nThe paper conducts experiments on three datasets of CUB, Stanford Cars, and FGVC Aircraft for retrieval tasks.\n\nSTRENGTHS:\n+ The direction of fine-grained image retrieval without attribute annotation is interesting with impactful real-world applications.\n+ The idea of detecting attributes by using local image patches is sensible and has been demonstrated by prior works to be effective for fine-grained classification [6,11]\n+ The paper shows promising retrieval improvements on CUB and Cars datasets.\n\nWEAKNESSES:\n+ The claim that \"we are the first to transform the retrieval model trained by image-level supervisions from category semantic prediction into attribute modeling\" is not well supported. To be specific, many works approach the problem of fine-grained recognition without attribute annotations [A, B] which also leverage the ideas of attribute localization just like the proposed method. Thus, the main difference between the proposed method and SOTA is simply the retrieval tasks instead of classification tasks, which lack novelty.\n+ The reviewer is not yet convinced by the significance of the experimental results, as the paper lacks strong comparisons with appropriate baselines such as [A, B] that are specifically designed to capture fine-grained attributes without annotation. Thus, it is unclear to the reviewer how effective the proposed method is compared to SOTA.\n+ The experiment datasets are small in scale and might not be challenging enough for grained classification. Specifically, CUB, CAR, and Aircraft datasets might not have a diverse number of attributes per class. Thus, the reviewer has some doubts about the effectiveness of the proposed method. It would be more convincing if experiments on recent fine-grained datasets such as DeepFashion [19] or iNaturalist [C].\n\n[A] Lin et al., Bilinear CNN Models for Fine-Grained Visual Recognition\n[B] Ding et al., Selective Sparse Sampling for Fine-Grained Image Recognition\n[C] Grant Van Horn et al., The iNaturalist Species Classification and Detection Dataset\n\nQUESTIONS:\nPlease refer to the weakness section\n\nLIMITATIONS:\nSufficiently addressed", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-10"}

comments: SUMMARY:
This paper works on an open-set fine-grain retrieval task. To align novel categories and the basic categories into a unified space, this paper proposes a Visual Attribute Parameterization Network (VAPNet) to learn visual attributes from known categories and parameterize them into the retrieval model. Extensive experiments on open-set fine-grained retrieval datasets validate the superior performance of our VAPNet over existing solutions.

STRENGTHS:
- The motivation of this paper is sufficient and reasonable. Adopting a series of visual attributes to represent an image makes it easier to embed unknown categories and known categories into a unified space. 

- The experimental results show this paper has achieved SOTA performance across three datasets.

WEAKNESSES:
- The visual attributes seem to be a series of local discriminative patch-level features by contrastive learning. It is a very implicit manner to extract attributes. With the development of large language models (LLM), it is possible to represent visual attributes as a structural language description through LLM-driven multi-modal model. Please discuss it.

- Additional attributes extraction model may bring additional computation. A detailed computation and parameters number discussion should be included.

QUESTIONS:
See weaknesses.

LIMITATIONS:
No limitations discussion is provided in this paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-27