# Modeling Multi-modal Cross-interaction for Multi-label Few-shot Image Classification Based on Local Feature Selection

Kun Yan, Zied Bouraoui, Fangyun Wei, Chang Xu, Ping Wang, Shoaib

Jameel, Steven Schockaert

<figure><img src="image_1.png" /></figure>

## To cite this version:

Kun Yan, Zied Bouraoui, Fangyun Wei, Chang Xu, Ping Wang, et al.. Modeling Multi-modal Cross-interaction for Multi-label Few-shot Image Classification Based on Local Feature Selection. ACM Transactions on Multimedia Computing, Communications and Applications, 2025, 21 (3), pp.80. 10.1145/3711867. hal-05078232

HAL Id: hal-05078232

https://hal.science/hal-05078232v1

Submitted on 17 Sep 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Modeling Multi-modal Cross-interaction for Multi-label Few-shot
Image Classification Based on Local Feature Selection

Kun Yan¹, Zied Bouraoui², Fangyun Wei³, Chang Xu³*, Ping Wang¹,⁴†,
Shoaib Jameel⁵, Steven Schockaert⁶

¹School of Computer Science, Peking University, Beijing, China

²CRIL - University of Artois & CNRS, France

³School of Computer Science, University of Sydney, Australia

⁴National Engineering Research Center for Software Engineering, Peking University, China

⁵Electronics and Computer Science, University of Southampton, UK

⁶School of Computer Science and Informatics, Cardiff University, UK

kyan2018@pku.edu.cn, zied.bouraoui@cril.fr, fwei8714@uni.sydney.edu.au
c.xu@sydney.edu.au, pwang@pku.edu.cn
M.S.Jameel@southampton.ac.uk, schockaerts1@cardiff.ac.uk

Abstract

The aim of multi-label few-shot image classification (ML-FSIC) is to assign semantic labels to images, in settings where only a small number of training examples are available for each label. A key feature of the multi-label setting is that an image often has several labels, which typically refer to objects appearing in different regions of the image. When estimating label prototypes, in a metric-based setting, it is thus important to determine which regions are relevant for which labels, but the limited amount of training data and the noisy nature of local features make this highly challenging. As a solution, we propose a strategy in which label prototypes are gradually refined. First, we initialize the prototypes using word embeddings, which allows us to leverage prior knowledge about the meaning of the labels. Second, taking advantage of these initial prototypes, we then use a Loss Change Measurement (LCM) strategy to select the local features from the training images (i.e. the support set) that are most likely to be representative of a given label. Third, we construct the final prototype of the label by aggregating these representative local features using a multi-modal cross-interaction mechanism, which again relies on the initial word embedding-based prototypes. Experiments on COCO, PASCAL VOC, NUS-WIDE, and iMaterialist show that our model substantially improves the current state-of-the-art.

# 1 Introduction

Multi-label image classification (ML-IC) has received considerable attention in recent years [Wang et al.(2016), Chen et al.(2019), Wang et al.(2017), Yazici et al.(2020)]. This task aims to assign descriptive labels to images, where images can have multiple labels. Each label typically describes an object that is shown in the image. As a baseline strategy, we can simply train a binary classifier for each label, to predict the presence of a given type of object. To improve on this strategy, various approaches have been proposed to exploit label dependencies, e.g. taking

<footnote>*Corresponding author.</footnote>

<footnote>†Also with the School of Software and Microelectronics, Peking University and the Key Laboratory of High Confidence Software Technologies (PKU), Ministry of Education.</footnote>
<figure><img src="image_3.png" /><figcaption>Figure 1: In the 1-shot single-label setting, a given training image can be interpreted as a prototype for the considered label (left). In the multi-label setting, labels are related to different regions of the image, and these regions need to be identified before meaningful prototypes can be obtained (right).</figcaption></figure>

advantage of the fact that the presence of one label makes the presence of another label more (or less) likely [Wang et al.(2016), Yazici et al.(2020), Chen et al.(2018), Chen et al.(2019)]. In this paper, we are specifically interested in the few-shot setting, i.e. the case where only a limited number of training images are available for the labels (e.g., 1 or 5 images per label). Clearly, in this setting, we cannot rely on standard strategies for training binary image classifiers, nor on label co-occurrence statistics.

The challenge of few-shot image classification (FSIC) has garnered significant attention, particularly in the single-label (SL) setting [Finn et al.(2017), Nichol et al.(2018), Snell et al.(2017), Vinyals et al.(2016), Yan et al.(2021c), Dai et al.(2023), Zhang et al.(2024)]. Metric-based approaches hold a prominent position in SL-FSIC due to their strong performance and fast adaptability to new categories. However, it is worth noting that such methods cannot be directly extended to the multi-label setting. Metric-based approaches usually involve learning a prototype for each image category and subsequently assigning images to the category whose prototype is closest. These prototypes are typically derived by averaging a representation of the training images. In the seminal ProtoNet model [Snell et al.(2017)], for instance, prototypes are simply defined as the average of the global feature maps of the available training examples. This strategy crucially relies on the assumption that most of the image is somehow relevant to its category. In the multi-label setting, such an assumption is highly questionable, given that different labels tend to refer to different parts of the image, as illustrated in Fig. 1. For instance, suppose the figure on the right is the only training example with the label *bus*. If we simply use the global feature map of this image as the corresponding prototype, the model is likely to mistakenly assume that other images of cityscapes should also be assigned this label. To obtain a high-quality prototype, it is thus imperative that we can estimate which regions of the image correspond to a given label.

In this paper, we propose a novel metric-based model for multi-label few-shot image classification (ML-FSIC). Given the aforementioned concerns, we need a strategy that is based on local image features, allowing us to focus on those parts of the training images that are most likely to be relevant. However, as we may only have a single training example for some labels, we cannot implement such a strategy without some kind of prior knowledge about the meaning of the labels. Therefore, we rely on word embeddings, such as those from the GloVe model [Pennington et al.(2014)], to provide us with this prior knowledge. Some previous works for the single-label setting have already relied on word embeddings for inferring prototypes directly [Xing et al.(2019), Yan et al.(2021a)], but as the resulting prototypes are inevitably noisy, they are normally used in combination with prototypes that are derived from visual features. Unfortunately, such visual prototypes are hard to obtain in the ML-FSIC setting.

For this reason, we adopt a different approach to efficiently leverage word embeddings for ML-FSIC. We first construct initial prototypes of the labels based on their word embeddings.
Specifically, we introduce a cross-modality loss to learn a joint embedding space, in which the representations of words are aligned with the corresponding visual features. The resulting word representations are too noisy to be used directly. To address this, we propose a multi-modal cross-interaction strategy guided by these noisy prototypes. Concretely, we develop multiple ways of interaction between word embeddings and visual features, including cross-modal attention and cross-modal dynamic convolution, to refine the generation of the final prototype. As an important advantage, this strategy can be applied to previously unseen labels without the need for any fine-tuning of the model. We will refer to the model described above as our Base model.

As a further improvement of this base model, we propose a strategy for identifying the most representative local features prior to prototype generation. To achieve this, we propose a loss change measurement module (LCM), drawing inspiration from [You et al.(2019)], to improve the selection of local features by assessing which features exert the greatest influence on the loss function. By filtering noisy local features based on this module, we end up with higher-quality prototypes, but the resulting model has the drawback that it needs to be fine-tuned before it can be applied to unseen labels. The above-described model will be referred to as the LCM model.

Besides, we propose a number of changes to the evaluation methodology for ML-FSIC systems. The most important change is concerned with how support sets are sampled, as part of an episode-based strategy. The standard *N*-way *K*-shot framework for evaluating FSIC systems is based on the idea that exactly *K* training images are available for each category of interest. While earlier work in ML-FSIC has aimed to mimic this *N*-way *K*-shot framework as closely as possible, we found this to have significant drawbacks when images can have multiple labels. We also propose some changes related to how the query set is sampled and the choice of evaluation metrics. Finally, we propose three new ML-FSIC datasets based on PASCAL VOC [Everingham et al.(2015)], NUS-WIDE [Chua et al.(2009b)] and iMaterialist [Guo et al.(2019)], by adapting these ML-IC datasets to the few-shot setting.

The contributions of this work are summarized as follows:

* A novel multi-modal cross-interaction mechanism for prototype generation in ML-FSIC is proposed, featuring channel-wise cross-attention guided by word embeddings and word embedding-based dynamic convolution.

* A loss-change measurement module is introduced to identify representative local features, which helps in mitigating the interference of background features.

* New ML-FSIC benchmarks from NUS-WIDE and iMaterialist are introduced, and extensive experiments on a total of four datasets demonstrate the effectiveness and superiority of our approach over current state-of-the-art methods.

This paper is an extended version of our previous work [Yan et al.(2022)], with several key updates. First, we enhance the overall approach by introducing a novel multi-modal cross-interaction mechanism, accompanied by updated experimental results. Second, we include results for the 5-shot setting and provide a more in-depth analysis. Third, we have introduced the Loss Change Measurement (LCM) module, which consistently improves the results. Our implementation and associated data are available at https://github.com/yk-pku/MMCI-ML-FSIC.

## 2 Related Work

In this section, we review the related work on *multi-label image classification* (ML-IC), *few-shot image classification* (FSIC), and the combined area of *multi-label few-shot image classification* (ML-FSIC). We also briefly discuss the impact of Vision-Language models on this field.
## 2.1 Multi-Label Image Classification

Early solutions for ML-IC simply learned a binary classifier for each label [Tsoumakas and Katakis (2007)]. More recently, various methods have been proposed to improve this basic strategy by exploiting label dependencies in some way. For instance, the CNN-RNN architecture [Wang et al. (2016)] learns a joint embedding space for representing both images and labels, which is used to predict image-label relevance. To avoid the need for a predefined label order, as in RNN-based architectures, [Yazici et al. (2020)] proposes Minimal Loss Alignment (MLA) and Predicted Label Alignment (PLA) to dynamically align the ground truth labels with the predicted label sequence. Some studies [Wang et al. (2020), Chen et al. (2019), You et al. (2020), Zhou et al. (2023c), Yuan et al. (2023)] also exploit Graph Convolutional Networks (GCN) [Kipf and Welling (2017)] to model label dependencies more explicitly. Besides, attention mechanisms play an important role in ML-IC, as they can associate labels with specific image regions [Wang et al. (2017), Zhu et al. (2017), Zhu and Wu (2021), Zhou et al. (2023a), Zhou et al. (2023b)]. AAMN [Zhou et al. (2023b)] proposes an attention-augmented memory network that mines contextual information from various categories within the dataset to enhance input feature representation. Recently, MMDSCR [Kuang and Li (2024)] introduced an approach for dynamically constructing a semantic relationship graph using multi-scale feature reconstruction coupled with channel dual-branch cross-attention. However, the above methods require a large amount of training data and can thus not be directly applied in the few-shot setting.

## 2.2 Few-Shot Image Classification

Different strategies for single-label few-shot image classification have already been proposed, with metric-based [Sung et al. (2018), Kim et al. (2019), Satorras and Estrach (2018), Ye et al. (2020), Cheng et al. (2023)] and meta-learning based [Ravi and Larochelle (2017), Finn et al. (2017), Li et al. (2017), Rusu et al. (2019)] methods being the most prominent. Methods of the latter kind use a meta-learner to learn aspects like model weight initialization [Finn et al. (2017)] and update step size [Li et al. (2017)], allowing the adaption of model parameters to new categories in the few-shot regime. It allows adapting architectures to novel tasks with just a few steps of a gradient-based task optimizer. However, our method is more closely related to metric-based methods, which aim to learn a generalizable visual embedding space in which different image categories are spatially separated. For instance, ProtoNet [Snell et al. (2017)] generates a visual prototype for each class by simply averaging the embeddings of the support images. The category of a query image is then determined by its Euclidean distance to these prototypes. Instead of using Euclidean distance, the Relation Network [Sung et al. (2018)] learns to model the distance between query and support images. Other notable models include FEAT [Ye et al. (2020)], which uses a transformer to contextualize the image features relative to the support set, and PSST [Chen et al. (2021)], which introduced a self-supervised learning strategy.

While most metric-based models rely on global features, methods exploiting local features have also been proposed [Li et al. (2019), Zhang et al. (2022)], but these methods are designed for single-label classification. For instance, [Li et al. (2019)] calculates the similarity between all local features of the query image and all local features of the support images. As such, there is no attempt to focus on particular regions of the support images. The use of word vectors has also been considered for estimating visual prototypes [Xing et al. (2019), Yan et al. (2021a), Li et al. (2020), Yan et al. (2021b)]. However, due to the inevitably noisy nature of the predicted prototypes, such methods are best used in combination with prototypes obtained from visual features.
## 2.3 Multi-Label Few-Shot Image Classification

The ML-FSIC problem has only received limited attention. LaSO [Alfassy et al.(2019)] was the first model designed to address this problem. It relies on a data augmentation strategy that generates synthesized feature vectors via label-set operations. KGGR [Chen et al.(2020)] uses a GCN to take label dependencies into account, where labels are modelled as nodes, and two nodes are connected if the corresponding labels tend to co-occur. The strength of these label dependencies is normally estimated from co-occurrence statistics, but for labels with limited training data, dependency strength is instead estimated based on GloVe word vectors [Pennington et al.(2014)]. In [Simon et al.(2022)], several FSIC methods are extended to the multi-label setting, and a neural module is introduced to estimate the label count of a given sample by exploiting relational inference. BCR [An et al.(2024)] is designed to uncover underlying correlations between instances and labels, incorporating varying levels of importance information. This approach analyzes correlations from both instance-to-label and label-to-instance perspectives. In [Li et al.(2021c)], an ML-FSIC method is proposed which learns compositional embeddings based on weak supervision. However, this method is not directly comparable with our method due to its use of weak supervision.

Compared to existing methods such as LaSO [Alfassy et al.(2019)], KGGR [Chen et al.(2020)], MetaRE [Simon et al.(2022)], and BCR [An et al.(2024)], our work introduces a novel multi-modal cross-interaction mechanism tailored for prototype refinement. This mechanism is based on multiple modes of interaction between representative local features and word embeddings. Our study is the first to comprehensively explore the use of various word embeddings, including GloVe [Pennington et al.(2014)], BERT [Devlin et al.(2019)], and CLIP [Radford et al.(2021)], specifically for multi-label few-shot image classification. Several other studies address multi-label image classification with limited labeled data in different ways, such as leveraging unlabeled data to boost model performance [Li et al.(2021b)], using visual-language models like CLIP to classify images without annotations [Abdelfattah et al.(2023)], and training classification models in the presence of ambiguous data [Zhao et al.(2021), Zhao et al.(2023)]. Nevertheless, our focus is on multi-label few-shot image classification, where only a small number of precisely labeled images (e.g., 1 or 5) are available for each label.

## 2.4 Vision-Language Model

Vision-Language (VL) models use self-supervision strategies to achieve a strong alignment of textual and visual features, demonstrating an impressive ability to learn generic visual representations [Radford et al.(2021), Li et al.(2022), Alayrac et al.(2022), Jia et al.(2021), Yang et al.(2023), Geng et al.(2023), Wei et al.(2024)]. For instance, CLIP [Radford et al.(2021)] is trained with 400 million (image, text) pairs and shows strong zero-shot and transfer capabilities on over 30 visual tasks. With the rise of such powerful vision-language models, various studies have proposed approaches to efficiently adapt these models to downstream tasks, such as prompt learning [Zhou et al.(2022b), Zhou et al.(2022a)], and visual adaptation [Gao et al.(2024), Silva-Rodriguez et al.(2023)]. Although these methods demonstrate remarkable generalization to various zero-shot or few-shot image classification tasks, due to the way they were trained, they primarily focus on classifying each image into a single label and thus are not well-suited for multi-label settings [Sun et al.(2022)]. Recently, TaI-DPT [Guo et al.(2023)] was proposed to extend CLIP to multi-label image classification by introducing text-as-image prompting. Although it achieves a 59.2 mAP score on the COCO dataset without requiring any task-specific training images, it requires collecting image captions and localized narratives from datasets for training. In contrast, with a single training image per category, our LCM model can achieve a 60.3 mAP score with an extremely shallow backbone Conv-4-64 [Simon et al.(2022), An et al.(2024)] and off-the-shelf standard word embedding GloVe [Pennington et al.(2014)]. In the area of multi-label few-shot image classification, our advantage over VL
models comes from the ability to achieve promising performance with a small and thus highly efficient backbone network. Additionally, our approach is compatible with the backbone of VL models, offering the potential for even better performance.

# 3 Problem Setting

We consider the following multi-label few-shot image classification (ML-FSIC) setting: we are given a set of base labels $C_{base}$ and a set of novel labels $C_{novel}$, where $C_{base} \cap C_{novel} = \emptyset$. We also have two sets of labelled images: $\mathcal{E}_{base}$, containing images with labels from $C_{base}$, and $\mathcal{E}_{novel}$, containing images with labels from $C_{novel}$, where $\mathcal{E}_{base} \cap \mathcal{E}_{novel} = \emptyset$. The images from $\mathcal{E}_{base}$ are used for training the model, while those in $\mathcal{E}_{novel}$ are used for testing. The goal of ML-FSIC is to obtain a model that performs well for the labels in $C_{novel}$, when given only a few examples of images that have these labels. Models are trained and evaluated using so-called episodes. Each training episode involves a support set and a query set. The support set corresponds to the examples that are available for learning to predict the labels in $C_{base}$, while the query set is used to assess how well the system has accomplished this goal.

To construct the support set of a given training episode, for every label in $C_{base}$, we sample $K$ images from $\mathcal{E}_{base}$ which have that label, with $K \in \{1, 5\}$. These images are sampled without replacement, meaning that the total number of images in the support set is given by $K \times |C_{base}|$. The query set is sampled in the same way, except that we sample 4 images per label. Testing episodes are constructed similarly, but with labels from $C_{novel}$ and images from $\mathcal{E}_{novel}$ instead.

Note that there are some differences between our sampling strategy and the standard methodology that is adopted for (single-label) FSIC. In particular, FSIC models are usually evaluated using episodes that contain a sub-sample of $N$ classes, where the support set contains exactly $K$ examples of each class. In ML-FSIC, it is difficult to guarantee that the support set contains exactly $K$ examples of each label because most images have multiple labels. The sampling strategy from [Alfassy et al.(2019)] nonetheless tries to ensure this, but we argue that doing this has two important drawbacks. First, adhering to the constraint that each label must occur $K$ times often leads to a limited pool of image combinations available for constructing support sets. This limitation persists even when the requirement is somewhat relaxed. This implies that only a limited number of episodes can be sampled, which in turn makes training the model more challenging and results in less stable test outcomes. Second, the total number of images in the support set can vary substantially. For example, if one image contains all labels, then we may have a support set that only contains that one image when $K = 1$. In contrast, our strategy circumvents these drawbacks. We sample $K$ images for each category without repetition, guaranteeing that at least $K$ examples are accessible for each label. Additionally, the support set of each episode is consistently composed of the same number of images, namely $K \times N$. The idea of setting $N = |C_{base}|$ during training and $N = |C_{novel}|$ during testing, rather than selecting a sub-sample of $N$ classes, conforms to the strategy that was used by [Alfassy et al.(2019)] and [Chen et al.(2020)]. Furthermore, our approach allows for a larger pool of available episodes. In particular, we sample 200 testing episodes, whereas LaSO only samples 10 testing episodes [Alfassy et al.(2019)]. Moreover, our sampling strategy facilitates episode-based training, closely resembling the evaluation process employed for testing models.

# 4 Method

Our base model consists of two main components. The first component jointly represents label embeddings and visual features in a shared vector space, aiming to predict visual prototypes from the label embeddings. Since such prototypes are noisy, they are not used directly for final label predictions. Instead, this component is merely used to learn a joint representation of visual features and labels, as shown in Fig. 2. The second component is aimed at computing the final
<figure><img src="image_4.png" /><figcaption>Figure 2: Overview of our methodology. A joint embedding space is learned in which both labels and images are represented. A customized multi-modal cross-interaction strategy is proposed to calculate label prototypes using local features from the relevant images in the support set, along with word vectors that provide prior knowledge about the considered label. In our visual representation, the green solid line indicates the local feature flow for the base model, while the green dotted line represents the flow for the LCM model.</figcaption></figure>

prototypes by the proposed multi-modal cross-interaction strategy, which mainly consists of a channel-wise cross-attention module and a word embedding-based dynamic convolution module.

In addition to the base model, we also consider a variant, referred to as the LCM model, which involves an additional step between the two components. In particular, in this variant, we introduce Loss Change Measurement (LCM) to make a selection of local features. This selection is also based on the label embeddings from the first module. We then only aggregate the selected local features to construct the final prototypes. The LCM module thus makes a hard selection of the local features, aimed at removing those that are not relevant, before the attention mechanism is used to make a soft selection of the remaining local features.

It is noteworthy that we adopt distinct feature extractors for images and texts for two primary reasons. First, prevalent vision-language models such as CLIP [Radford et al.(2021)], BLIP [Li et al.(2022)] and Flamingo [Alayrac et al.(2022)] typically also employ separate extractors for images and texts due to their inherent modality distinctions. Second, in this way, our framework can accommodate a variety of image or text extractors, enhancing its adaptability. We now describe these different steps in more detail.

## 4.1 Joint Embedding of Visual Features and Labels

Given an input image $I$, we first use a feature extractor to obtain its local feature map $\mathbf{f}_{loc}^I \in \mathbb{R}^{n \times h \times w}$, where $n$ is the number of channels, $h$ is the height and $w$ is the width. In this paper, we use a fully convolutional network such as ResNet [He et al.(2016)] for this purpose. The global visual feature vector $\mathbf{f}_{glo}^I \in \mathbb{R}^n$ for image $I$ is obtained by averaging the local feature map across the height and width of the image. In particular let us write $f_{i,j,k}$ for the element at position $(i, j, k)$ in the tensor $\mathbf{f}_{loc}^I$. Then we define $\mathbf{f}_{glo}^I$ as the vector whose $i^{th}$ component is given by:

$$f_i^* = \frac{1}{h \cdot w} \sum_{1=j}^{h} \sum_{1=k}^{w} f_{i,j,k}. \qquad (1)$$
We use pre-trained word embeddings to represent the labels in $C_{base} \cup C_{novel}$. Let us write $\mathbf{w}_c$ for the word vector representing label $c$, and let $d_w$ be the dimensionality of the word vectors. To represent images and labels in the same vector space, we learn two linear transformations:

$$ \widehat{\mathbf{f}}_{glo}^{I} = \mathbf{A}_{visual} \widehat{\mathbf{f}}_{glo}^{I} \qquad \widehat{\mathbf{w}}_{c} = \mathbf{A}_{text} \mathbf{w}_{c}, \qquad (2) $$

where $\mathbf{A}_{visual} \in \mathbb{R}^{d_j \times n}$ is used to project the global feature vector for $I$ onto a space of $d_j$ dimensions. Similarly, $\mathbf{A}_{text} \in \mathbb{R}^{d_j \times d_w}$ is used to project the $d_w$-dimensional embedding of a label $c$ onto the same $d_j$-dimensional space. To ensure that the resulting image vectors $\widehat{\mathbf{f}}_{glo}^I$ and label representations $\widehat{\mathbf{w}}_c$ are semantically compatible, we propose to use the following loss, which treats the vectors $\widehat{\mathbf{w}}_c$ as prototypes to classify images:

$$ \mathcal{L}_{cm} = - \sum_{I \in S} \sum_{i=1}^{[C]} y_i^I \cdot \log \sigma(s_i^I) + (1 - y_i^I) \cdot \log(1 - \sigma(s_i^I)), \qquad (3) $$

where $S$ represents the set of images from the support set of the current training episode, $C = \{c_1, ..., c_{|C|}\}$ is the set of labels, $\sigma(\cdot)$ is the sigmoid function and $y_i^I$ represents the ground truth, i.e. $y_i^I = 1$ if image $I$ has label $c_i$ and $y_i^I = 0$ otherwise. Finally, we have

$$ s_i^I = \lambda \cos(\widehat{\mathbf{f}}_{glo}^I, \widehat{\mathbf{w}}_{c_i}), \qquad (4) $$

where $\cos$ denotes the cosine similarity and the scalar $\lambda$ is a hyper-parameter to address the fact that the cosine similarity is bounded between -1 and 1. Note that we will not be using the vectors $\widehat{\mathbf{w}}_c$ as prototypes for the final classification. We only use this classification loss to align the two modalities (word vectors and visual features). For this reason, we will refer to Eq. (3) as the Cross-Modality (CM) loss.

It should be noted that the proposed alignment process is conceptually straightforward, being based on two standard linear transformations. While more intricate methods may be conceived, we found this approach to achieve satisfactory results. Moreover, linear mappings have proven surprisingly capable for the problem of aligning embedding spaces, often outperforming deep learning based approaches [Mikolov et al. (2013a)].

## 4.2 Identifying Representative Local Features using Loss Change Measurement

In our base model, the global image embedding $\widehat{\mathbf{f}}_{glo}^I$ is obtained by averaging the local features across all regions of the image. However, not all these regions are equally important. Ideally, we only want to include those regions that contain the object being described by a given label, omitting local features that describe the image background. In this section, we consider the optional Loss Change Measurement (LCM) module to make a hard selection, aiming to remove those local features that are not relevant.

Let us write $(j, k)$ for the position of some local feature, i.e. the local feature corresponding to the region on the $j$th row and $k$th column. We now introduce a parameter $\rho_{j,k} \in [0, 1]$ to capture the importance of the local feature at position $(j, k)$ of a given image. To learn this parameter, we use the CM loss Eq. (3), with two changes. First, the remaining parameters (i.e. the matrices $\mathbf{A}_{visual}$ and $\mathbf{A}_{text}$) are frozen during this step. Second, when computing the global image embedding $\widehat{\mathbf{f}}_{glo}^I$, we first multiply each local feature by its corresponding importance $\rho_{j,k}$. In particular, the $i$th component of the vector $\widehat{\mathbf{f}}_{glo}^I$ is now defined as follows:

$$ f_i^* = \frac{1}{h \cdot w} \sum_{1=j}^{h} \sum_{1=k}^{w} \rho_{j,k} f_{i,j,k}. \qquad (5) $$
Essentially, the importance weight $\rho_{j,k}$ reflects how compatible the local feature at position $(j, k)$ is with the relevant label embeddings (i.e. the embeddings of the labels assigned to the image). After each iteration, the importance weights are normalised, to ensure that $\rho_{j,k} \in [0, 1]$. In particular, we update each $\rho_{j,k}$ as follows:

$$ \rho_{j,k} := \frac{\rho - \rho_{min}}{\rho_{max} - \rho_{min}}, \qquad (6) $$

with

$$ \rho_{min} = \min_{1 \le j \le h} (\min_{1 \le k \le w} (\rho_{j,k})), \quad \rho_{max} = \max_{1 \le j \le h} (\max_{1 \le k \le w} (\rho_{j,k})) \qquad (7) $$

If we end up with $\rho_{j,k} = 0$, this would mean that the local feature $(j, k)$ is permanently removed. To prevent this from happening, in such a case, we instead set the value of $\rho_{j,k}$ as the lowest non-zero value of the importance weight $\rho_{j',k'}$, across all local features.

Rather than directly using the weights $\rho_{j,k}$ to make the selection of local features, taking inspiration from [You et al.(2019), Yan et al.(2021c)], we measure the impact on the classification loss when a given feature is (partially) removed. Intuitively, if removing a given local feature has a large impact on the classification loss, it is likely to be representative of the object we are trying to model (i.e. the object described by the target label). In contrast, if the impact is minimal, it is more likely that this feature describes the image background or a different object. The impact of removing the local feature $(j, k)$ on the loss function is computed as follows:

$$ g(j, k) = |\mathcal{L}_{\Omega}^{(j,k)}(\rho_{j,k}) - \mathcal{L}_{\Omega}^{(j,k)}(0)|, \qquad (8) $$

where $\mathcal{L}_{\Omega}$ represents a classification loss with parameters $\Omega$. We again use the CM loss $\mathcal{L}_{cm}$ as the classification loss, where we assume that $f_{glo}^I$ is computed based on the importance weights, according to Eq. (5). The argument of $\mathcal{L}_{\Omega}^{(j,k)}$ represents the importance weight of the local feature at position $(j, k)$. Note that $\mathcal{L}_{\Omega}^{(j,k)}(0)$ thus represents the value of the loss function we obtain after setting $\rho_{j,k}$ to 0, i.e. if we completely disregard the region of the image at position $(j, k)$. In other words, $g(j, k)$ captures the difference in the loss function between (i) using $\rho_{j,k}$ as the importance of the local feature at position $(j, k)$ and (ii) setting this weight to 0. Intuitively, if $g(j, k)$ is above some threshold (for the value of $\rho_{j,k}$ upon convergence), we keep the local feature at position $(j, k)$; otherwise, we remove it. However, in practice, we do not use $g(j, k)$ directly. In particular, (i) we use an approximation of $g(j, k)$ to make the computation more efficient and (ii) rather than only evaluating $g(j, k)$ after the value of $\rho_{j,k}$ has converged, we use a momentum based strategy to improve stability. We now explain these two steps in more detail.

To simplify the implementation, we take advantage of the Taylor approximation of $\mathcal{L}_{\Omega}^{j,k}(x)$ at $x_0$:

$$ \mathcal{L}_{\Omega}^{j,k}(x) = \mathcal{L}_{\Omega}^{j,k}(x_0) + \frac{\mathcal{L}_{\Omega}^{j,k(1)}(x_0)}{1!}(x - x_0) + \dots + \frac{\mathcal{L}_{\Omega}^{j,k(n)}(x_0)}{n!}(x - x_0)^n + R_n(x), \qquad (9) $$

where we write $\mathcal{L}_{\Omega}^{j,k(i)}$ for the $i^{th}$ derivative of $\mathcal{L}_{\Omega}^{j,k}$. We find that $\mathcal{L}_{\Omega}^{j,k}(0)$ can be estimated as $\mathcal{L}_{\Omega}^{j,k}(\rho) - \rho\mathcal{L}_{\Omega}^{j,k(1)}(\rho)$ by setting $x = 0$ and $x_0 = \rho$. Therefore, Eq. 8 can be estimated as:

$$ g(j, k) \approx |\rho_{j,k} \mathcal{L}_{\Omega}^{j,k(1)}(\rho_{j,k})|. \qquad (10) $$

Note that the derivatives $\mathcal{L}_{\Omega}^{j,k(1)}(\rho_{j,k})$ can be computed in parallel for all local features, which makes using this approximation much faster than evaluating Eq. (8) for every local feature.

As already mentioned, the importance weights $\rho_{j,k}$ are iteratively learned by optimising the CM loss. These weights are initialized as $\rho_{j,k} = 1$. Rather than only using the converged
values of $\rho_{j,k}$, we use a momentum-based strategy to improve the stability of the estimation. In particular, drawing on the experience of [Tarvainen and Valpola(2017)], we compute the following estimate in iteration $i$:

$$f_i(j, k) = \alpha_i f_{i-1}(j, k) + (1 - \alpha_i) g_i(j, k), \quad \alpha_i = \min \left( 1 - \frac{1}{(i+1)}, \alpha \right), \qquad (11)$$

where we set $f_0(\rho) = 0$ and $\alpha = 0.95$ following [Tarvainen and Valpola(2017)]. The value of $g_i(j, k)$ is computed according to Eq. (10), using the importance weights $\rho_{j,k}$ that were obtained in the previous iteration.

After the final iteration $n$, we make the following decision: the local feature at position $(j, k)$ is retained if $\sigma(f_n(j, k)) \ge \theta$, where $\sigma(\cdot)$ denotes the sigmoid function. Otherwise, this local feature is removed.

## 4.3 Multi-modal Cross-interaction for Prototype Construction

We now explain how the final label prototypes are constructed. The multi-modal cross-interaction module, depicted in Fig. 2, allows two distinct types of information interaction between visual and text features. The first is to aggregate various visual patterns using word vectors as query within a customized cross-attention mechanism. The second is to employ word embeddings to generate convolutional kernels that interact with image features.

### 4.3.1 Channel-wise Cross-attention Guided by Word Embedding

As pointed out in [Chu et al.(2016)], features learned from different channels correspond to distinct visual patterns. Utilizing the word embedding-based prototypes from Sec. 4.1 as prior knowledge, we extract various visual patterns from local features. Concretely, consider a label $c$ and suppose this label has been assigned to $m$ images from the support set. We can derive a total of $l = h \cdot w \cdot m$ local feature vectors from these $m$ images. Let us write these local feature vectors as $\mathbf{u}_1, ..., \mathbf{u}_l$. These feature vectors are first projected into the joint embedding space: $\hat{\mathbf{u}}_i = \mathbf{A}_{visual} \mathbf{u}_i$. Subsequently, the feature vectors are split evenly along the channel dimension to serve as value and key inputs for $n_a$ attention heads within the attention mechanism [Vaswani et al.(2017)]. The corresponding label embedding $\hat{\mathbf{w}}_c$ provides the query for each attention head through a fully connected (FC) layer. Specifically, we have:

$$\mathbf{q}_c^j = \mathbf{Q}_j \hat{\mathbf{w}}_c, \quad \mathbf{k}_i^j = \mathbf{v}_i^j = \hat{\mathbf{u}}_i[(j-1) \cdot d_a : j \cdot d_a], \qquad (12)$$

$$\mathbf{p}_c^j = \sum_{i=1}^{l} \mu_i^j \mathbf{v}_i^j, \quad (\mu_1^j, ..., \mu_l^j) = \mathrm{Softmax} \left( \frac{\mathbf{q}_c^j \cdot \mathbf{k}_1^j}{\sqrt{d_a}}, ..., \frac{\mathbf{q}_c^j \cdot \mathbf{k}_l^j}{\sqrt{d_a}} \right) \qquad (13)$$

where $d_a = \frac{d_i}{n_a}$ is the dimensionality of the vectors $\mathbf{q}_c^j, \mathbf{k}_i^j$, and $\mathbf{v}_i^j$. The vector $\mathbf{p}_c^j$ represents the contribution of the jth attention head to the prototype of label $c$. As there are $n_a$ attention heads in total, the prototype of label $c$ in this cross-attention mechanism is given by:

$$\mathbf{p}_c^{att} = MLP(\mathbf{p}_c^1 \oplus \dots \oplus \mathbf{p}_c^{n_a}), \qquad (14)$$

where we write $\oplus$ for vector concatenation and MLP consists of two linear layers with GeLU activation and dropout.

### 4.3.2 Word Embedding-based Dynamic Convolution

Inspired by the use of dynamic convolution in [Sun et al.(2021)] to enhance object features, we propose generating convolutional kernels from word embeddings to interact with image features. Specifically, we begin by selecting the top $n_d$ local features from the joint embedding space,
denoted as $\hat{\mathbf{u}}_1, \dots, \hat{\mathbf{u}}_{n_d}$, based on their cosine similarities to $\hat{\mathbf{w}}_c$. We then utilize linear layers to produce two sets of convolutional kernels from $\hat{\mathbf{w}}_c$, i.e. $\Theta_c^1 \in \mathbb{R}^{1 \times 1 \times d_j \times d_c}$ and $\Theta_c^2 \in \mathbb{R}^{1 \times 1 \times d_c \times d_j}$. Finally, the operation of word embedding-based dynamic convolution is defined as :

$$ \mathbf{u}_i = \text{ReLU}(\text{Norm}(\Theta_c^1 \hat{\mathbf{u}}_i)), \qquad (15) $$

$$ \mathbf{p}_c^{dyn} = \frac{1}{n_d} \sum_{i=1}^{n_d} \text{ReLU}(\text{Norm}(\Theta_c^2 \mathbf{u}_i)), \qquad (16) $$

where ReLU and Norm denote the ReLU activate function and Layer Normalization [Sun et al.(2021)], respectively. Finally, the prototype of category c is obtained as:

$$ \mathbf{p}_c = \mathbf{p}_c^{att} + \mathbf{p}_c^{dyn}. \qquad (17) $$

### 4.3.3 Loss Function

To train our multi-modal cross-interaction module, we use the following loss function, which we refer to as the Query Loss:

$$ \mathcal{L}_{query} = - \sum_{I \in \mathcal{Q}} \sum_{i=1}^{|\mathcal{Q}|} y_i^I \cdot \log \sigma(q_i^I) + (1 - y_i^I) \cdot \log(1 - \sigma(q_i^I)), \qquad (18) $$

where $\mathcal{Q}$ represents the set of images from the query set of the current training episode. As before, $\mathcal{C}$ is the set of labels and $y_i^I$ represents the ground truth. The predictions $q_i$ are obtained as follows:

$$ q_i^I = \lambda \cos(\hat{\mathbf{f}}_{glo}^I, \mathbf{p}_{c_i}), \qquad (19) $$

with $\lambda$ the same scalar as in Eq. (4). Here we assume that the global feature vector $\hat{\mathbf{f}}_{glo}^I$ is computed as a standard average, according to Eq. (1), as we are not able to compute importance weights for query images. As an alternative to using the global feature vector $\hat{\mathbf{f}}_{glo}^I$ we could also compare $\mathbf{p}_{c_i}$ to the local image features. However, as we will see in our analysis in Section 5, this variant does not lead to clear improvements while having a considerably higher computational cost. Note that there are two key differences between $\mathcal{L}_{cm}$ and $\mathcal{L}_{query}$: (i) $\mathcal{L}_{cm}$ is trained using the support images while $\mathcal{L}_{query}$ is trained using the query images; and (ii) prototypes in $\mathcal{L}_{cm}$ are estimated from word vectors while prototypes in $\mathcal{L}_{query}$ are those obtained by aggregating local visual features.

## 4.4 Model Training and Evaluation

The model is trained by repeatedly sampling training episodes from $\mathcal{C}_{base}$, as explained in Section 3. Given a training episode with support set $\mathcal{S}$ and query set $\mathcal{Q}$, the model parameters are updated using the following loss:

$$ \mathcal{L}_{all} = \mathcal{L}_{cm} + \gamma \mathcal{L}_{query}, \qquad (20) $$

with $\gamma$ a hyperparameter to control the relative importance of both components.

After the model has been trained, it can be evaluated on test episodes as follows. For the base model, without the LCM step, we first construct the prototypes using the support set, as in Eq. (14). Note that we can do this without fine-tuning any model parameters. For each query image $I$, the probability that it has label $c_i$ is computed as $\sigma(q_i^I)$ with $q_i^I$ as defined in Eq. (19). For the variant with LCM, we first need to use the Loss Measurement Module to identify the most representative local features in the images in the support set. The labels are then predicted in the same way as for the base model.
# 5 Experiments

## 5.1 Experimental Setup

### 5.1.1 Datasets

We have conducted experiments on four datasets, starting with COCO [Lin et al.(2014)]. This dataset was already used for evaluating LaSO in [Alfassy et al.(2019)], where a split into 64 training labels and 16 test labels was proposed. However, as this split did not include a validation set, we split their 64 training labels into 12 labels for validation (cow, dining table, zebra, sandwich, bear, toaster, person, laptop, bed, teddy bear, baseball bat, skis) and 52 labels for training, while keeping the same 16 labels for testing. We follow [Alfassy et al.(2019)] to include images from the COCO 2014 training and validation sets. The images which do not contain any of the test and validation labels are used as the training set. Similarly, the validation set only contains images that do not contain any training or test labels. Second, we propose a new ML-FSIC dataset based on PASCAL VOC [Everingham et al.(2015)], which has 20 labels. To use as many images as possible, we select the following six labels for the novel set Cnovel: dog, sofa, cat, potted plant, tv monitor, sheep. The following six labels were selected for the validation split: boat, cow, train, aeroplane, bus, bird. The remaining eight labels are used for training. We use the images from the VOC 2007 training, validation, and test splits, as well as the VOC 2012 training and validation splits (noting that the labels of the VOC 2012 test split are not publicly available). Third, we propose a new ML-FSIC dataset based on NUS-WIDE [Chua et al.(2009a)]. NUS-WIDE contains 161789 images for training and 107859 images for testing, which were manually annotated with 81 concepts. In our setting, we use both training and testing images. We have selected 17 labels for the novel set (bear, cityscape, statue, bridge, sunset, sign, coral, fish, fire, cars, sand, vehicle, police, temple, horses, reflection, street) and 12 labels for the validation set (tower, tree, food, sun, town, beach, military, birds, flowers, house, lake, garden). The training set consists of the remaining labels. The complete NUS-WIDE dataset is no longer available from the official website<fnref n="1" />, as many image download links have expired. Therefore, we have instead collected the images from the NUS-WIDE dataset from a Kaggle website². Lastly, we introduce a new ML-FSIC dataset based on iMaterialist [Guo et al.(2019)]. Following a similar methodology as MoFSOD [Lee et al.(2022)], we extract 46 classes. Out of these, 16 labels (sock, jumpsuit, glove, scarf, coat, belt, zipper, cardigan, jacket, hood, vest, flower, tie, collar, ruffle, shirtBlouse) are selected for the novel set, while 12 labels (watch, lapel, shorts, bead, glasses, hat, sweater, tights, stockings, buckle, skirt, applique, rivet) are allocated to the validation set. The remaining labels constitute the training set. An overview of the labels contained in the four datasets can be found in the Appendix.

### 5.1.2 Methodology

Every model is trained for 200 epochs, with the first 10 epochs used as a warm-up. We used the Adam optimizer with an initial learning rate of 0.001. The variant with LCM requires a training step during the test phase, to learn the importance weights of the local features of the support images. We set the number of epochs for this training step to 20. For baseline methods that require a training or fine-tuning step, we set the number of epochs to 40 (noting that 20 epochs are not sufficient for some of these methods to converge). Note that our base model is used without any training or fine-tuning. During the test phase, we sample 200 test episodes. The results are averaged over all test episodes. We report results in terms of macro and micro metrics. The macro metric is evaluated by averaging per-class predictions while the micro metric is an overall measure for all images over all labels. Specifically, we use

<footnote n="1">¹https://lms.comp.nus.edu.sg/wp-content/uploads/2019/research/nuswide/NUS-WIDE.html
²https://www.kaggle.com/datasets/xinleili/nuswide?resource=downloadx</footnote>
macro/micro average precision (Ma-AP/Mi-AP) and macro/micro F1-measure (Ma-F1/Mi-F1), following earlier work in ML-IC [Zhu et al.(2017)]. Ma-AP is also denoted as mAP in many works [Alfassy et al.(2019), Cole et al.(2021), Durand et al.(2019)]. We assume that a label is predicted as positive if its estimated probability is greater than 0.5 following the standard setting [Zhu et al.(2017)].

### 5.1.3 Implementation Details

We use ResNet-50 [He et al.(2016)], ViT [Dosovitskiy et al.(2020)] and GoogleNet-v3 [Szegedy et al.(2016)] as feature extractors. The dimensionality of the joint embedding space was set as $d_j = 512$. As word embeddings, we mainly consider 300-dimensional GloVe [Pennington et al.(2014)] vectors<fnref n="3" /> as this is the most common choice in the literature for few-shot image classification [Li et al.(2020), Xing et al.(2019)] and multi-label image classification [Chen et al.(2020), Chen et al.(2019)]. In addition, we also experimented with the standard pretrained FastText and Word2Vec embeddings from an online repository⁴. Beyond standard word embeddings, we also experimented with embeddings that were extracted from BERT [Devlin et al.(2019)] using a number of different strategies. In particular, we use mirrorBERT [Liu et al.([n. d.])] and the biencoder model from [Gajbhiye et al.(2022)] (referred to as BERT<sub>biencoder</sub>), both of which are fine-tuned BERT models that can map any word or phrase onto a vector. In addition, we also use mirrorWiC [Liu et al.(2021)], and the strategies from [Li et al.(2021a)] (referred to as BERT<sub>avg</sub>) and [Li et al.(2023)] (referred to as BERT<sub>concn</sub>), all of which learn embeddings of words in context. To use these strategies to learn the embedding of a given word, we randomly select 500 sentences that mention the word from Wikipedia. Then we obtain the contextualised representations of these word mentions, using the aforementioned models, and finally compute the average of these vectors across the 500 sentences. Finally, we also consider word vectors extracted from CLIP [Radford et al.(2021)], using “a photo of a [CLASS]” as the prompt to guide its text encoder. Based on the validation split, for both datasets, the number of attention heads was set to 8, the $\theta$ in LCM was set to 0.65, and $\gamma$ was set to 1. We present our results for two variants: the base model and the LCM model. Note that an important advantage of the base model is its ability to classify novel categories without requiring any parameter adjustments. On the other hand, using the LCM module should lead to better results.

## 5.2 Baselines

We compare with LaSO [Alfassy et al.(2019)], KGGR [Chen et al.(2020)], NLC [Simon et al.(2022)], BCR [An et al.(2024)], which were designed for the ML-FSIC setting. To put the results in context, we also include comparisons with methods tailored for ML-IC. First, following LaSO [Alfassy et al.(2019)], we attach a standard multi-label classifier to a number of different feature extractors: ResNet-50, ResNet-101, ViT-Base [Dosovitskiy et al.(2020)] and ResNet-50 + ViT-Base. Second, we report results for the recent state-of-the-art CNN-RNN based method PLA [Yazici et al.(2020)], which, to the best of our knowledge, has not previously been evaluated in the ML-FSIC setting. Furthermore, we include the seminal meta-learning method MAML [Finn et al.(2017)] designed for FSIC. We also compare our results with our previous work, WGA [Yan et al.(2022)]. It is worth noting that the base model in this work is an enhanced version of WGA, incorporating additional multi-modal interaction operations, as detailed in Sec. 4.3. Since we did not have access to the source code of KGGR, NLC, and BCR, we only compared our method against the published results from the original paper, which followed the experimental setting from [Alfassy et al.(2019)]. For PLA, LaSO, and our method,

<footnote n="3">³Specifically, we use vectors that were trained from Wikipedia 2014 and Gigaword 5, which we obtained from the GloVe project page, at https://nlp.stanford.edu/projects/glove.
⁴https://developer.syn.co.in/tutorial/bot/oscova/pretrained-vectors.html</footnote>
<table><caption>Table 1: Overview of the main results on COCO, in terms of micro/macro AP and F1 scores (in %). The best results are highlighted in bold font. “-” indicates that no word embeddings are used.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Word-Emb.</th><th colspan="4">1-shot</th><th colspan="4">5-shot</th></tr><tr><th>Mi-AP</th><th>Mi-F1</th><th>Ma-AP</th><th>Ma-F1</th><th>Mi-AP</th><th>Mi-F1</th><th>Ma-AP</th><th>Ma-F1</th></tr></thead><tbody><tr><td>ResNet-50</td><td>-</td><td>11.31</td><td>12.51</td><td>19.67</td><td>13.27</td><td>19.12</td><td>16.08</td><td>24.17</td><td>17.22</td></tr><tr><td>ResNet-101</td><td>-</td><td>10.22</td><td>12.04</td><td>18.87</td><td>12.18</td><td>17.93</td><td>16.32</td><td>21.67</td><td>15.74</td></tr><tr><td>ViT</td><td>-</td><td>10.07</td><td>11.02</td><td>16.94</td><td>11.13</td><td>24.73</td><td>21.10</td><td>28.78</td><td>20.93</td></tr><tr><td>ResNet-50 + ViT</td><td>-</td><td>10.13</td><td>10.21</td><td>16.71</td><td>11.17</td><td>23.65</td><td>20.41</td><td>26.65</td><td>19.56</td></tr><tr><td>PLA</td><td>Self-learned</td><td>21.33</td><td>20.89</td><td>30.61</td><td>20.96</td><td>27.87</td><td>29.91</td><td>36.38</td><td>28.25</td></tr><tr><td>PLA (GloVe)</td><td>GloVe</td><td>23.02</td><td>22.15</td><td>30.13</td><td>20.37</td><td>27.11</td><td>28.48</td><td>35.13</td><td>27.47</td></tr><tr><td>LaSO</td><td>-</td><td>16.83</td><td>15.09</td><td>25.41</td><td>16.11</td><td>22.24</td><td>20.14</td><td>28.41</td><td>21.59</td></tr><tr><td>MAML</td><td>-</td><td>25.42</td><td>23.11</td><td>35.30</td><td>23.82</td><td>28.30</td><td>30.50</td><td>37.89</td><td>29.13</td></tr><tr><td>WGA</td><td>GloVe</td><td>35.30</td><td>34.21</td><td>42.84</td><td>28.91</td><td>38.97</td><td>38.10</td><td>44.91</td><td>33.12</td></tr><tr><td>Base model</td><td>GloVe</td><td>39.68</td><td>38.24</td><td>47.02</td><td>32.94</td><td>42.84</td><td>42.17</td><td>48.85</td><td>37.47</td></tr><tr><td>LCM model</td><td>GloVe</td><td>41.02</td><td>38.67</td><td>48.92</td><td>33.55</td><td>46.35</td><td>44.86</td><td>52.79</td><td>39.01</td></tr><tr><td>Base model</td><td>mirrorBERT</td><td>41.79</td><td>39.42</td><td>49.49</td><td>34.46</td><td>45.15</td><td>43.26</td><td>51.49</td><td>39.54</td></tr><tr><td>LCM model</td><td>mirrorBERT</td><td><strong>43.20</strong></td><td><strong>40.39</strong></td><td><strong>51.55</strong></td><td><strong>35.55</strong></td><td><strong>47.72</strong></td><td><strong>45.81</strong></td><td><strong>54.98</strong></td><td><strong>42.04</strong></td></tr></tbody></table>

we used ResNet-50 as the feature extractor in the main experiments. However, to ensure a fair comparison, we adopt the same feature extractors used in KGGR, LaSO, NLC, and BCR, for comparing our method in the LaSO setting.

## 5.3 Experimental Results

The experimental results for COCO are shown in Tab. 1 for the 1-shot and 5-shot settings. The results show that our proposed method outperforms the other methods by a substantial margin. From Tab. 1, it is evident that large models are prone to overfitting, noting that in terms of model size, we have: ResNet-50 + ViT-Base > ViT-Base > ResNet-101 > ResNet-50. This is not unexpected given the small number of labeled examples in the 1-shot setting. However, in the 5-shot setting, where more training examples are available, ViT-based models can outperform ResNet-based models. LaSO can improve the ResNet-50 baseline because of its data augmentation strategy. Somewhat surprisingly, PLA performs better than LaSO, which shows that its LSTM component can model label dependencies in a meaningful way. PLA also needs word embeddings, but in the original model, these embeddings are learned from the training data itself. For comparison, we also report results for a variant where these word vectors are initialized using GloVe vectors instead; the results are shown as PLA (GloVe). As can be seen, using pre-trained word vectors does not lead to a meaningful improvement over the original PLA model and can even degrade the performance on macro metrics, as shown in Tab. 1. Interestingly, MAML [Finn et al. (2017)], which was designed for the FSIC setting, performs better than the conventional ML-IC approaches, both in the 1-shot and 5-shot settings. The results also show that adding the LCM module improves the result, with the gains being somewhat clearer in the 5-shot setting compared to the 1-shot setting. Please refer to the Appendix for the experimental results for PASCAL VOC, NUS-WIDE, and iMaterialist. To provide a more thorough evaluation, we conducted additional experiments on both COCO and NUS-WIDE datasets using the data split proposed in FsPML [Zhao et al. (2023)]. As shown in Tab. 2 and Tab. 3, we can again see that our proposed method achieves the best performance.

To facilitate a comparison with KGGR, NLC, and BCR, we adopt the 1-shot and 5-shot settings from [Alfassy et al. (2019)]. For each method, we utilize the results reported in their respective original papers. As shown in Tab. 4, our method significantly outperforms existing approaches. Notably, our method achieves superior performance even with a simple and
<table><caption>Table 2: Overview of the main results on COCO, in terms of micro/macro AP and F1 scores (in %), following the same data split as in FsPML [Zhao et al.(2023)]. The best results are highlighted in bold font. “-” indicates that no word embeddings are used.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Word-Emb.</th><th colspan="4">1-shot</th><th colspan="4">5-shot</th></tr><tr><th>Mi-AP</th><th>Mi-F1</th><th>Ma-AP</th><th>Ma-F1</th><th>Mi-AP</th><th>Mi-F1</th><th>Ma-AP</th><th>Ma-F1</th></tr></thead><tbody><tr><td>ResNet-50</td><td>-</td><td>18.43</td><td>18.38</td><td>25.03</td><td>18.92</td><td>26.53</td><td>24.62</td><td>32.91</td><td>25.33</td></tr><tr><td>ResNet-101</td><td>-</td><td>15.85</td><td>17.93</td><td>24.61</td><td>18.77</td><td>23.79</td><td>23.38</td><td>28.37</td><td>22.98</td></tr><tr><td>ViT</td><td>-</td><td>14.53</td><td>16.29</td><td>21.00</td><td>17.36</td><td>30.46</td><td>26.77</td><td>35.83</td><td>26.41</td></tr><tr><td>ResNet-50 + ViT</td><td>-</td><td>14.44</td><td>15.18</td><td>22.56</td><td>18.28</td><td>31.02</td><td>26.58</td><td>34.61</td><td>26.22</td></tr><tr><td>PLA</td><td>Self-learned</td><td>29.34</td><td>29.01</td><td>37.68</td><td>28.30</td><td>35.84</td><td>37.17</td><td>43.75</td><td>36.71</td></tr><tr><td>PLA (GloVe)</td><td>GloVe</td><td>30.44</td><td>29.16</td><td>37.81</td><td>27.56</td><td>35.06</td><td>36.61</td><td>42.14</td><td>35.02</td></tr><tr><td>LaSO</td><td>-</td><td>24.75</td><td>22.21</td><td>33.02</td><td>23.81</td><td>29.59</td><td>27.62</td><td>35.47</td><td>29.35</td></tr><tr><td>MAML</td><td>-</td><td>33.02</td><td>29.37</td><td>42.03</td><td>29.84</td><td>35.91</td><td>37.72</td><td>44.78</td><td>36.05</td></tr><tr><td>WGA</td><td>GloVe</td><td>39.26</td><td>37.43</td><td>47.25</td><td>33.60</td><td>42.71</td><td>43.04</td><td>49.55</td><td>39.01</td></tr><tr><td>Base model</td><td>GloVe</td><td>46.44</td><td>44.82</td><td>52.86</td><td>39.21</td><td>49.68</td><td>48.76</td><td>55.21</td><td>44.75</td></tr><tr><td>LCM model</td><td>GloVe</td><td>47.37</td><td>45.17</td><td>54.78</td><td>40.57</td><td>52.99</td><td>50.76</td><td>58.72</td><td>45.92</td></tr><tr><td>Base model</td><td>mirrorBERT</td><td>48.81</td><td>45.60</td><td>55.94</td><td>40.83</td><td>51.08</td><td>49.43</td><td>57.90</td><td>45.43</td></tr><tr><td>LCM model</td><td>mirrorBERT</td><td><b>50.06</b></td><td><b>47.35</b></td><td><b>56.75</b></td><td><b>41.02</b></td><td><b>53.92</b></td><td><b>51.84</b></td><td><b>60.19</b></td><td><b>47.85</b></td></tr></tbody></table>

shallow Conv-4-64 backbone, whereas both LaSO and KGGR rely on the deeper GoogleNet-v3 backbone. Furthermore, when equipped with GoogleNet-v3, our approach establishes a new state-of-the-art.

## 5.4 Ablation Study

Here we analyze the importance of the main components of our model. All experiments in this section are conducted on the COCO dataset, employing the *ResNet-50* feature extractor and *GloVe* vectors, unless stated otherwise. The results are reported by default in terms of Ma-AP, commonly referred to as mAP [Alfassy et al.(2019), Cole et al.(2021), Durand et al.(2019)]. For additional ablation study results and qualitative analysis, please refer to the Appendix.

### 5.4.1 Effectiveness of Each Component

We ablate the effectiveness of each component in our method as shown in Tab. 5. When the component of the CM loss is removed, we only use $L_{query}$ for training. Note that the alignment between the word vectors and visual features then has to be learned indirectly, together with the parameters of the overall model. As indicated in the first row of Tab. 5, the absence of the CM loss significantly diminishes performance. The second to fourth rows illustrate that removing any multi-modal interaction module also leads to a reduction in performance. Conversely, the inclusion of LCM, as shown in the last row, markedly enhances performance, particularly in the 5-shot setting. One may wonder whether the CM loss is still needed when using a vision-and-language model such as CLIP, given that such models have text and visual spaces which are inherently aligned. We analyse this in Table 6, where we show results on the base model that were obtained with the CLIP (Vit-B/32CLIP) image encoder together with the word embeddings from the CLIP text encoder (WordCLIP). As can be seen, even in this case, using the CM loss leads to substantial performance gains.

### 5.4.2 Different Feature Extractors

For the main experiments, we have primarily used ResNet-50 [He et al.(2016)] as the feature extractor (backbone) for convenient comparison with other methods. Tab. 7 shows the impact of using different feature extractors. In addition to the standard ResNet-50, we also explore
<table><caption>Table 3: Overview of the main results on NUS-WIDE, in terms of micro/macro AP and F1 scores (in %), following the same data split as in FsPML [Zhao et al.(2023)]. The best results are highlighted in bold font. “-” indicates that no word embeddings are used.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Word-Emb.</th><th colspan="4">1-shot</th><th colspan="4">5-shot</th></tr><tr><th>Mi-AP</th><th>Mi-F1</th><th>Ma-AP</th><th>Ma-F1</th><th>Mi-AP</th><th>Mi-F1</th><th>Ma-AP</th><th>Ma-F1</th></tr></thead><tbody><tr><td>ResNet-50</td><td>-</td><td>18.76</td><td>23.98</td><td>22.04</td><td>22.91</td><td>24.53</td><td>28.21</td><td>27.53</td><td>28.69</td></tr><tr><td>ResNet-101</td><td>-</td><td>16.55</td><td>22.76</td><td>20.20</td><td>21.31</td><td>24.91</td><td>27.11</td><td>26.43</td><td>27.81</td></tr><tr><td>ViT</td><td>-</td><td>20.19</td><td>23.97</td><td>22.44</td><td>24.29</td><td>27.40</td><td>30.29</td><td>29.45</td><td>28.67</td></tr><tr><td>ResNet-50 + ViT</td><td>-</td><td>19.26</td><td>20.45</td><td>20.35</td><td>18.41</td><td>28.86</td><td>30.48</td><td>30.24</td><td>28.79</td></tr><tr><td>PLA</td><td>Self-learned</td><td>26.14</td><td>28.09</td><td>32.40</td><td>27.86</td><td>33.72</td><td>34.31</td><td>38.85</td><td>33.15</td></tr><tr><td>PLA (GloVe)</td><td>GloVe</td><td>25.53</td><td>27.42</td><td>32.01</td><td>26.54</td><td>32.92</td><td>32.69</td><td>38.43</td><td>33.57</td></tr><tr><td>LaSO</td><td>-</td><td>22.82</td><td>26.27</td><td>29.70</td><td>26.03</td><td>28.62</td><td>31.87</td><td>33.74</td><td>31.54</td></tr><tr><td>MAML</td><td>-</td><td>28.32</td><td>30.02</td><td>33.76</td><td>27.95</td><td>36.35</td><td>35.43</td><td>39.31</td><td>33.82</td></tr><tr><td>WGA</td><td>GloVe</td><td>36.56</td><td>34.39</td><td>41.70</td><td>33.35</td><td>40.87</td><td>39.04</td><td>45.34</td><td>40.62</td></tr><tr><td>Base model</td><td>GloVe</td><td>41.18</td><td>38.48</td><td>45.26</td><td>38.56</td><td>45.29</td><td>43.32</td><td>50.30</td><td>44.14</td></tr><tr><td>LCM model</td><td>GloVe</td><td>42.75</td><td>39.16</td><td>46.78</td><td>40.48</td><td>48.25</td><td>45.47</td><td>54.58</td><td>46.23</td></tr><tr><td>Base model</td><td>mirrorBERT</td><td>42.01</td><td>38.64</td><td>46.17</td><td>38.68</td><td>47.67</td><td>44.70</td><td>52.74</td><td>46.07</td></tr><tr><td>LCM model</td><td>mirrorBERT</td><td><strong>43.32</strong></td><td><strong>40.15</strong></td><td><strong>48.13</strong></td><td><strong>41.05</strong></td><td><strong>48.74</strong></td><td><strong>47.13</strong></td><td><strong>55.56</strong></td><td><strong>47.58</strong></td></tr></tbody></table>

<table><caption>Table 4: Experimental results (mAP in %) on COCO, when following the data and evaluation setting from LaSO [Alfassy et al.(2019)]. The best results are indicated by being highlighted in bold font; “-” indicates no word embeddings are used.</caption><thead><tr><th>Method</th><th>Backbone</th><th>Word Emb.</th><th>1-shot</th><th>5-shot</th></tr></thead><tbody><tr><td>LaSO [Alfassy et al.(2019)]</td><td>GoogleNet-v3</td><td>-</td><td>45.3</td><td>58.1</td></tr><tr><td>KGGR [Chen et al.(2020)]</td><td>GoogleNet-v3</td><td>GloVe</td><td>49.4</td><td>61.0</td></tr><tr><td>ProtoNets + NLC [Simon et al.(2022)]</td><td>Conv-4-64</td><td>-</td><td>50.2</td><td>60.4</td></tr><tr><td>RelationNets + NLC [Simon et al.(2022)]</td><td>Conv-4-64</td><td>-</td><td>53.3</td><td>60.8</td></tr><tr><td>LPN + NLC [Simon et al.(2022)]</td><td>Conv-4-64</td><td>-</td><td>56.8</td><td>64.8</td></tr><tr><td>BCR [An et al.(2024)]</td><td>GoogleNet-v3</td><td>-</td><td>61.5</td><td>70.1</td></tr><tr><td>Base model</td><td>Conv-4-64</td><td>GloVe</td><td>59.8</td><td>66.7</td></tr><tr><td>LCM model</td><td></td><td></td><td>60.3</td><td>68.3</td></tr><tr><td>Base model</td><td>GoogleNet-v3</td><td>GloVe</td><td>62.9</td><td>71.7</td></tr><tr><td>LCM model</td><td></td><td></td><td><strong>64.2</strong></td><td><strong>73.9</strong></td></tr></tbody></table>

backbones from foundation models. Specifically, we leverage ResNet-50 and ViT-B/32 from the CLIP[Radford et al.(2021)] image encoder (denoted as ResNet-50CLIP and ViT-B/32CLIP respectively). To adapt them to our approach, we remove the final attention layer and keep all grid features for ResNet-50CLIP and ViT-B/32CLIP respectively to receive 3D local features. The results presented in Tab. 7 demonstrate remarkable improvements by utilizing feature extractors from CLIP. This observation can be attributed to the extensive training of the CLIP model on a large volume of text-image data, resulting in its image encoder possessing a rich visual knowledge that aligns closely with textual features.

### 5.4.3 Robustness of Our Approach

To verify the robustness of our approach, we train our model 10 times with different seeds, and the margin of error is presented by the standard deviation across these diverse runs. The results in Tab. 7 show the robustness of our approach.
<table><caption>Table 5: Ablation study (in %) on components in our method. ‘Cro. Att.’ denotes the channel-wise cross-attention guided by word embedding. ‘Dyn. Conv.’ represents the word-embedding-based dynamic convolution. The best results are highlighted in bold.</caption><thead><tr><th>CM-loss</th><th>Cro. Att.</th><th>Dyn. Conv.</th><th>LCM</th><th>1-shot</th><th>5-shot</th><th>Notes</th></tr></thead><tbody><tr><td></td><td>✓</td><td>✓</td><td></td><td>44.01</td><td>46.59</td><td>Base model w/o CM-loss</td></tr><tr><td>✓</td><td></td><td>✓</td><td></td><td>44.92</td><td>46.85</td><td>Base model w/o Cro. Att.</td></tr><tr><td>✓</td><td>✓</td><td></td><td></td><td>45.67</td><td>47.03</td><td>Base model w/o Dyn. Conv.</td></tr><tr><td>✓</td><td>✓</td><td>✓</td><td></td><td>47.02</td><td>48.85</td><td>Base model</td></tr><tr><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td><strong>48.92</strong></td><td><strong>52.79</strong></td><td>LCM model</td></tr></tbody></table>

<table><caption>Table 6: Results (in %) on COCO for using “vision and language” model CLIP. The best results are highlighted in bold.</caption><thead><tr><th>Feature Ext.</th><th>Word Emb.</th><th>CM Loss</th><th>1-shot</th><th>5-shot</th></tr></thead><tbody><tr><td rowspan="2">ViT-B/32<sub>CLIP</sub></td><td rowspan="2">Word<sub>CLIP</sub></td><td></td><td>53.17</td><td>56.36</td></tr><tr><td>✓</td><td><strong>55.88</strong></td><td><strong>60.59</strong></td></tr></tbody></table>

### 5.4.4 Word Embeddings

Tab. 8 compares the results we obtained with different word embeddings: FastText [Bojanowski et al.(2017)], Word2Vec [Mikolov et al.(2013b)], GloVe [Pennington et al.(2014)], BERT [Devlin et al.(2019)], and word vectors from the CLIP text encoder. To obtain pre-trained label embeddings from BERT, we considered a number of different strategies, as explained in Sec. 5.1.3. In comparison to the image encoder of CLIP, as demonstrated in Tab. 7, the word vectors produced by the text encoder of CLIP (obtained as explained in Section 5.1.3) exhibit weaker performance than GloVe and some of the BERT strategies.

### 5.4.5 Number of Attention Heads

In Fig. 3a, we analyze the importance of using multiple attention heads. It is important to note that adjustments to the number of attention heads necessitate corresponding changes in the division of visual feature channels. However, despite these changes, the dimensionality of the final prototypes remains constant. The results in Fig. 3a show that with fewer than eight

<figure><img src="image_5.png" /><figcaption>(a) Results (in %) on COCO for varying numbers of attention heads when using the base model.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>(b) Results (in %) on COCO for different values of the $\theta$ threshold when using the LCM model.</figcaption></figure>

<caption>Figure 3: Ablation study for the number of attention heads and threshold $\theta$.</caption>
<table><caption>Table 7: Results (in %) on COCO for different feature extractors. The best results are highlighted in bold.</caption><thead><tr><th rowspan="2">Method</th><th rowspan="2">Feature Extractor</th><th colspan="2">Results</th><th colspan="2">Margin of errors</th></tr><tr><th>1-shot</th><th>5-shot</th><th>1-shot</th><th>5-shot</th></tr></thead><tbody><tr><td>Base model</td><td rowspan="2">ResNet-50</td><td>47.02</td><td>48.85</td><td>0.031</td><td>0.039</td></tr><tr><td>LCM model</td><td>48.92</td><td>52.79</td><td>0.038</td><td>0.041</td></tr><tr><td>Base model</td><td rowspan="2">ResNet-50<sub>CLIP</sub></td><td>53.83</td><td>56.31</td><td>0.054</td><td>0.038</td></tr><tr><td>LCM model</td><td>54.88</td><td>60.54</td><td>0.057</td><td>0.046</td></tr><tr><td>Base model</td><td rowspan="2">ViT-B/32<sub>CLIP</sub></td><td>58.70</td><td>61.53</td><td>0.055</td><td>0.048</td></tr><tr><td>LCM model</td><td><b>60.35</b></td><td><b>65.01</b></td><td>0.042</td><td>0.037</td></tr></tbody></table>

<table><caption>Table 8: Results (in %) on COCO for different word embeddings. The best results are highlighted in bold.</caption><thead><tr><th>Method</th><th>Word Emb.</th><th>1-shot</th><th>5-shot</th></tr></thead><tbody><tr><td>Base model</td><td rowspan="2">FastText</td><td>39.84</td><td>41.19</td></tr><tr><td>LCM model</td><td>40.67</td><td>41.55</td></tr><tr><td>Base model</td><td rowspan="2">Skip-Gram</td><td>42.68</td><td>43.11</td></tr><tr><td>LCM model</td><td>43.56</td><td>44.87</td></tr><tr><td>Base model</td><td rowspan="2">GloVe</td><td>47.02</td><td>48.85</td></tr><tr><td>LCM model</td><td>48.92</td><td>52.79</td></tr><tr><td>Base model</td><td rowspan="2">Word<sub>CLIP</sub></td><td>45.34</td><td>48.08</td></tr><tr><td>LCM model</td><td>46.67</td><td>49.32</td></tr><tr><td>Base model</td><td rowspan="2">BERT<sub>avg</sub></td><td>40.47</td><td>43.41</td></tr><tr><td>LCM model</td><td>42.86</td><td>44.60</td></tr><tr><td>Base model</td><td rowspan="2">mirrorWIC</td><td>42.54</td><td>46.16</td></tr><tr><td>LCM model</td><td>45.32</td><td>48.07</td></tr><tr><td>Base model</td><td rowspan="2">BERT<sub>biencoder</sub></td><td>47.79</td><td>49.05</td></tr><tr><td>LCM model</td><td>49.30</td><td>52.61</td></tr><tr><td>Base model</td><td rowspan="2">BERT<sub>concn</sub></td><td>49.29</td><td>51.25</td></tr><tr><td>LCM model</td><td>51.27</td><td>54.17</td></tr><tr><td>Base model</td><td rowspan="2">mirrorBERT</td><td>49.49</td><td>51.49</td></tr><tr><td>LCM model</td><td><b>51.55</b></td><td><b>54.98</b></td></tr></tbody></table>

attention heads, the performance is notably lower, while there is no obvious benefit to increasing the number of attention heads beyond eight.

### 5.4.6 LCM Threshold

The hyper-parameter $\theta$ controls the threshold for selecting local features based on the learned importance weights. Fig. 3b shows the impact of $\theta$ for our LCM model on the COCO dataset. Note that $\theta = 0.5$ corresponds to the base model since 0.5 is the minimum value that can be obtained after applying the sigmoid activation function to a non-negative input. As shown, the LCM model achieves the best performance in both 1-shot and 5-shot settings when $\theta = 0.65$.

## 5.5 Discussion

### 5.5.1 Potential for Zero-shot Classification

While our method was not designed for the zero-shot setting, the fact that word embeddings are aligned with visual features in the joint space means that they can be used as noisy prototypes,
<table><caption>Table 9: Zero-shot results (in %) on COCO of various different word embeddings. The best result is highlighted in bold.</caption><thead><tr><td>Word Embedding</td><td>mAP</td></tr></thead><tbody><tr><td>GloVe</td><td>18.79</td></tr><tr><td>WordCLIP</td><td>18.02</td></tr><tr><td>mirrorBERT</td><td>23.65</td></tr></tbody></table>

<figure><img src="image_7.png" /><figcaption>Figure 4: Visualization of attention weights among local features used to construct label prototypes for large objects. Darker areas indicate higher importance of the corresponding local features.</figcaption></figure>

which could in principle be used for zero-shot classification. We test the effectiveness of this approach for three types of word embeddings: the conventional GloVe embeddings [Pennington et al.(2014)], WordCLIP from the vision-language model CLIP [Radford et al.(2021)], and mirrorBERT [Liu et al.([n. d.])] from the BERT language model. As shown in Tab. 9, GloVe is slightly better than WordCLIP. Notably, mirrorBERT significantly outperforms both, with improvements of 4.86 and 5.63 mAP scores over GloVe and WordCLIP, respectively. These results suggest that embeddings from language models, when aligned with visual features, may be more effective for multi-label zero-shot learning (ML-ZSL) than other label embedding strategies. While these findings are clearly preliminary, we hope these insights can guide the selection of word embeddings in future ML-ZSL methods.

### 5.5.2 Potential Future Directions

There are two key limitations in our method. First, using off-the-shelf word embeddings for representing labels may not be the optimal strategy. For instance, some approaches for inferring word embeddings from language models are based on contextualised representations, meaning that suitable mentions of these words in a corpus have to be selected. Previous work has shown that selecting the most informative mentions can have an important impact on the quality of the resulting embeddings [Wang et al.(2022)]. One promising avenue might thus be to study strategies for selecting those mentions that are most relevant for vision tasks. Exploring the use of Large Language Models for obtaining more informative embeddings also remains as a promising avenue. Second, our multi-modal cross-interaction module includes two operations between word embeddings and visual features: channel-wised cross-attention guided by word embedding and word embedding-based dynamic convolution. There may be more effective multi-modal interaction operations to explore.

### 5.5.3 More Than Local Objects

As illustrated in Fig. 4, our model can effectively identify relevant image regions, even when the label pertains to the content of the whole image. This highlights that our method is not
constrained to local objects within an image.

# 6 Conclusion

We introduced a novel metric-based method for multi-label few-shot image classification. The main challenge, in this setting, comes from the fact that only a subset of the local features from support images correspond to the given label. To construct informative prototypes, it is essential to estimate which local features are most relevant. With limited support images, we argue that incorporating prior knowledge can effectively address this challenge. In particular, we have proposed to use word embeddings to obtain noisy prototypes, which are then processed by a multi-modal cross-interaction mechanism. This mechanism aims to construct the final prototypes by leveraging multiple interactions between visual features and word embeddings. To further alleviate the influence of irrelevant regions in the image, we have also proposed a loss change measurement module, which analyzes the gradient of the loss function to identify the most representative local features. Our model achieved substantially better results than existing models, both on COCO and newly proposed splits of the PASCAL VOC, NUS-WIDE, and iMaterialist datasets.

# Acknowledgments

This research was supported in part by the China Postdoctoral Science Foundation (2023M740079, GZC20230058). Zied Bouraoui is supported by ANR-22-CE23-0002 ERIANA. Shoaib Jameel is supported by the NVIDIA Academic Hardware Grant. Steven Schockaert was supported by EPSRC grant EP/V025961/1.

# References

[Abdelfattah et al. (2023)] Rabab Abdelfattah, Qing Guo, Xiaoguang Li, Xiaofeng Wang, and Song Wang. 2023. Cdul: Clip-driven unsupervised learning for multi-label image classification. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*. 1348–1357.

[Alayrac et al. (2022)] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. *Advances in Neural Information Processing Systems* 35 (2022), 23716–23736.

[Alfassy et al. (2019)] Amit Alfassy, Leonid Karlinsky, Amit Aides, Joseph Shtok, Sivan Harary, Rogerio Feris, Raja Giryes, and Alex M Bronstein. 2019. Laso: Label-set operations networks for multi-label few-shot learning. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 6548–6557.

[An et al. (2024)] Yuexuan An, Hui Xue, Xingyu Zhao, Ning Xu, Pengfei Fang, and Xin Geng. 2024. Leveraging Bilateral Correlations for Multi-Label Few-Shot Learning. *IEEE Transactions on Neural Networks and Learning Systems* (2024).

[Bojanowski et al. (2017)] Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching Word Vectors with Subword Information. *Transactions of the Association for Computational Linguistics* 5 (2017), 135–146. https://doi.org/10.1162/tacl_a_00051
[Chen et al.(2018)] Shang-Fu Chen, Yi-Chen Chen, Chih-Kuan Yeh, and Yu-Chiang Wang. 2018. Order-free rnn with visual attention for multi-label classification. In *Proceedings of the AAAI Conference on Artificial Intelligence*, Vol. 32.

[Chen et al.(2020)] Tianshui Chen, Liang Lin, Xiaolu Hui, Riquan Chen, and Hefeng Wu. 2020. Knowledge-guided multi-label few-shot learning for general image recognition. *IEEE Transactions on Pattern Analysis and Machine Intelligence* (2020).

[Chen et al.(2021)] Zhengyu Chen, Jixie Ge, Heshen Zhan, Siteng Huang, and Donglin Wang. 2021. Pareto Self-Supervised Training for Few-Shot Learning. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 13663–13672.

[Chen et al.(2019)] Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo. 2019. Multi-label image recognition with graph convolutional networks. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 5177–5186.

[Cheng et al.(2023)] Hao Cheng, Yufei Wang, Haoliang Li, Alex C Kot, and Bihan Wen. 2023. Disentangled feature representation for few-shot image classification. *IEEE Transactions on Neural Networks and Learning Systems* (2023).

[Chu et al.(2016)] Xiao Chu, Wanli Ouyang, Hongsheng Li, and Xiaogang Wang. 2016. Struc- tured feature learning for pose estimation. In *Proceedings of the IEEE conference on com- puter vision and pattern recognition*. 4715–4723.

[Chua et al.(2009a)] Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. 2009a. NUS-WIDE: a real-world web image database from National Uni- versity of Singapore. In *Proceedings of the 8th ACM International Conference on Image and Video Retrieval, CIVR 2009, Santorini Island, Greece, July 8-10, 2009*. ACM.

[Chua et al.(2009b)] Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. 2009b. Nus-wide: a real-world web image database from national univer- sity of Singapore. In *Proceedings of the ACM international conference on image and video retrieval*. 1–9.

[Cole et al.(2021)] Elijah Cole, Oisin Mac Aodha, Titouan Lorieul, Pietro Perona, Dan Morris, and Nebojsa Jojic. 2021. Multi-label learning from single positive labels. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 933–942.

[Dai et al.(2023)] Leichao Dai, Lin Feng, Xinglin Shang, and Han Su. 2023. Cross modal adaptive few-shot learning based on task dependence. *Chinese Journal of Electronics* 32, 1 (2023), 85–96.

[Devlin et al.(2019)] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In *Proceedings NAACL-HLT*.

[Dosovitskiy et al.(2020)] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weis- senborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. *arXiv preprint arXiv:2010.11929* (2020).

[Durand et al.(2019)] Thibaut Durand, Nazanin Mehrasa, and Greg Mori. 2019. Learning a deep convnet for multi-label classification with partial labels. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*. 647–657.
[Everingham et al.(2015)] Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. 2015. The pascal visual object classes challenge: A retrospective. *International journal of computer vision* 111, 1 (2015), 98–136.

[Finn et al.(2017)] Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In *International conference on machine learning*. 1126–1135.

[Gajbhiye et al.(2022)] Amit Gajbhiye, Luis Espinosa-Anke, and Steven Schockaert. 2022. Modelling commonsense properties using pre-trained bi-encoders. *arXiv preprint arXiv:2210.02771* (2022).

[Gao et al.(2024)] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. 2024. Clip-adapter: Better vision-language models with feature adapters. *International Journal of Computer Vision* 132, 2 (2024), 581–595.

[Geng et al.(2023)] Shijie Geng, Jianbo Yuan, Yu Tian, Yuxiao Chen, and Yongfeng Zhang. 2023. HiCLIP: Contrastive language-image pretraining with hierarchy-aware attention. *arXiv preprint arXiv:2303.02995* (2023).

[Guo et al.(2019)] Sheng Guo, Weilin Huang, Xiao Zhang, Prasanna Srikhanta, Yin Cui, Yuan Li, Hartwig Adam, Matthew R Scott, and Serge Belongie. 2019. The imaterialist fashion attribute dataset. In *Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops*. 0–0.

[Guo et al.(2023)] Zixian Guo, Bowen Dong, Zhilong Ji, Jinfeng Bai, Yiwen Guo, and Wangmeng Zuo. 2023. Texts as images in prompt tuning for multi-label image recognition. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2808–2817.

[He et al.(2016)] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In *Proc. CVPR*. 770–778.

[Jia et al.(2021)] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In *International conference on machine learning*. PMLR, 4904–4916.

[Kim et al.(2019)] Jongmin Kim, Taesup Kim, Sungwoong Kim, and Chang D Yoo. 2019. Edge-labeling graph neural network for few-shot learning. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*. 11–20.

[Kipf and Welling(2017)] Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. In *5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings*.

[Kuang and Li(2024)] Wenlan Kuang and Zhixin Li. 2024. Multi-label image classification with multi-layered multi-perspective dynamic semantic representation. *Machine Learning* 113, 6 (2024), 3443–3461.

[Lee et al.(2022)] Kibok Lee, Hao Yang, Satyaki Chakraborty, Zhaowei Cai, Gurumurthy Swaminathan, Avinash Ravichandran, and Onkar Dabeer. 2022. Rethinking few-shot object detection on a multi-domain benchmark. In *European Conference on Computer Vision*. 366–382.
[Li et al.(2020)] Aoxue Li, Weiran Huang, Xu Lan, Jiashi Feng, Zhenguo Li, and Liwei Wang. 2020. Boosting few-shot learning with adaptive margin loss. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*. 12576-12584.

[Li et al.(2022)] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In *International Conference on Machine Learning*. 12888-12900.

[Li et al.(2021a)] Na Li, Zied Bouraoui, José Camacho-Collados, Luis Espinosa Anke, Qing Gu, and Steven Schockaert. 2021a. Modelling General Properties of Nouns by Selectively Averaging Contextualised Embeddings. In *Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021*. 3850-3856.

[Li et al.(2023)] Na Li, Hanane Kteich, Zied Bouraoui, and Steven Schockaert. 2023. Distilling semantic concept embeddings from contrastively fine-tuned language models. *arXiv preprint arXiv:2305.09785* (2023).

[Li et al.(2019)] Wenbin Li, Lei Wang, Jinglin Xu, Jing Huo, Yang Gao, and Jiebo Luo. 2019. Revisiting local descriptor based image-to-class measure for few-shot learning. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 7260-7268.

[Li et al.(2021b)] Zhixin Li, Lan Lin, Canlong Zhang, Huifang Ma, Weizhong Zhao, and Zhiping Shi. 2021b. A semi-supervised learning approach based on adaptive weighted fusion for automatic image annotation. *ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)* 17, 1 (2021), 1-23.

[Li et al.(2021c)] Zeqian Li, Michael Mozer, and Jacob Whitehill. 2021c. Compositional embeddings for multi-label one-shot learning. In *Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision*. 296-304.

[Li et al.(2017)] Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. 2017. Meta-sgd: Learning to learn quickly for few-shot learning. *arXiv preprint arXiv:1707.09835* (2017).

[Lin et al.(2014)] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In *European conference on computer vision*. Springer, 740-755.

[Liu et al.([n. d.])] Fangyu Liu, Ivan Vulic, Anna Korhonen, and Nigel Collier. [n. d.]. Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders. In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021*. 1442-1459.

[Liu et al.(2021)] Qianchu Liu, Fangyu Liu, Nigel Collier, Anna Korhonen, and Ivan Vulic. 2021. MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Models. In *Proceedings of the 25th Conference on Computational Natural Language Learning, CoNLL 2021*. 562-574.

[Mikolov et al.(2013a)] Tomás Mikolov, Quoc V. Le, and Ilya Sutskever. 2013a. Exploiting Similarities among Languages for Machine Translation. CoRR abs/1309.4168 (2013). arXiv:1309.4168 http://arxiv.org/abs/1309.4168

[Mikolov et al.(2013b)] Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013b. Linguistic Regularities in Continuous Space Word Representations. In *Proceedings NAACL-HLT*. 746-751.
[Nichol et al.(2018)] Alex Nichol, Joshua Achiam, and John Schulman. 2018. On first-order meta-learning algorithms. *arXiv preprint arXiv:1803.02999* (2018).

[Pennington et al.(2014)] Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In *Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)*. 1532–1543.

[Radford et al.(2021)] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In *International conference on machine learning*. 8748–8763.

[Ravi and Larochelle(2017)] Sachin Ravi and Hugo Larochelle. 2017. Optimization as a model for few-shot learning. In *International conference on learning representations*.

[Rusu et al.(2019)] Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. 2019. Meta-Learning with Latent Embedding Optimization. In *7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019*.

[Satorras and Estrach(2018)] Victor Garcia Satorras and Joan Bruna Estrach. 2018. Few-shot learning with graph neural networks. In *International conference on learning representations*.

[Silva-Rodriguez et al.(2023)] Julio Silva-Rodriguez, Sina Hajimiri, Ismail Ben Ayed, and Jose Dolz. 2023. A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models. *arXiv preprint arXiv:2312.12730* (2023).

[Simon et al.(2022)] Christian Simon, Piotr Koniusz, and Mehrtash Harandi. 2022. Meta-learning for multi-label few-shot classification. In *Proceedings of the IEEE/CVF winter conference on applications of computer vision*. 3951–3960.

[Snell et al.(2017)] Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. *Advances in neural information processing systems* 30 (2017).

[Sun et al.(2021)] Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, et al. 2021. Sparse r-cnn: End-to-end object detection with learnable proposals. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*. 14454–14463.

[Sun et al.(2022)] Ximeng Sun, Ping Hu, and Kate Saenko. 2022. Dualcoop: Fast adaptation to multi-label recognition with limited annotations. *Advances in Neural Information Processing Systems* 35 (2022), 30569–30582.

[Sung et al.(2018)] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. 2018. Learning to compare: Relation network for few-shot learning. In *Proceedings of the IEEE conference on computer vision and pattern recognition*. 1199–1208.

[Szegedy et al.(2016)] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In *Proceedings of the IEEE conference on computer vision and pattern recognition*. 2818–2826.

[Tarvainen and Valpola(2017)] Antti Tarvainen and Harri Valpola. 2017. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. *Advances in neural information processing systems* 30 (2017).
[Tsoumakas and Katakis (2007)] Grigorios Tsoumakas and Ioannis Katakis. 2007. Multi-label classification: An overview. *International Journal of Data Warehousing and Mining* (IJDWM) 3, 3 (2007), 1–13.

[Vaswani et al. (2017)] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In *Advances in neural information processing systems*. 5998–6008.

[Vinyals et al. (2016)] Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. 2016. Matching Networks for One Shot Learning. In *Proc. NIPS*. 3630–3638.

[Wang et al. (2016)] Jiang Wang, Yi Yang, Junhua Mao, Zhiheng Huang, Chang Huang, and Wei Xu. 2016. Cnn-rnn: A unified framework for multi-label image classification. In *Proceedings of the IEEE conference on computer vision and pattern recognition*. 2285–2294.

[Wang et al. (2022)] Yixiao Wang, Zied Bouraoui, Luis Espinosa Anke, and Steven Schockaert. 2022. Sentence Selection Strategies for Distilling Word Embeddings from BERT. In *Proceedings of the Thirteenth Language Resources and Evaluation Conference, LREC 2022, Marseille, France, 20-25 June 2022*. European Language Resources Association, 2591–2600.

[Wang et al. (2020)] Ya Wang, Dongliang He, Fu Li, Xiang Long, Zhichao Zhou, Jinwen Ma, and Shilei Wen. 2020. Multi-label classification with label graph superimposing. In *Proceedings of the AAAI Conference on Artificial Intelligence*, Vol. 34. 12265–12272.

[Wang et al. (2017)] Zhouxia Wang, Tianshui Chen, Guanbin Li, Ruijia Xu, and Liang Lin. 2017. Multi-label image recognition by recurrently discovering attentional regions. In *Proceedings of the IEEE international conference on computer vision*. 464–472.

[Wei et al. (2024)] Zihao Wei, Zixuan Pan, and Andrew Owens. 2024. Efficient Vision-Language Pre-training by Cluster Masking. *arXiv preprint arXiv:2405.08815* (2024).

[Xing et al. (2019)] Chen Xing, Negar Rostamzadeh, Boris Oreshkin, and Pedro O O Pinheiro. 2019. Adaptive cross-modal few-shot learning. *Advances in Neural Information Processing Systems* 32 (2019).

[Yan et al. (2021a)] Kun Yan, Zied Bouraoui, Ping Wang, Shoaib Jameel, and Steven Schockaert. 2021a. Aligning visual prototypes with BERT embeddings for few-shot learning. In *Proceedings of the 2021 International Conference on Multimedia Retrieval*. 367–375.

[Yan et al. (2021b)] Kun Yan, Zied Bouraoui, Ping Wang, Shoaib Jameel, and Steven Schockaert. 2021b. Few-shot image classification with multi-facet prototypes. In *ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 1740–1744.

[Yan et al. (2021c)] Kun Yan, Lingbo Liu, Jun Hou, and Ping Wang. 2021c. Representative Local Feature Mining for Few-Shot Learning. In *ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 1730–1734.

[Yan et al. (2022)] Kun Yan, Chenbin Zhang, Jun Hou, Ping Wang, Zied Bouraoui, Shoaib Jameel, and Steven Schockaert. 2022. Inferring prototypes for multi-label few-shot image classification with word vector guided attention. In *Proceedings of the AAAI Conference on Artificial Intelligence*, Vol. 36. 2991–2999.
[Yang et al.(2023)] Kaicheng Yang, Jiankang Deng, Xiang An, Jiawei Li, Ziyong Feng, Jia Guo, Jing Yang, and Tongliang Liu. 2023. Alip: Adaptive language-image pre-training with synthetic caption. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*. 2922–2931.

[Yazici et al.(2020)] Vacit Oguz Yazici, Abel Gonzalez-Garcia, Arnau Ramisa, Bartlomiej Twardowski, and Joost van de Weijer. 2020. Orderless recurrent models for multi-label classification. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 13440–13449.

[Ye et al.(2020)] Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. 2020. Few-shot learning via embedding adaptation with set-to-set functions. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*. 8808–8817.

[You et al.(2020)] Renchun You, Zhiyao Guo, Lei Cui, Xiang Long, Yingze Bao, and Shilei Wen. 2020. Cross-modality attention with semantic graph embedding for multi-label classification. In *Proceedings of the AAAI Conference on Artificial Intelligence*, Vol. 34. 12709–12716.

[You et al.(2019)] Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang. 2019. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. *Advances in Neural Information Processing Systems* 32 (2019).

[Yuan et al.(2023)] Jin Yuan, Shikai Chen, Yao Zhang, Zhongchao Shi, Xin Geng, Jianping Fan, and Yong Rui. 2023. Graph attention transformer network for multi-label image classification. *ACM Transactions on Multimedia Computing, Communications and Applications* 19, 4 (2023), 1–16.

[Zhang et al.(2022)] Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. 2022. Deepemd: Differentiable earth mover's distance for few-shot learning. *IEEE Transactions on Pattern Analysis and Machine Intelligence* (2022).

[Zhang et al.(2024)] Ruru Zhang, E Haihong, Meina Song, and Xun Cao. 2024. FSCIL-EACA: Few-Shot Class-Incremental Learning Network Based on Embedding Augmentation and Classifier Adaptation for Image Classification. *Chinese Journal of Electronics* 33, 1 (2024), 139–152.

[Zhao et al.(2021)] Yunfeng Zhao, Guoxian Yu, Lei Liu, Zhongmin Yan, Carlotta Domeniconi, and Lizhen Cui. 2021. Few-shot partial multi-label learning. In *2021 IEEE International Conference on Data Mining (ICDM)*. IEEE, 926–935.

[Zhao et al.(2023)] Yunfeng Zhao, Guoxian Yu, Lei Liu, Zhongmin Yan, Carlotta Domeniconi, Xiayan Zhang, and Lizhen Cui. 2023. Few-shot partial multi-label learning via prototype rectification. *Knowledge and Information Systems* 65, 4 (2023), 1851–1880.

[Zhou et al.(2022a)] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022a. Conditional prompt learning for vision-language models. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*. 16816–16825.

[Zhou et al.(2022b)] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022b. Learning to prompt for vision-language models. *International Journal of Computer Vision* 130, 9 (2022), 2337–2348.

[Zhou et al.(2023a)] Wei Zhou, Peng Dou, Tao Su, Haifeng Hu, and Zhijie Zheng. 2023a. Feature learning network with transformer for multi-label image classification. *Pattern Recognition* 136 (2023), 109203.
[Zhou et al.(2023b)] Wei Zhou, Yanke Hou, Dihu Chen, Haifeng Hu, and Tao Su. 2023b. Attention-augmented memory network for image multi-label classification. *ACM Transactions on Multimedia Computing, Communications and Applications* 19, 3 (2023), 1-24.

[Zhou et al.(2023c)] Wei Zhou, Zhiwu Xia, Peng Dou, Tao Su, and Haifeng Hu. 2023c. Double attention based on graph attention network for image multi-label classification. *ACM Transactions on Multimedia Computing, Communications and Applications* 19, 1 (2023), 1-23.

[Zhu et al.(2017)] Feng Zhu, Hongsheng Li, Wanli Ouyang, Nenghai Yu, and Xiaogang Wang. 2017. Learning spatial regularization with image-level supervisions for multi-label image classification. In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*. 5513-5522.

[Zhu and Wu(2021)] Ke Zhu and Jianxin Wu. 2021. Residual attention: A simple but effective method for multi-label recognition. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*. 184-193.
# A Experiments

<table><caption>Table 10: Labels covered by the training, validation, and novel sets for the four considered multi-label few-shot image classification datasets.</caption><thead><tr><th colspan="5">ML-FSIC Datasets</th></tr><tr><th></th><th>COCO</th><th>PASCAL VOC</th><th>NUS-WIDE</th><th>iMaterialist</th></tr></thead><tbody><tr><td>Training Set:</td><td>{ hair drier, clock, wine glass, book, cake, tie, motorcycle, sheep, bottle, giraffe, cell phone, suitcase, remote, bench, mouse, carrot, banana, train, sports ball, toothbrush, fire hydrant, airplane, tv, bus, refrigerator, couch, knife, toilet, elephant, truck, parking meter, car, potted plant, kite, skateboard, orange, horse, cat, tennis racket, bowl, scissors, baseball glove, apple, traffic light, handbag, donut, dog, hot dog, oven, umbrella, sink, pizza }</td><td>{ person, car, chair, bottle, horse, bicycle, motorbike, diningtable }</td><td>{ animal, whales, railroad, clouds, person, leaf, elk, frost, map, snow, waterfall, toy, wedding, protest, book, running, cat, airport, harbor, valley, soccer, train, sky, moon, swimmers, window, dog, boats, buildings, plants, road, tattoo, nighttime, castle, dancing, plane, zebra, tiger, earthquake, sports, grass, water, glacier, rainbow, fox, rocks, cow, ocean, surf, mountain, computer, flags }</td><td>{epaulette, fringe, cape, top-t-shirt_sweatshirt, sequin, bag-wallet, bow, leg warmer, shoe, sleeve, pants, dress, headband_head-covering_hair-accessory, tassel, ribbon, pocket, umbrella, neckline }</td></tr><tr><td>Validation Set:</td><td>{ cow, dining table, zebra, sandwich, bear, toaster, person, laptop, bed, teddy bear, baseball bat, skis }</td><td>{ boat, cow, train, aeroplane, bus, bird }</td><td>{ tower, tree, food, sun, town, beach, military, birds, flowers, house, lake, garden }</td><td>{watch, lapel, shorts, bead, glasses, hat, sweater, tights.stockings, buckle, skirt, applique, rivet }</td></tr><tr><td>Novel Set:</td><td>{ bicycle, boat, stop sign, bird, backpack, frisbee, snowboard, surfboard, cup, fork, spoon, broccoli, chair, keyboard, microwave, vase }</td><td>{ dog, sofa, cat, potted plant, tv monitor, sheep }</td><td>{ bear, statue, bridge, sunset, sign, coral, fish, fire, cars, sand, vehicle, cityscape, police, temple, horses, reflection, street }</td><td>{sock, jumpsuit, glove, scarf, coat, belt, zipper, cardi-gan, jacket, hood, vest, flower, tie, collar, ruffle, shirt-blouse }</td></tr></tbody></table>

## A.1 Datasets

An overview of the labels contained in four datasets, including COCO [Lin et al.(2014)], PASCAL VOC [Everingham et al.(2015)], NUS-WIDE [Chua et al.(2009a)], and iMaterialist [Guo et al.(2019)], can be found in Tab. 10.
<table><caption>Table 11: Overview of the main results on PASCAL VOC, in terms of micro/macro AP and F1 scores (in %). The best results are highlighted in bold font. “-” indicates that no word embeddings are used.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Word-Emb.</th><th colspan="4">1-shot</th><th colspan="4">5-shot</th></tr><tr><th>Mi-AP</th><th>Mi-F1</th><th>Ma-AP</th><th>Ma-F1</th><th>Mi-AP</th><th>Mi-F1</th><th>Ma-AP</th><th>Ma-F1</th></tr></thead><tbody><tr><td>ResNet-50</td><td>-</td><td>16.57</td><td>20.77</td><td>27.24</td><td>21.92</td><td>24.47</td><td>28.53</td><td>35.06</td><td>29.43</td></tr><tr><td>ResNet-101</td><td>-</td><td>19.23</td><td>23.86</td><td>30.11</td><td>22.17</td><td>27.07</td><td>29.37</td><td>39.01</td><td>31.26</td></tr><tr><td>ViT</td><td>-</td><td>16.69</td><td>17.76</td><td>27.72</td><td>17.15</td><td>31.70</td><td>37.35</td><td>40.13</td><td>36.29</td></tr><tr><td>ResNet-50 + ViT</td><td>-</td><td>18.71</td><td>13.70</td><td>28.76</td><td>12.86</td><td>34.69</td><td>35.83</td><td>42.73</td><td>33.99</td></tr><tr><td>PLA</td><td>Self-learned</td><td>41.98</td><td>36.81</td><td>47.29</td><td>37.83</td><td>48.09</td><td>42.01</td><td>55.54</td><td>45.40</td></tr><tr><td>PLA (GloVe)</td><td>GloVe</td><td>41.31</td><td>35.84</td><td>46.54</td><td>36.55</td><td>47.50</td><td>40.60</td><td>55.26</td><td>45.33</td></tr><tr><td>LaSO</td><td>-</td><td>22.12</td><td>27.02</td><td>32.50</td><td>25.20</td><td>30.61</td><td>33.64</td><td>41.89</td><td>34.96</td></tr><tr><td>MAML</td><td>-</td><td>40.21</td><td>35.11</td><td>44.15</td><td>36.08</td><td>45.95</td><td>39.26</td><td>50.03</td><td>42.74</td></tr><tr><td>WGA</td><td>GloVe</td><td>46.28</td><td>40.19</td><td>53.26</td><td>43.35</td><td>51.10</td><td>45.03</td><td>58.27</td><td>48.20</td></tr><tr><td>Base model</td><td>GloVe</td><td>49.92</td><td>44.00</td><td>56.59</td><td>47.05</td><td>54.35</td><td>48.21</td><td>61.48</td><td>51.21</td></tr><tr><td>LCM model</td><td>GloVe</td><td>50.98</td><td>44.18</td><td>58.03</td><td>47.31</td><td>56.74</td><td>49.08</td><td>65.41</td><td>52.00</td></tr><tr><td>Base model</td><td>mirrorBERT</td><td>51.19</td><td>44.37</td><td>58.43</td><td>47.96</td><td>55.36</td><td>49.37</td><td>63.33</td><td>51.55</td></tr><tr><td>LCM model</td><td>mirrorBERT</td><td><b>53.30</b></td><td><b>46.05</b></td><td><b>59.13</b></td><td><b>49.65</b></td><td><b>58.30</b></td><td><b>51.19</b></td><td><b>66.78</b></td><td><b>54.01</b></td></tr></tbody></table>

## A.2 Experimental Results

The experimental results for PASCAL VOC, NUS-WIDE, and iMaterialist are shown in Tab. 11, Tab. 12, and Tab. 13, respectively. Our method consistently achieves the best performance.

## A.3 Ablation Study

### A.3.1 Importance of the Attention Mechanism

To evaluate the effect of the proposed attention mechanism, in Tab. 14, we report results for a variant that relies on a simpler mechanism for generating prototypes (shown as *Simple attention*). In particular, in this variant, we generate the $p_c^{att}$ of a label c by taking a weighted average of the global features of the support images which have that label. The weights are obtained by computing the cosine similarity between the vectors $\hat{w}_c$ and the feature vectors $\hat{f}_{glo}^I$, multiplying these cosine similarities with the scalar $\lambda$, and feeding the resulting values to a softmax layer. As shown in Tab. 14, the performance degradation in this simplified setting is clearly evident. We also present results for a variant employing the standard attention mechanism, denoted as *Standard Attention*. In this variant, the inputs of *value* and *key* are generated by processing visual features through fully connected layers, rather than splitting them evenly along the channel dimension. Additionally, we report results when local features are used instead of global features within our cross-attention mechanism (*Attention with global features*). Finally, we experimented with a variant that learns visual prototypes in the same way as in KGGR, where the weights of local features are generated by a low-rank bilinear pooling method followed by a fully connected layer with softmax activation (*Low-rank bilinear pooling*). The results in Tab. 14 show that all of these variants result in lower performance.

### A.3.2 Selection of Local Features

There are a number of ways in which the LCM module can be simplified. First, we could use the importance weights $\rho_{j,k}$ directly, i.e. we could select those local features whose $\rho_{j,k}$-value is above a certain threshold. Next, the implementation of loss change measurement itself could also be simplified. For instance, we could evaluate the loss change $g(\cdot)$ directly using Eq. 8. A final possibility we consider is to use the Taylor approximation, i.e. to use Eq. 10, but without the
<table><caption>Table 12: Overview of the main results on NUS-WIDE, in terms of micro/macro AP and F1 scores (in %). The best results are highlighted in bold font. “-” indicates that no word embeddings are used.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Word-Emb.</th><th colspan="4">1-shot</th><th colspan="4">5-shot</th></tr><tr><th>Mi-AP</th><th>Mi-F1</th><th>Ma-AP</th><th>Ma-F1</th><th>Mi-AP</th><th>Mi-F1</th><th>Ma-AP</th><th>Ma-F1</th></tr></thead><tbody><tr><td>ResNet-50</td><td>-</td><td>16.46</td><td>21.49</td><td>20.78</td><td>21.77</td><td>21.71</td><td>25.33</td><td>24.89</td><td>26.02</td></tr><tr><td>ResNet-101</td><td>-</td><td>15.10</td><td>20.27</td><td>17.90</td><td>20.18</td><td>22.60</td><td>25.82</td><td>24.81</td><td>26.25</td></tr><tr><td>ViT</td><td>-</td><td>17.08</td><td>20.76</td><td>19.94</td><td>21.97</td><td>24.85</td><td>28.58</td><td>27.04</td><td>26.57</td></tr><tr><td>ResNet-50 + ViT</td><td>-</td><td>16.05</td><td>18.92</td><td>18.10</td><td>15.13</td><td>26.32</td><td>27.84</td><td>29.09</td><td>25.15</td></tr><tr><td>PLA</td><td>Self-learned</td><td>22.31</td><td>23.92</td><td>27.85</td><td>24.63</td><td>30.32</td><td>31.59</td><td>35.24</td><td>30.33</td></tr><tr><td>PLA (GloVe)</td><td>GloVe</td><td>22.47</td><td>23.86</td><td>27.69</td><td>24.29</td><td>29.78</td><td>30.47</td><td>36.01</td><td>31.04</td></tr><tr><td>LaSO</td><td>-</td><td>20.27</td><td>23.17</td><td>25.95</td><td>22.33</td><td>25.35</td><td>28.91</td><td>29.63</td><td>27.77</td></tr><tr><td>MAML</td><td>-</td><td>25.32</td><td>26.94</td><td>30.09</td><td>24.82</td><td>32.95</td><td>32.52</td><td>36.17</td><td>30.50</td></tr><tr><td>WGA</td><td>GloVe</td><td>32.91</td><td>31.53</td><td>38.12</td><td>30.71</td><td>36.81</td><td>35.16</td><td>41.25</td><td>35.63</td></tr><tr><td>Base model</td><td>GloVe</td><td>37.88</td><td>35.31</td><td>42.39</td><td>35.53</td><td>42.17</td><td>40.36</td><td>47.02</td><td>40.80</td></tr><tr><td>LCM model</td><td>GloVe</td><td>38.77</td><td>36.09</td><td>44.96</td><td>36.08</td><td>45.65</td><td>42.31</td><td>50.91</td><td>43.65</td></tr><tr><td>Base model</td><td>mirrorBERT</td><td>38.85</td><td>35.32</td><td>43.53</td><td>35.89</td><td>44.16</td><td>41.85</td><td>48.54</td><td>42.54</td></tr><tr><td>LCM model</td><td>mirrorBERT</td><td><b>39.64</b></td><td><b>36.61</b></td><td><b>45.00</b></td><td><b>37.30</b></td><td><b>46.19</b></td><td><b>43.81</b></td><td><b>52.58</b></td><td><b>44.43</b></td></tr></tbody></table>

<figure><img src="image_8.png" /><figcaption>Figure 5: Visualization of the attention weights among local features for constructing label prototypes, using GloVe and mirrorBERT word vectors, across various categories. The color intensity indicates the relative importance of the corresponding local feature.</figcaption></figure>

momentum-based strategy. These different variants are compared with the full LCM model in Tab. 15. In addition, we include results of the base model in the table for comparison. Notably, all variants exhibit superior performance compared to the base model, with the proposed LCM model attaining the best overall results.

### A.3.3 Local Features for Query Images

In our method, we initially compare the global features of query images with prototypes. Additionally, we experimented with a variant of the base model where the local features of the query image are matched against those of the prototypes. In this variant, the similarity score is derived from the sum of the top-k values, with k = 12, chosen based on the validation split. This variant achieves mAP scores of 47.54 and 49.11 in the 1-shot and 5-shot settings, respectively. While this represents a slight improvement over our base model (47.02/48.85), it incurs a significantly higher computational cost.

### A.3.4 Visualization of the Attention Mechanism

Fig. 5 illustrates which regions the proposed attention mechanism focuses on according to different categories for two choices of the word vectors. We take the sum of attention weights
<table><caption>Table 13: Overview of the main results on iMaterialist, in terms of micro/macro AP and F1 scores (in %). The best results are highlighted in bold font. “-” indicates that no word embeddings are used.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Word-Emb.</th><th colspan="4">1-shot</th><th colspan="4">5-shot</th></tr><tr><th>Mi-AP</th><th>Mi-F1</th><th>Ma-AP</th><th>Ma-F1</th><th>Mi-AP</th><th>Mi-F1</th><th>Ma-AP</th><th>Ma-F1</th></tr></thead><tbody><tr><td>ResNet-50</td><td>-</td><td>23.58</td><td>27.02</td><td>22.16</td><td>19.33</td><td>34.90</td><td>31.84</td><td>30.06</td><td>25.10</td></tr><tr><td>ResNet-101</td><td>-</td><td>16.71</td><td>26.35</td><td>21.76</td><td>21.45</td><td>38.79</td><td>32.89</td><td>34.39</td><td>28.32</td></tr><tr><td>ViT</td><td>-</td><td>23.39</td><td>29.45</td><td>24.20</td><td>15.54</td><td>38.98</td><td>37.18</td><td>35.44</td><td>28.20</td></tr><tr><td>ResNet-50 + ViT</td><td>-</td><td>24.32</td><td>30.50</td><td>25.27</td><td>17.13</td><td>40.46</td><td>38.96</td><td>35.73</td><td>28.44</td></tr><tr><td>PLA</td><td>Self-learned</td><td>31.61</td><td>27.82</td><td>36.41</td><td>27.65</td><td>41.69</td><td>30.78</td><td>37.08</td><td>30.65</td></tr><tr><td>PLA (GloVe)</td><td>GloVe</td><td>30.43</td><td>25.94</td><td>34.97</td><td>27.46</td><td>40.46</td><td>29.91</td><td>36.49</td><td>30.09</td></tr><tr><td>LaSO</td><td>-</td><td>29.69</td><td>25.31</td><td>30.51</td><td>24.82</td><td>38.78</td><td>29.25</td><td>35.87</td><td>27.82</td></tr><tr><td>MAML</td><td>-</td><td>31.42</td><td>29.40</td><td>37.96</td><td>27.32</td><td>41.70</td><td>31.10</td><td>39.89</td><td>30.17</td></tr><tr><td>WGA</td><td>GloVe</td><td>41.01</td><td>38.64</td><td>46.37</td><td>34.74</td><td>52.13</td><td>41.48</td><td>47.26</td><td>38.32</td></tr><tr><td>Base model</td><td>GloVe</td><td>45.90</td><td>43.45</td><td>51.23</td><td>40.06</td><td>57.96</td><td>46.60</td><td>52.57</td><td>43.49</td></tr><tr><td>LCM model</td><td>GloVe</td><td>47.93</td><td>44.13</td><td>53.09</td><td>41.07</td><td>61.50</td><td><strong>48.97</strong></td><td>56.74</td><td>45.15</td></tr><tr><td>Base model</td><td>mirrorBERT</td><td>48.26</td><td>43.80</td><td>54.25</td><td>42.13</td><td>59.83</td><td>47.07</td><td>54.96</td><td>44.89</td></tr><tr><td>LCM model</td><td>mirrorBERT</td><td><strong>50.09</strong></td><td><strong>44.71</strong></td><td><strong>55.27</strong></td><td><strong>42.25</strong></td><td><strong>62.43</strong></td><td>48.45</td><td><strong>57.62</strong></td><td><strong>45.18</strong></td></tr></tbody></table>

<table><caption>Table 14: Results (in %) on COCO for different attention mechanism. The best results are highlighted in bold.</caption><thead><tr><th>Method</th><th>Difference</th><th>1-shot</th><th>5-shot</th></tr></thead><tbody><tr><td>Base model</td><td rowspan="2">Simple attention</td><td>41.42</td><td>43.25</td></tr><tr><td>LCM model</td><td>41.83</td><td>44.17</td></tr><tr><td>Base model</td><td rowspan="2">Standard attention</td><td>45.84</td><td>46.91</td></tr><tr><td>LCM model</td><td>47.00</td><td>49.45</td></tr><tr><td>Base model</td><td rowspan="2">Attention with global features</td><td>42.68</td><td>45.97</td></tr><tr><td>LCM model</td><td>43.11</td><td>47.60</td></tr><tr><td>Base model</td><td rowspan="2">Low-rank bilinear pooling</td><td>38.82</td><td>41.00</td></tr><tr><td>LCM model</td><td>39.29</td><td>42.71</td></tr><tr><td>Base model</td><td rowspan="2">-</td><td>47.02</td><td>48.85</td></tr><tr><td>LCM model</td><td><strong>48.92</strong></td><td><strong>52.79</strong></td></tr></tbody></table>

across all attention heads as the visualization scores of the local features. As illustrated in the first example of Fig. 5, the model is often successful in identifying the most relevant image region for different categories and the attention weights are indeed label-specific. In this example, the model correctly selects the frisbee or the duck depending on the selected label. This is even though no images with these labels were present in the training data. On the other hand, as the last example shows, for labels that are semantically closely related, such as *zebra* and *cow* in this case, word vectors are not sufficiently informative. For both labels, the model correctly selects the group of animals, but it fails to make a finer selection.

### A.3.5 Visualization of LCM Importance Weights

Fig. 6 visualizes the importance of the different local features, as predicted by the LCM module. The darker the color, the more important the local features are considered to be. Most background local features are given low scores. In fact, many of these features will have a score below the threshold $\theta$ and will thus be completely ignored when aggregating prototypes.
<table><caption>Table 15: Results (in %) on COCO for different ways to select representative local features for the LCM model based on the 5-shot setting. The best results are highlighted in bold.</caption><thead><tr><th>Method</th><th>Ma-AP</th><th>Mi-AP</th></tr></thead><tbody><tr><td>Base model</td><td>48.85</td><td>42.84</td></tr><tr><td>Using ρ<sub>i,k</sub></td><td>51.46</td><td>45.18</td></tr><tr><td>Directly using g(·)</td><td>50.16</td><td>43.55</td></tr><tr><td>Taylor approximation</td><td>51.73</td><td>45.63</td></tr><tr><td>LCM model</td><td><strong>52.79</strong></td><td><strong>46.35</strong></td></tr></tbody></table>

<figure><img src="image_9.png" /><figcaption>Figure 6: Visualization of the distribution of importance scores among local features in the LCM module. The more important the local feature, the darker the corresponding visualization.</figcaption></figure>